Automated testing of code modules against individually relevant data sets

• US 10,127,144 B2
• Filed: 10/24/2016
• Issued: 11/13/2018
• Est. Priority Date: 10/24/2016
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Johnston, Jay K., Mortensen, Magnus, White, Jr., David C.
• Primary Examiner(s): Aquino, Wynuel S
• Assistant Examiner(s): Gooray, Mark A
• Application Number:
  US15/332,080
• Publication Number:
  US 20180113798A1
• Time in Patent Office: 750 Days
• Field of Search: None
• CPC Class Codes: G06F 11/36 Preventing errors by testing or debugging software
• CPC Class Codes: G06F 11/3604 Software analysis for verifying properties of programs byte-code verification G06F9/44589
• CPC Class Codes: G06F 11/3612 by runtime analysis performance monitoring G06F11/3466
• CPC Class Codes: G06F 11/3664 Environments for testing or debugging software
• CPC Class Codes: G06F 11/3668 Software testing software testing in telephone exchanges H04M3/242, testing of hardware G06F11/22
• CPC Class Codes: G06F 11/3672 Test management
• CPC Class Codes: G06F 11/3684 for test design, e.g. generating new test cases
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites

Abstract

A testing server tests new code modules, or re-tests previously tested code modules, using individualized sets of test data. The testing server receives test datasets from an execution engine, which runs tested code modules to process operational datasets of computing devices. The testing server receives an untested code module from a user, and obtains test dataset parameters applicable to the untested code module. The testing server generates an individualized set of test datasets based on the test dataset parameters. The testing server tests the untested code module by processing each test dataset in the individualized set of test datasets to produce test results.

Claims

1. A computer-implemented method comprising:

monitoring and receiving a plurality of test datasets from an execution engine, the plurality of test datasets being device datasets of a plurality of computing devices that the execution engine processed by utilizing the device datasets as input data for a plurality of tested and validated code modules comprising of one or more technologies, wherein the monitoring further classifies each of the received plurality of test datasets as one of recently received test datasets, worst-case test datasets, or tagged datasets, the tagged datasets include first tags that indicate the one or more technologies of the tested and validated code modules;

receiving an untested code module from a user client, wherein the untested code module includes a second tag that identifies one or more technologies of the untested code module;

obtaining test datasets parameters for-the untested code module, wherein the test dataset parameters include at least one of a first number of tagged datasets, a second number of most recently received test datasets, and a third number of worst-case test datasets to be selected from the plurality of test datasets for the untested code module;

generating a first set of test datasets applicable to the untested code module from the plurality of test datasets based on the test dataset parameters by selecting the first number of tagged datasets that match the second tag of the untested code module, the second number of most recently received test datasets, and the third number of worst-case test datasets from the classified plurality of test datasets as indicated in the test dataset parameters;

testing the untested code module by processing each test dataset in the first set of test datasets to produce test results.

2. The method of claim 1, further comprising notifying an author of the untested code module with the test results.

3. The method of claim 1, further comprising determining whether each particular test dataset of the plurality of test datasets is a worst-case test dataset based on results of the particular test dataset being processed by the plurality of tested code modules on the execution engine.

4. The method of claim 1, further comprising:

re-testing a previously tested code module of the plurality of tested and validated code modules as the untested code module; and

based on unsatisfactory results of the re-testing, removing the previously tested code module from the plurality of tested and validated code modules.

5. The method of claim 4, wherein each of the tested code modules is periodically re-tested as the untested code module.

6. The method of claim 1, wherein the one or more tags include a Virtual Private Network (VPN) tag, a Network Address Translation (NAT) tag, a Session Initiation Protocol (SIP) tag, a routing tag, or a security tag.

7. An apparatus comprising:

a network interface unit configured to communicate with an execution engine that processes a plurality of device datasets of a plurality of computing devices by utilizing the device datasets as input data for a plurality of tested and validated code modules comprising one or more technologies;

a memory; and

a processor coupled to the network interface unit and memory, the processor configured to;

monitor and receive a plurality of test datasets from the execution engine via the network interface unit, the plurality of test datasets being the device datasets of the plurality of computing devices, wherein monitoring further classifies each of the received plurality of test datasets as one of recently received test datasets, worst-case test datasets, or tagged datasets, the tagged datasets include first tags that indicate the one or more technologies of the tested and validated code modules;

receive an untested code module from a user client, wherein the untested code module includes a second tag that identifies one or more technologies of the untested code modules;

obtain test dataset parameters for the untested code module, wherein the test dataset parameters include at least one of a first number of tagged datasets, a second number of most recently received test datasets, and a third number of worst-case test datasets to be selected from the plurality of test datasets for the untested code module;

generate a first set of test datasets applicable to the untested code module from the plurality of test datasets based on the test dataset parameters by selecting the first number of tagged datasets that match the second tag of the untested code module, the second number of most recently received test datasets, and the third number of worst-case test datasets from the classified plurality of test datasets as indicated in the test dataset parameters; and

test the untested code module by processing each test dataset in the first set of test datasets to produce test results.

8. The apparatus of claim 7, wherein the processor is further configured to notify an author of the untested code module with the test results.

9. The apparatus of claim 7, wherein the processor is further configured to determine whether each particular test dataset of the plurality of test datasets is a worst-case test dataset based on results of the particular test dataset being processed by the plurality of tested code modules on the execution engine.

10. The apparatus of claim 7, wherein the one or more tags include a Virtual Private Network (VPN) tag, a Network Address Translation (NAT) tag, a Session Initiation Protocol (SIP) tag, a routing tag, or a security tag.

11. The apparatus of claim 7, wherein the processor is further configured to:

re-test a previously tested code module of the plurality of tested and validated code modules as the untested code module; and

based on unsatisfactory results of the re-testing, remove the previously tested code module from the plurality of tested and validated code modules.

12. The apparatus of claim 11, wherein the processor is further configured to periodically re-test each of the tested code modules as the untested code module.

13. One or more non-transitory computer readable storage media encoded with computer executable instructions operable to cause a processor to:

monitor and receive a plurality of test datasets from an execution engine, the plurality of datasets being device datasets of a plurality of computing devices that the execution engine processed by utilizing the device datasets as input data for a plurality of tested and validated code modules comprising of one or more technologies, wherein the monitoring further classifies each of the received plurality of test datasets as one of recently received test datasets, worst-cast test datasets, or tagged datasets, the tagged datasets include first tags that indicate the one or more technologies of the tested and validated code modules;

receive an untested code module from a user client, wherein the untested code module includes a second tag that identifies one or more technologies of the untested code module;

obtain test dataset parameters for the untested code module, wherein the test dataset parameters includes at least one of a first number of tagged dataset, a second number of most recently received test datasets, and a third number of worst-case test datasets to be selected from the plurality of test datasets for the untested code module;

generate a first set of test datasets applicable to the untested code module from the plurality of test datasets based on the test dataset parameters by selecting the first number of tagged datasets that match the tag of the untested code module, the second number of most recently received test datasets, and the third number of worst-case test datasets from the classified plurality of test datasets as indicated in the test dataset parameters; and

test the untested code module by processing each test dataset in the first set of test datasets to produce test results.

14. The computer readable storage media of claim 13, further comprising instructions operable to cause the processor to notify an author of the untested code module with the test results.

15. The computer readable storage media of claim 13, further comprising instructions operable to cause the processor to determine whether each particular test dataset of the plurality of test datasets is a worst-case test dataset based on results of the particular test dataset being processed by the plurality of tested code modules on the execution engine.

16. The computer readable storage media of claim 13, wherein the one or more tags include a Virtual Private Network (VPN) tag, a Network Address Translation (NAT) tag, a Session Initiation Protocol (SIP) tag, a routing tag, or a security tag.

17. The computer readable storage media of claim 13, further comprising instructions operable to cause the processor to:

periodically re-test a previously tested code module of the plurality of tested and validated code modules as the untested code module; and

based on unsatisfactory results of the re-testing, removing the previously tested code module from the plurality of tested code modules.