Deployment testing for infrastructure delivery automation

• US 10,268,574 B2
• Filed: 09/01/2016
• Issued: 04/23/2019
• Est. Priority Date: 09/01/2016
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Siggers, Ben Matthew, Collins, Michael
• Primary Examiner(s): Bryan, Jason B
• Application Number:
  US15/255,065
• Publication Number:
  US 20180060226A1
• Time in Patent Office: 964 Days
• CPC Class Codes: G06F 11/3664 Environments for testing or debugging software
• CPC Class Codes: G06F 11/3684 for test design, e.g. generating new test cases
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 11/3692 for test results analysis

Abstract

The disclosed deployment testing system includes a test loader that loads and parses test objects and the antecedent test objects on which the selected test object depends, to prepare a test sequence. A database of test objects apply to one or more SUT and specify aspects of deployed SUT configuration that include hardware configuration, active connection of the SUT to other network devices, configuration of the actively connected network devices to accept access by the SUT, presence of files and applications in storage, and services running on the SUT; identify dependencies on antecedent test objects; specify test object features that extend object features found in the antecedent test objects; and override test parameter values in the antecedent test objects. A test executor obtains administrative credentials for accessing and testing a SUT and uses the test and antecedent test objects to verify the SUT; and a test reporter publishes tester results.

Claims

1. A deployment testing system coupled to a computer network, the deployment testing system comprising:

a memory, and

at least one processor coupled to the memory, wherein the at least one processor is configured to;

obtain administrative credentials to access and test at least one system under test (SUT);

load a selected test object and antecedent test objects on which the selected test object depends from a test object database of a plurality of sets of test objects and respective antecedent test objects,

wherein the plurality of sets of test objects and respective antecedent test objects comprise configuration parameters of the at least one SUT, and

wherein the configuration parameters are configured to validate the at least one SUT;

parse the selected test object and the antecedent test objects that are loaded;

build a test sequence using the selected test object and the antecedent test objects as parsed;

wherein to build the test sequence, the at least one processor is further configured to;

identify dependencies on the parsed antecedent test objects, including nested dependencies; and

extend first object features of the parsed selected test object with second object features of the parsed antecedent test objects, comprising;

overriding at least one first object feature of the first object features with test parameter values from the second object features; and

inheriting the dependencies on the parsed antecedent test objects, including the nested dependencies, into the first object features as additional test parameter values;

run the test sequence to verify the configuration parameters of the at least one SUT using the extended first object features having the test parameter values and the additional test parameter values; and

provide results of the run of the test sequence.

2. The deployment testing system of claim 1, wherein the at least one processor is further configured to:

identify, using the plurality of sets of test objects and respective antecedent test objects, a target SUT of the at least one SUT by at least one service role performed by the target SUT; and

verify the at least one service role of the target SUT.

3. The deployment testing system of claim 1, wherein the at least one processor is further configured to:

provide the results of the run of the test sequence to a watcher service on a client device.

4. The deployment testing system of claim 1, wherein the at least one processor is further configured to:

identify, using the plurality of sets of test objects and respective antecedent test objects, a cluster of systems under test of the at least one SUT;

and

verify the identified cluster of systems under test.

5. The deployment testing system of claim 1, wherein the at least one processor is further configured to:

identify, using the plurality of sets of test objects and respective antecedent test objects, a data center; and

verify the at least one SUT in the identified data center.

6. The deployment testing system of claim 1, wherein the configuration parameters comprise data specifying a plurality of hardware configurations of the at least one SUT, a plurality of active connections of the at least one SUT to other network devices, a plurality of configurations of actively connected network devices to accept access by the at least one SUT, identification of files and applications in storage of the at least one SUT, identification of services installed on the at least one SUT, or a plurality of configurations of service initialization parameters for the services installed on the at least one SUT.

7. The deployment testing system of claim 1, wherein the at least one processor is further configured to:

communicate with a subscriber service to access the results of the run of the test sequence.

8. A method of deployment testing of at least one system under test (SUT) of a computer network, the method comprising:

obtaining administrative credentials from an authorization service to access and test the at least one SUT;

loading a selected test object and antecedent test objects on which the selected test object depends from a test object database of a plurality of sets of test objects and respective antecedent test objects,

wherein the plurality of sets of test objects and respective antecedent test objects comprise configuration parameters of the at least one SUT, and

wherein the configuration parameters are configured to validate the at least one SUT;

parsing the selected test object and the antecedent test objects that are loaded;

building a test sequence using the selected test object and the antecedent test objects as parsed;

wherein building the test sequence further comprises;

identifying dependencies on the parsed antecedent test objects, including nested dependencies; and

extending first object features of the parsed selected test object with second object features of the parsed antecedent test objects, wherein the extending comprises;

overriding at least one first object feature of the first object features with test parameter values from the second object features; and

inheriting the dependencies on the parsed antecedent test objects, including the nested dependencies, into the first object features as additional test parameter values;

running the test sequence to verify the configuration parameters of the at least one SUT using the extended first object features having the test parameter values and the additional test parameter values; and

providing results of the run of the test sequence.

9. The method of claim 8, further comprising:

identifying, using the plurality of sets of test objects and respective antecedent test objects, a target SUT of the at least one SUT by at least one service role performed by the target SUT; and

verifying the at least one service role of the target SUT.

10. The method of claim 8, further comprising:

identifying, using the plurality of sets of test objects and respective antecedent test objects, a cluster of systems under test of the at least one SUT; and

verifying the identified cluster of systems under test.

11. The method of claim 8, further comprising:

identifying, using the plurality of sets of test objects and respective antecedent test objects, a data center; and

verifying the at least one SUT in the identified data center.

12. The method of claim 8, wherein the configuration parameters comprise data specifying a plurality of hardware configurations of the at least one SUT, a plurality of active connections of the at least one SUT to other network devices, a plurality of configurations of actively connected network devices to accept access by the at least one SUT, identification of files and applications in storage of the at least one SUT, identification of services installed on the at least one SUT, or a plurality of configurations of service initialization parameters for the services installed on the at least one SUT.

13. The method of claim 8, further comprising:

communicating with a subscriber service to access the results of the run of the test sequence.

14. The method of claim 8, further comprising:

providing the results of the run of the test sequence to a watcher service on a client device.

15. A tangible non-transitory computer-readable device having instructions stored thereon that, when executed by a computing device, cause the computing device to perform operations comprising:

obtaining administrative credentials from an authorization service to access and test at least one system under test (SUT) of a computer network;

loading a selected test object and antecedent test objects on which the selected test object depends from a test object database of a plurality of sets of test objects and respective antecedent test objects,

wherein the plurality of sets of test objects and respective antecedent test objects comprise configuration parameters of the at least one SUT, and

wherein the configuration parameters are configured to validate the at least one SUT;

parsing the selected test object and the antecedent test objects that are loaded;

building a test sequence using the selected test object and the antecedent test objects as parsed;

wherein building the test sequence further comprises;

identifying dependencies on the parsed antecedent test objects, including nested dependencies; and

extending first object features of the parsed selected test object with second object features of the parsed antecedent test objects, wherein the extending comprises;

overriding at least one first object feature of the first object features with test parameter values from the second object features; and

inheriting the dependencies on the parsed antecedent test objects, including the nested dependencies, into the first object features as additional test parameter values;

running the test sequence to verify the configuration parameters of the at least one SUT using the extended first object features having the test parameter values and the additional test parameter values; and

providing results of the run of the test sequence.

16. The computer-readable device of claim 15, the operations performed by the computing device further comprising:

identifying, using the plurality of sets of test objects and respective antecedent test objects, a target SUT of the at least one SUT by at least one service role performed by the target SUT; and

verifying the at least one service role of the target SUT.

17. The computer-readable device of claim 15, the operations performed by the computing device further comprising:

identifying, using the plurality of sets of test objects and respective antecedent test objects, a cluster of systems under test of the at least one SUT; and

verifying the identified cluster of systems under test.

18. The computer-readable device of claim 15, the operations performed by the computing device further comprising:

identifying, using the plurality of sets of test objects and respective antecedent test objects, a data center; and

verifying the at least one SUT in the identified data center.

19. The computer-readable device of claim 15, wherein the configuration parameters comprise data specifying a plurality of hardware configurations of the at least one SUT, a plurality of active connections of the at least one SUT to other network devices, a plurality of configurations of actively connected network devices to accept access by the at least one SUT, identification of files and applications in storage of the at least one SUT, identification of services installed on the at least one SUT, or a plurality of configurations of service initialization parameters for the services installed on the at least one SUT.

20. The computer-readable device of claim 15, the operations performed by the computing device further comprising:

communicating with a subscriber service to access the results of the run of the test sequence.

21. The computer-readable device of claim 15, the operations performed by the computing device further comprising:

providing the results of the run of the test sequence to a watcher service on a client device.