Method and system for automated distributed software testing

• US 8,166,458 B2
• Filed: 11/07/2005
• Issued: 04/24/2012
• Est. Priority Date: 11/07/2005
• Status: Active Grant

• Current Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Original Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Inventors: Li, Richard Ding, Peck, William F., van de Ven, Adriaan
• Primary Examiner(s): Puente, Emerson
• Assistant Examiner(s): SWIFT, CHARLES M
• Application Number:
  US11/267,276
• Publication Number:
  US 20070168970A1
• Time in Patent Office: 2,360 Days
• Field of Search: 717/124, 717/120, 717/127, 717/130, 718/102, 709/223, 709/226, 714/38.1, 714/38.14, 714/4.1
• US Class Current: 717/124
• CPC Class Codes: G06F 11/3664 Environments for testing or debugging software
• CPC Class Codes: G06F 11/3696 Methods or tools to render software testable
• CPC Class Codes: G06F 2209/5021 Priority
• CPC Class Codes: G06F 9/5038 considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration scheduling strategies G06F9/4881 and subgroups
• CPC Class Codes: G06F 9/5044 considering hardware capabilities
• CPC Class Codes: G06F 9/5055 considering software capabilities, i.e. software resources associated or available to the machine

Abstract

Embodiments of the present invention provide methods and systems for automated distributed testing of software. A testing service may utilize a distributed architecture that provides a standardized framework for writing tests, scheduling the tests, and gathering and reporting results of the tests. Multiple distributed labs are integrated into the testing service and their environments can be centrally managed by the testing service. The testing service permits the scheduling and performance of tests across multiple machines within a test lab, or tests that span across multiple test labs. Any of the machines in the test labs may be selected based on variety of criteria. The testing service may then automatically locate the appropriate machines that match or satisfy the criteria and schedule the tests when the machines are available. The testing service may also automatically create and set up a desired test environment according to the desired specifications for the test.

Claims

1. A method of testing software, said method comprising:

receiving, at a testing control service that comprises a server, a request to conduct a test of software;

identifying, by the testing control service, test parameters to test the software from the request, wherein the test parameters comprise requirements of testing the software;

identifying a remote test lab from a plurality of remote tests labs, the remote test lab comprising a plurality of target machines comprising resources that satisfy the requirements of the test, wherein each target machine comprises a virtual machine monitor, at least one target virtual machine to perform the test, and a management virtual machine to track the at least one target virtual machine performing the test;

scheduling the test with the plurality of target machines of the remote test lab;

connecting to a sentry server machine located at the remote test lab, wherein the sentry server machine is configured to monitor whether the test resulted in at least one target machine of the remote test lab failing by periodically retrieving information from the management virtual machine in a target machine; and

receiving information from the sentry server machine that indicates a result of the test at the remote test lab.

2. The method of claim 1, further comprising determining a cost to conduct the test in the remote test lab.

3. The method of claim 1, wherein an operating system running inside a target virtual machine is based on the requirements of the test.

4. The method of claim 1, wherein the requirements of the test comprise at least one of a name assigned to the test, a description of the test, a location for storing results of the test, an architecture that is appropriate for the test, a release of an operating system that is appropriate for the test, and a maximum amount of time for which the test should execute.

5. The method of claim 1, wherein identifying the remote test lab comprising the plurality of target machines comprising resources that satisfy the requirements of the test comprises:

determining a load of each target machine of the set of the plurality of target machines; and

selecting at least one target machine from the set of the plurality of target machines based on its respective load.

6. The method of claim 1, further comprising:

receiving a request to add an additional target machine to the remote test lab;

determining a configuration of the additional target machine; and

adding the additional target machine to the remote test lab based on the configuration.

7. The method of claim 1, wherein the test at the remote test lab comprises a full test on a full application stack conducted at the remote test lab.

8. A distributed testing system, said system comprises:

a plurality of test labs, wherein each of the test labs comprises;

a set of configurable target machines, wherein each target machine comprises a virtual machine monitor, at least one target virtual machine to perform a test of software, and a management virtual machine to track the at least one target virtual machine performing the test, and

a local sentry server machine configured to monitor the set of configurable target machines and determine whether a scheduled test results in at least one target machine failing by periodically retrieving information from the management virtual machine in a target machine; and

a control server coupled to the plurality of test labs and configured to receive a request to conduct the test at one of the plurality of test labs, identify test parameters to test the software from the request, wherein the test parameters comprise requirements of testing the software, identify capabilities of the set of configurable target machines in the one of the plurality of test labs, determine that the set of configurable target machines comprise resources that satisfy the requirements of the test, schedule the test to be performed on the set of configurable target machines of the one of the plurality of test labs, and provide the local sentry server machine with a constraint on the test to be performed.

9. The system of claim 8, wherein the local sentry server is further configured to monitor a progress of the test being performed in the one of the plurality of test labs without interrupting the progress of the test by retrieving information from the management virtual machine in a target machine.

10. The system of claim 8, wherein the one of the plurality of test labs comprises a local results server configured to gather results of the test from the set of configurable target machines in the one of the plurality of test labs.

11. The system of claim 8, wherein the local sentry server is further configured to interrupt the test on each target machine in the set of configurable target machines in response to an event by sending a command to the management virtual machine in a target machine.

12. The system of claim 8, wherein the local sentry server machine is further configured to monitor a progress of the test on each target machine in the set of configurable target machines by retrieving information from the management virtual machine in a target machine and interrupt the test on each target machine in the set of configurable target machines in response to a failure of an event by sending a command to the management virtual machine in a target machine.

13. The system of claim 8, wherein the control server is further configured to schedule the test based on an availability of the set of target configurable machines in the one of the plurality of test labs.

14. The system of claim 8, wherein the control server is further configured to schedule the test on a plurality of target machines across the plurality of test labs.

15. The system of claim 8, wherein the control server is further configured to gather information from the one of the plurality of test labs in response to an error generated during the test by retrieving information from the management virtual machine in a target machine.

16. The system of claim 8, wherein the control server is further configured to create an installation package for the test that is installed on target machines of the set of configurable target machines that are scheduled to perform the test.

17. The system of claim 8, further comprising a results server configured to gather a result of the test performed in the one of the plurality of test labs.

18. The system of claim 8, further comprising a results server configured to gather a result of the test performed in the one of the plurality of test labs from a local results server associated with the one of the plurality of test labs.

19. The system of claim 8, further comprising a results server configured to gather a result of the test performed in the one of the plurality of test labs and format the result according to a format.

20. The system of claim 8, further comprising a results server configured to gather a result of the test performed in the one of the plurality of test labs and report a test failure.

21. The system of claim 8, further comprising a test repository server configured to provide a previous test that was previously performed in the one of the plurality of test labs.

22. The system of claim 8, further comprising a test repository server configured to provide a template for the test.

23. The system of claim 8, further comprising a test repository server configured to provide a helper package that assists in configuring the test.

24. A method of testing software, said method comprising:

receiving, at a remote test lab of a plurality of remote test labs comprising a plurality of target machines, a request to schedule a test on a target machine in the remote test lab, wherein the request is received in response to a remote testing control service generating a test script for the test, wherein the test script comprises requirements of the test, and determining that the target machine comprises resources that satisfy the requirements of the test, wherein the target machine comprises a virtual machine monitor, at least one target virtual machine to perform the test, and a management virtual machine to track the at least one target virtual machine performing the test;

locally monitoring the target machine in the remote test lab, by a sentry server machine, during the test to determine whether the test results in the target machine fail by retrieving information from the management virtual machine in the target machine; and

sending, to the remote testing control service, information indicating that the target machine failed.

25. A computer readable storage medium including instructions that, when executed by a computer, cause the computer to perform a method comprising:

receiving a request to conduct a test of software;

identifying, by the computer, test parameters to test the software from the request, wherein the test parameters comprise requirements of testing the software;

identifying a remote test lab from a plurality of remote test labs, the remote test lab comprising a plurality of target machines comprising resources that satisfy the requirements of the test, wherein each target machine comprises a virtual machine monitor, at least one target virtual machine to perform the test, and a management virtual machine to track the at least one target virtual machine performing the test;

scheduling the test with the plurality of target machines of the remote test lab;

connecting to a sentry server machine located at the remote test lab, wherein the sentry server machine is configured to monitor whether the test resulted in at least one target machine of the remote test lab failing by periodically retrieving information from the management virtual machine in a target machine; and

receiving information from the sentry server machine that indicates a result of the test at the remote test lab.