Method and system for automated distributed software testing

• US 20070168970A1
• Filed: 11/07/2005
• Published: 07/19/2007
• Est. Priority Date: 11/07/2005
• Status: Active Grant

• Current Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Original Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Inventors: Li, Richard, Ven, Adriaan, Peck, William
• Granted Patent: US 8,166,458 B2
• Time in Patent Office: Days
• US Class Current: 717/124
• CPC Class Codes: G06F 11/3664 Environments for testing or debugging software
• CPC Class Codes: G06F 11/3696 Methods or tools to render software testable
• CPC Class Codes: G06F 2209/5021 Priority
• CPC Class Codes: G06F 9/5038 considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration scheduling strategies G06F9/4881 and subgroups
• CPC Class Codes: G06F 9/5044 considering hardware capabilities
• CPC Class Codes: G06F 9/5055 considering software capabilities, i.e. software resources associated or available to the machine

Abstract

Embodiments of the present invention provide methods and systems for automated distributed testing of software. A testing service may utilize a distributed architecture that provides a standardized framework for writing tests, scheduling the tests, and gathering and reporting results of the tests. Multiple distributed labs are integrated into the testing service and their environments can be centrally managed by the testing service. The testing service permits the scheduling and performance of tests across multiple machines within a test lab, or tests that span across multiple test labs. Any of the machines in the test labs may be selected based on variety of criteria. The testing service may then automatically locate the appropriate machines that match or satisfy the criteria and schedule the tests when the machines are available. The testing service may also automatically create and set up a desired test environment according to the desired specifications for the test.

Claims

1. A method of testing software, said method comprising:

receiving a request to conduct a test;

identifying requirements of the test based on information in the request;

determining at least one test lab having a target machine that satisfies at least some of the requirements of the test;

scheduling the test with the at least one test lab; and

receiving information that indicates results of the test.

2. The method of claim 1, further comprising determining a cost to conduct the test in the at least one test lab.

3. The method of claim 1, wherein identifying requirements of the test comprises identifying an execution environment required for the test.

4. The method of claim 1, wherein identifying requirements of the test comprises identifying at least one of a name assigned to the test, a description of the test, a location for storing results of the test, one or more architectures that are appropriate for the test, releases of an operating system that are appropriate for the test, and a maximum amount of time that the test should execute.

5. The method of claim 1, wherein scheduling the test with the at least one test lab comprises:

identifying a set of target machines in the at least one test lab that satisfy the requirements of the test;

determining a load of the set of target machines; and

selecting at least one target machine from the set of target machines based on their respective load.

6. The method of claim 1, further comprising:

receiving a request to add a target machine to the at least one test lab;

determining a configuration of the target machine to be added; and

adding the target machine to the at least one test lab based on its configuration.

7. The method of claim 1, wherein receiving information that indicates results of the test comprises receiving results for a test on a full application stack conducted on the target machine.

8. A distributed testing system, said system comprising:

a plurality of test labs comprising a set of configurable target machines, and a sentry server that monitors the set of configurable target machines; and

a control server coupled to the plurality of test labs and configured to track capabilities of the set of configurable target machines, schedule tests to be performed in the plurality of test labs, and provide the sentry server constraints on the tests to be performed in the plurality of test labs.

9. The system of claim 8, wherein at least one of the plurality of test labs comprises a local sentry server configured to monitor progress of a test as it is being performed in target machines in the at least one test lab, wherein the local sentry server is configured to monitor the test without interrupting the progress of the test.

10. The system of claim 8, wherein at least one of the plurality of test labs comprises a local results server configured to gather results of a test from the target machines in the at least one test lab.

11. The system of claim 8, wherein the plurality of test labs comprises a sentry server that monitors progress of tests on each target machine in the set of configurable target machines and is configured to interrupt tests on each target machine in response to an event.

12. The system of claim 8, wherein the plurality of test labs comprises a sentry server that monitors progress of tests on each target machine in the set of configurable target machines and is configured to interrupt tests on each target machine in response to a failure of an event.

13. The system of claim 8, wherein the control server is configured to schedule tests performed in the plurality of test labs based on availability of the set of target machines in each of the plurality of the test labs.

14. The system of claim 8, wherein the control server is configured to schedule a test on a plurality of target machines across a plurality of the test labs.

15. The system of claim 8, wherein the control server is configured to gather information from the test labs in response to errors generated during tests.

16. The system of claim 8, wherein the control server is configured to create an installation package for the test that is installed on the target machines that are scheduled to perform the test.

17. The system of claim 8, further comprising a results server for gathering results of the tests performed in all of the plurality of test labs.

18. The system of claim 8, further comprising a results server configured to gather results of tests from at least one of the plurality of test labs and wherein the at least one test lab comprises a local results server configured to gather results of tests performed in the at least one test lab.

19. The system of claim 8, further comprising a results server for gathering results of the tests performed in all of the plurality of test labs and for formatting the results according to a format.

20. The system of claim 8, further comprising a results server for gathering results of the tests performed in all of the plurality of test labs and for reporting test failures in the plurality of test labs.

21. The system of claim 8, further comprising a test repository server that is configured to provide tests previously performed in the plurality of test labs.

22. The system of claim 8, further comprising a test repository server that is configured to provide a set of templates for the tests.

23. The system of claim 8, further comprising a test repository server that is configured to provide a set of helper packages that assist in configuring the test.