Network distributed automated testing system

• US 6,304,982 B1
• Filed: 07/14/1998
• Issued: 10/16/2001
• Est. Priority Date: 07/14/1998
• Status: Expired due to Term

• Current Assignee: Autodesk, Inc.
• Original Assignee: Autodesk, Inc.
• Inventors: DeAguiar, John Ricardo, Cribbs, Dorothy Mack, Mongan, John Thomas
• Primary Examiner(s): Wright, Norman M.
• Application Number:
  US09/114,981
• Time in Patent Office: 1,190 Days
• Field of Search: 714/25, 714/26, 714/31, 714/38, 714/46, 714/4, 714/37, 714/45, 709/217, 709/218, 709/219
• US Class Current: 714/38.14
• CPC Class Codes: G06F 11/3672 Test management

Abstract

A server computer acts as a central repository for tests performed by any number of connected client computers, as well the results of these tests returned by the client computers. A test manager executed by the server computer analyzes the results of the tests performed by the client computers, and determines which tests have passed or failed during execution, whether the tests failed because of an application or other error, and which tests should be re-executed by the same or different client computer. A test grouper analyzes the error messages produced by the application being tested, so that tests that reveal the same defect can be grouped together. A test reducer is iteratively applied to selected tests to reduce the test to the smallest subset of the original test that still reveals the defect. In this manner, the present invention maximizes the testing efficiency of the resources used and minimizes the amount of time required of the operator to confirm failures.

Claims

1. An application program testing apparatus, comprising:

(a) one or more server computers providing a repository for one or more tests and results therefrom; and

(b) one or more client computers, connected to the server computers, for retrieving the tests from the repository, for executing the tests in conjunction with one or more application programs executed by the client computers, and for storing the results from the executed tests in the repository;

wherein each of the server computers includes a test manager for analyzing the results of the tests performed by the client computers and for determining which of the tests have passed and which of the tests have failed and for grouping the tests based on the analyzed results.

2. The apparatus of claim 1, wherein each of the client computers includes means for gathering information about the client computer.

3. The apparatus of claim 2, wherein the gathered information is selected from group comprising (1) information concerning an operating environment of the client computer, (2) information concerning the application program being tested, and (3) information concerning the test being performed.

4. The apparatus of claim 2, wherein each of the server computers includes means for selecting a test for the client computer based on the gathered information.

5. The apparatus of claim 1, wherein the each of the server computers includes a test queue for storing any number of tests available for execution on the client computers.

6. The apparatus of claim 5, wherein each test is represented in the queue by an object that includes information about the test'"'"'s current status, execution history, and which client computers should execute the test.

7. The apparatus of claim 5, wherein a test is selected from the queue in response to a request from the client computer based on a match with information provided by the client computer.

8. The apparatus of claim 1, wherein each of the server computers includes a test container for storing any number of tests being executed on the client computers.

9. The apparatus of claim 1, wherein each of the server computers includes a timer for each test being executed on the client computers.

10. The apparatus of claim 1, wherein the test manager includes means for determining whether the tests failed because of a defect in the application program.

11. The apparatus of claim 1, wherein the test manager includes means for determining whether an error is universal for the application program or specific to a particular type or configuration of client computer.

12. The apparatus of claim 1, wherein the test manager includes means for determining whether the tests should be re-executed by a same or different client computer.

13. The apparatus of claim 1, wherein the test manager groups tests that succeeded, tests with errors, tests that may reveal a defect in an application program but are not reproducible, and tests that reproducibly reveal an application program defect.

14. The apparatus of claim 1, further comprising a test reducer for reducing and re-executing the tests to reveal a portion thereof that reproduces an application program defect.

15. The apparatus of claim 14, wherein the test reducer further comprises means for iteratively reducing and re-executing the tests based on results from previously executed tests.

16. The apparatus of claim 14, wherein the test reducer further comprises means for reducing the tests into syntactically discrete portions.

17. The apparatus of claim 1, further comprising means for ensuring fault tolerance from faults in the client computers.

18. The apparatus of claim 1, further comprising means for ensuring fault tolerance from faults in the server computers.

19. A method for testing application programs, comprising the steps of:

(a) storing one or more tests and results therefrom in a repository of one or more server computers;

(b) retrieving the tests from the repository to one or more client computers connected to the server computers, executing the retrieved tests in conjunction with one or more application programs executed by the client computers, and storing the results of the executed tests from the client computers to the repository of the server computers; and

(c) analyzing the results of the tests performed by the client computers including determining which of the tests have passed and which of the tests have failed based on the results of the tests and grouping the tests based on the results of the tests.

20. The method of claim 19, wherein each of the client computers includes means for gathering information about the client computer.

21. The method of claim 20, wherein the gathered information is selected from group comprising (1) information concerning an operating environment of the client computer, (2) information concerning the application program being tested, and (3) information concerning the test being performed.

22. The method of claim 20, wherein each of the server computers includes means for selecting a test for the client computer based on the gathered information.

23. The method of claim 19, wherein the each of the server computers includes a test queue for storing any number of tests available for execution on the client computers.

24. The method of claim 23, wherein each test is represented in the queue by an object that includes information about the test'"'"'s current status, execution history, and which client computers should execute the test.

25. The method of claim 23, wherein a test is selected from the queue in response to a request from the client computer based on a match with information provided by the client computer.

26. The method of claim 19, wherein each of the server computers includes a test container for storing any number of tests being executed on the client computers.

27. The method of claim 19, wherein each of the server computers includes a timer for each test being executed on the client computers.

28. The method of claim 19, wherein the step of analyzing the results further includes determining whether the tests failed because of a defect in the application program.

29. The method of claim 19, wherein the step of analyzing the results further includes determining whether an error is universal for the application program or specific to a particular type or configuration of client computer.

30. The method of claim 19, wherein the step of analyzing the results further includes determining whether the tests should be re-executed by a same or different client computer.

31. The method of claim 19, wherein grouping the tests comprises grouping tests that succeeded, tests with errors, tests that may reveal a defect in an application program but are not reproducible, and tests that reproducibly reveal an application program defect.

32. The method of claim 19, further comprising the step of:

(a) storing one or more tests and results therefrom in a repository of one or more server computers;

(b) retrieving the tests from the repository to one or more client computers connected to the server computers, executing the retrieved tests in conjunction with one or more application programs executed by the client computers, and storing the results of the executed tests from the client computers to the repository of the server computers; and

(c) reducing and re-executing the tests to reveal a portion thereof that reproduces an application program defect.

33. The method of claim 32, wherein reducing and re-executing further comprises iteratively reducing and re-executing the tests based on results from previously executed tests.

34. The method of claim 32, wherein reducing further comprises reducing the tests into syntactically discrete portions.

35. The method of claim 19, further comprising the step of ensuring fault tolerance from faults in the client computers.

36. The method of claim 19, further comprising the step of ensuring fault tolerance from faults in the server computers.

37. An article of manufacture comprising a computer program carrier readable by a computer and embodying one or more instructions executable by the computer to perform method steps for testing application programs, the method comprising the steps of:

(a) storing one or more tests and results therefrom in a repository of one or more server computers;

(b) retrieving the tests from the repository to one or more client computers connected to the server computers, executing the retrieved tests in conjunction with one or more application programs executed by the client computers, and storing the results of the executed tests from the client computers to the repository of the server computers; and

(c) analyzing the results of the tests performed by the client computers including determining which of the tests have passed and which of the tests have failed based on the results of the tests and grouping the tests based on the results of the tests.

38. The article of manufacture of claim 37, wherein each of the client computers includes means for gathering information about the client computer.

39. The article of manufacture of claim 38, wherein the gathered information is selected from group comprising (1) information concerning an operating environment of the client computer, (2) information concerning the application program being tested, and (3) information concerning the test being performed.

40. The article of manufacture of claim 38, wherein each of the server computers includes means for selecting a test for the client computer based on the gathered information.

41. The article of manufacture of claim 37, wherein the each of the server computers includes a test queue for storing any number of tests available for execution on the client computers.

42. The article of manufacture of claim 41, wherein each test is represented in the queue by an object that includes information about the test'"'"'s current status, execution history, and which client computers should execute the test.

43. The article of manufacture of claim 41, wherein a test is selected from the queue in response to a request from the client computer based on a match with information provided by the client computer.

44. The article of manufacture of claim 37, wherein each of the server computers includes a test container for storing any number of tests being executed on the client computers.

45. The article of manufacture of claim 37, wherein each of the server computers includes a timer for each test being executed on the client computers.

46. The article of manufacture of claim 37, wherein the step of analyzing the results further includes determining whether the tests failed because of a defect in the application program.

47. The article of manufacture of claim 37, wherein the step of analyzing the results further includes determining whether an error is universal for the application program or specific to a particular type or configuration of client computer.

48. The article of manufacture of claim 37, wherein the step of analyzing the results further includes determining whether the tests should be re-executed by a same or different client computer.

49. The article of manufacture of claim 37, wherein grouping the tests comprises grouping tests that succeeded, tests with errors, tests that may reveal a defect in an application program but are not reproducible, and tests that reproducibly reveal an application program defect.

50. The article of manufacture of claim 37, wherein the method further comprises reducing and re-executing the tests to reveal a portion thereof that reproduces an application program defect.

51. The article of manufacture of claim 50, wherein reducing and re-executing further comprises iteratively reducing and re-executing the tests based on results from previously executed tests.

52. The article of manufacture of claim 50, wherein reducing further comprises reducing the tests into syntactically discrete portions.

53. The method of claim 37, further comprising the step of ensuring fault tolerance from faults in the client computers.

54. The method of claim 37, further comprising the step of ensuring fault tolerance from faults in the server computers.

55. An application program testing apparatus, comprising:

(a) one or more server computers providing a repository for one or more tests and results therefrom;

(b) one or more client computers, connected to the server computers, for retrieving the tests from the repository, for executing the tests in conjunction with one or more application programs executed by the client computers, and for storing the results from the executed tests in the repository; and

(c) a test reducer for reducing and re-executing the tests to reveal a portion thereof that reproduces an application program defect;

wherein the test reducer further comprises means for reducing the tests according to one or more of the following techniques;

(a) eliminate all portions of the test following the portion in which the application program defect was detected;

(b) determine if the defect can be detected using (in order) the last portion, the last two portions, etc., before the portion which detects the defect; and

(c) eliminate one or more randomly selected portions;

if re-execution still detects the application program defect;

otherwise replace the removed portions and eliminate a new or complementary set of portions.

56. A method for testing application programs, comprising the steps of:

(a) storing one or more tests and results therefrom in a repository of one or more server computers;

(b) retrieving the tests from the repository to one or more client computers connected to the server computers, executing the retrieved tests in conjunction with one or more application programs executed by the client computers, and storing the results of the executed tests from the client computers to the repository of the server computers; and

(c) reducing and re-executing the tests to reveal a portion thereof that reproduces an application program defect;

wherein reducing further comprises reducing the tests according to one or more of the following techniques;

(a) eliminating all portions of the test following the portion in which the application program defect was detected;

(b) determine if the defect can be detected using (in order) the last portion, the last two portions, etc., before the portion which detects the defect; and

(c) eliminate one or more randomly selected portions;

if re-execution still detects the application program defect;

otherwise replace the removed portions and eliminate a new or complementary set of portions.

57. An article of manufacture comprising a computer program carrier readable by a computer and embodying one or mote instructions executable by the computer to perform method steps for testing application programs, the method comprising the steps of:

(a) storing one or more tests and results therefrom in a repository of one or more server computers;

(b) retrieving the tests from the repository to one or more client computers connected to the server computers, executing the retrieved tests in conjunction with one or mote application programs executed by the client computers, and storing the results of the executed tests from the client computers to the repository of the server computers; and

(c) reducing and re-executing the tests to reveal a portion thereof that reproduces an application program defect;

wherein reducing further comprises reducing the tests according to one or more of the following techniques;

(a) eliminating all portions of the test following the portion in which the application program defect was detected;

(b) determining if the defect can be detected using (in order) the last portion, the last two portions, etc., before the portion which detects the defect; and

(c) eliminating one or more randomly selected portions;

if re-execution still detects the application program defect;

otherwise replace the removed portions and eliminate a new or complementary set of portions.

58. An application program testing apparatus, comprising:

(a) one or more server computers providing a repository for one or more tests and results therefrom; and

(b) one or more client computers, connected to the server computers, for retrieving the tests from the repository, for executing the tests in conjunction with one or more application programs executed by the client computers, and for storing the results from the executed tests in the repository;

wherein the test grouper further comprises means for analyzing error messages generated by the application program during the execution of the test, for comparing the analyzed error messages with an error message database compiled from previously executed tests, and for grouping the tests based on the comparison.

59. An article of manufacture comprising a computer program carrier readable by a computer and embodying one or more instructions executable by the computer to perform method steps for testing application programs, the method comprising the steps of:

(a) providing a repository for one or more tests and results therefrom; and

(b) retrieving the tests from the repository by one or more client computers which execute the tests in conjunction with one or more application programs executed by the client computers, and store the results from the executed tests in the repository;

(c) analyzing error messages generated by the application program during the execution of the test including comparing the analyzed error messages with an error message database compiled from previously executed tests and grouping the tests based on the comparison.

60. A method for testing application programs, comprising the steps of:

(a) providing a repository for one or more tests and results therefrom; and

(b) retrieving the tests from the repository by one or more client computers which execute the tests in conjunction with one or more application programs executed by the client computers, and store the results from the executed tests in the repository;

(c) analyzing error messages generated by the application program during the execution of the test including comparing the analyzed error messages with an error message database compiled from previously executed tests and grouping the tests based on the comparison.