Automated testing of cluster data services

• US 7,370,101 B1
• Filed: 12/19/2003
• Issued: 05/06/2008
• Est. Priority Date: 12/19/2003
• Status: Active Grant

• Current Assignee: Oracle America, Inc. (Oracle Corporation)
• Original Assignee: Sun Microsystems Incorporated (Oracle Corporation)
• Inventors: Kesavan, Ambujavalli, Lakkapragada, Sekhar
• Primary Examiner(s): Nguyen; Dustin
• Application Number:
  US10/741,101
• Time in Patent Office: 1,600 Days
• Field of Search: 709/219, 709/224, 709/208, 709/220, 714/4, 714/38, 714/124, 713/201, 702/120, 702/186
• US Class Current: 709/223
• CPC Class Codes: G06F 11/2294 by remote test
• CPC Class Codes: G06F 11/3664 Environments for testing or debugging software

Abstract

Methods and apparatus, including computer program products, implementing and using techniques for testing a data service on a computing cluster having several computing nodes. A test package is installed on a test administration machine and on one or more of the computing nodes in the computing cluster. Data service configuration information is collected for the data service to be tested. Computing cluster configuration information is collected. The data service configuration information and the computing cluster configuration information are distributed to one or more of the computing nodes in the computing cluster. The data service is tested on the computing cluster by applying one or more data service test suites in the test package to the data service. A report containing results of the application of one or more of the data service test suites to the data service is generated.

Claims

1. A computer-implemented method for testing a data service operable to run on a computing cluster having a plurality of computing nodes, comprising:

installing a test package on a test administration machine and on one or more of the computing nodes in the computing cluster;

invoking information gathering tool on test administration machine either locally or from a remote location;

collecting data service configuration information for the data service to be tested, wherein the data service configuration information comprises information about at least one application that is to be tested and has been configured to run on a plurality of nodes in the computer cluster, rather than on a single server, such that the running of the application on a plurality of nodes behaves as a single application;

invoking a RunTest program operable to automatically run one or more of the data service test suites identified by the information gathering tool to test the data service;

determining whether a valid master configuration file exists;

collecting computing cluster configuration information for the computing cluster on which the data service will be invoked if the valid master configuration file does not exist;

distributing the data service configuration information and the computing cluster configuration information to one or more of the computing nodes in the computing cluster;

testing the data service on the computing cluster by applying one or more data service test suites in the test package to the data service; and

generating a report containing results of the application of one or more of the data service test suites to the data service.

2. The method of claim 1, further comprising:

installing the test package on one or more client computers that are operable to interact with data service;

collecting client configuration information for the one or more clients; and

distributing the client configuration information to one or more of the computing nodes in the computing cluster.

3. The method of claim 1, wherein collecting data service configuration information includes:

invoking an information-gathering tool on the test administration machine to collect the data service configuration information.

4. The method of claim 1, wherein collecting data service configuration information includes:

receiving a user input defining whether the data service is a failover data service, a scalable data service, a combination of a failover and a scalable data service, or a preconfigured data service.

5. The method of claim 2, further comprising:

distributing the data service configuration information and the computing cluster configuration information to one or more clients.

6. The method of claim 1, wherein testing the data service comprises:

applying one or more data service test suites in the test package to the data service using test administration machine.

7. The method of claim 1, wherein testing the data service comprises:

performing functional testing of the data service.

8. The method of claim 1, wherein testing the data service comprises:

performing fault injection testing of the data service.

9. The method of claim 1, further comprising:

receiving a user input defining which data service test suites to apply during the testing step.

10. The method of claim 1, wherein at least some of the data service test suites are arranged to test the data service while performing at least one of:

creating and deleting resource groups, creating and deleting resources, enabling and disabling resources, bringing resource groups online and offline, enabling and disabling fault monitors for a resource, managing and unmanaging resource groups, adding and deleting resource types, switching over a resource group to a different computing node in the computing cluster, reboot failover, panic failover, failover with failback, kill application daemons, kill fault monitor daemons, and kill application daemons multiple times to cause failover.

11. The method of claim 1, further comprising:

generating a summarized report; and

presenting one or more of the report and the summarized report to the user.

12. The method of claim 1, wherein the data service is a highly available data service.

13. A computer program product, stored on a machine-readable medium, comprising instructions operable to cause a computer to:

install a test package on a test administration machine and on one or more of the computing nodes in the computing cluster;

invoking information gathering tool on test administration machine either locally or from a remote location;

collect data service configuration information for the data service to be tested, wherein the data service configuration information comprises information about at least one application that is to be tested and has been configured to run on a plurality of nodes in the computer cluster, rather than on a single server, such that the running of the application on a plurality of nodes behaves as a single application;

invoking a RunTest program operable to automatically run one or more of the data service test suites identified by the information gathering tool to test the data service;

determining whether a valid master configuration file exists;

collect computing cluster configuration information for the computing cluster on which the data service will be invoked if the valid master configuration file does not exist;

distribute the data service configuration information and the computing cluster configuration information to one or more of the computing nodes in the computing cluster;

test the data service on the computing cluster by applying one or more data service test suites in the test package to the data service; and

generate a report containing results of the application of one or more of the data service test suites to the data service.

14. The computer program product of claim 13, further comprising instructions to:

install the test package on one or more client computers that are operable to interact with data service;

collect client configuration information for the one or more clients; and

distribute the client configuration information to one or more of the computing nodes in the computing cluster.

15. The computer program product of claim 13, wherein the instructions to collect data service configuration information include instructions to:

invoke an information-gathering tool on the test administration machine to collect the data service configuration information.

16. The computer program product of claim 13, wherein the instructions to collect data service configuration information include instructions to:

receive a user input defining whether the data service is a failover data service, a scalable data service, a combination of a failover and a scalable data service, or a preconfigured data service.

17. The computer program product of claim 14, further comprising instructions to:

distribute the data service configuration information and the computing cluster configuration information to one or more clients.

18. The computer program product of claim 13, wherein the instructions to test the data service comprise instructions to:

apply one or more data service test suites in the test package to the data service using test administration machine.

19. The computer program product of claim 13, wherein the instructions to test the data service comprise instructions to:

perform functional testing of the data service.

20. The computer program product of claim 13, wherein instructions to test the data service comprise instructions to:

perform fault injection testing of the data service.

21. The computer program product of claim 13, further comprising instructions to:

receive a user input defining which data service test suites to apply during the testing step.

22. The computer program product of claim 13, wherein at least some of the data service test suites are arranged to test the data service while performing at least one of:

creating and deleting resource groups, creating and deleting resources, enabling and disabling resources, bringing resource groups online and offline, enabling and disabling fault monitors for a resource, managing and unmanaging resource groups, adding and deleting resource types, switching over a resource group to a different computing node in the computing cluster, reboot failover, panic failover, failover with failback, kill application daemons, kill fault monitor daemons, and kill application daemons multiple times to cause failover.

23. The computer program product of claim 13, further comprising instructions to:

generate a summarized report; and

present one or more of the report and the summarized report to the user.

24. The computer program product of claim 13, wherein the data service is a highly available data service.

25. A computer-implemented method for testing a data service operable to run on a computing cluster having a plurality of computing nodes, comprising:

installing a test package on a test administration machine and on one or more of the computing nodes in the computing cluster, the test package including one or more data service test suites;

invoking information gathering tool on test administration machine either locally or from a remote location;

collect data service configuration information for the data service to be tested

invoking a RunTest program operable to automatically run one or more of the data service test suites identified by the information gathering tool to test the data service;

determining whether a valid master configuration file exists;

distribute the data service configuration information to one or more of the computing nodes in the computing cluster;

automatically running a pre-defined set of data service test suites to test the data service wherein the data service comprises at least one application that is to be tested and has been configured to run on a plurality of nodes in the computer cluster, rather than on a single server, such that the running of the application on a plurality of nodes behaves as a single application; and

automatically generating a report indicating the outcome of the testing of the data service.

26. The method of claim 25, further comprising:

obtaining configuration information for the data service and the computing cluster on which the data service will be invoked.

27. The method of claim 25, further comprising:

installing the test package on one or more client computers that are operable to interact with data service; and

collecting client configuration information for the one or more clients.

28. The method of claim 26, wherein obtaining configuration information includes:

receiving a user input defining whether the data service is a failover data service, a scalable data service, a combination of a failover and a scalable data service, or a preconfigured data service.

29. The method of claim 25, wherein automatically running a pre-defined set of data service test suites:

applying one or more data service test suites in the test package to the data service using test administration machine.

30. The method of claim 25, wherein automatically running a pre-defined set of data service test suites comprises:

performing functional testing of the data service.

31. The method of claim 30, wherein performing functional testing comprises one or more of:

automatically creating and deleting resource groups used by the data service, automatically creating and deleting resources used by the data service, automatically enabling and disabling resources used by the data service, automatically bringing resource groups used by the data service online and offline, automatically enabling and disabling fault monitors for a resource used by the data service, automatically managing and unmanaging resource groups used by the data service, automatically adding and deleting resource types after the data service has been installed, and automatically switching over a resource group used by the data service to a different computing node in the computing cluster.

32. The method of claim 25, wherein automatically running a pre-defined set of data service test suites comprises:

performing fault injection testing of the data service.

33. The method of claim 32, wherein performing fault injection testing comprises one or more of:

automatically performing reboot failover of a node in the computing cluster that is used by the data service with failback enabled, automatically performing panic failover of a node in the computing cluster that is used by the data service, automatically performing failover of a node in the computing cluster that is used by the data service, automatically kill an application daemon that is used by the data service, automatically kill a fault monitor daemon that is used by the data service, and automatically kill an application daemon that is used by the data service multiple times to cause a failover.

34. The method of claim 25, wherein the data service is a highly available data service.

35. A test package for testing a data service encoded on a computer readable storage media and operable to run on a computing cluster having a plurality of computing nodes, comprising:

a plurality of data service test suites, each data service test suite being operable to test at least one component of the computing cluster;

an information gathering tool operable to receive information about the computing cluster, any clients of the computing cluster, and the data service to be tested, wherein the data service comprises at least one application that is to be tested and has been configured to run on a plurality of nodes in the computer cluster, rather than on a single server, such that the running of the application on a plurality of nodes behaves as a single application; and

a RunTest program operable to automatically run one or more of the data service test suites identified by the information gathering tool to test the data service; and

a report tool operable to produce a summary report indicative of the test results.

36. The test package of claim 35, wherein the information gathering tool is operable to receive a user input defining the computing cluster'"'"'s configuration and about which data service test suites to run.

37. The test package of claim 35, further comprising:

a report tool operable to produce a summary report that indicates whether the data service test suites identified by the information gathering tool passed or failed during the testing.

38. The test package of claim 35, wherein:

the information gathering tool further is operable to receive a user input defining whether the data service is a failover data service, a scalable data service, a combination of a failover and a scalable data service, or a pre-configured data service.

39. The test package of claim 35, wherein:

the data service test suites include fault injection tests and functional tests.

40. The test package of claim 39, wherein:

the RunTest program is operable to test the data service while performing at least one of;

creating and deleting resource groups, creating and deleting resources, enabling and disabling resources, bringing resource groups online and offline, enabling and disabling fault monitors for a resource, managing and unmanaging resource groups, adding and deleting resource types, switching over a resource group to a different computing node in the computing cluster, reboot failover, panic failover, failover with failback, kill application daemons, kill fault monitor daemons, and kill application daemons multiple times to cause failover.