Integration testing method and system for web services

• US 9,367,435 B2
• Filed: 12/12/2013
• Issued: 06/14/2016
• Est. Priority Date: 12/12/2013
• Status: Active Grant

• Current Assignee: Vertafore Incorporated
• Original Assignee: Vertafore Incorporated
• Inventors: Sinyagin, Aleksey, Vinall, Justin P., Wang, Fang
• Primary Examiner(s): Zhen, Wei
• Assistant Examiner(s): LUU, BINH K
• Application Number:
  US14/104,749
• Publication Number:
  US 20150169432A1
• Time in Patent Office: 915 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 11/3684 for test design, e.g. generating new test cases

Abstract

A method of testing a web service includes obtaining web service metadata from the web service. Test cases are generated automatically using the web service metadata, heuristic algorithm modules selected based on the contents of the web service metadata, and stochastic processes. Energy selection logic is used to update test parameters included in the test cases between rounds of testing until a set of test cases is generated that produces test results meeting one or more predetermined criteria. Testing is performed periodically using that set of test cases until the test results indicate that the web service has changed. New test cases are automatically generated and used to test the web service until test results meet the one or more predetermined criteria.

Claims

1. A method performed by at least one computer system that tests a web service, the method comprising:

obtaining web service metadata from the web service;

automatically generating a first set of test cases at least in part based on the web service metadata obtained from the web service;

automatically testing the web service using the first set of test cases;

automatically generating first learning data regarding the first set of test cases by associating each test case of the first set of test cases with at least one respective test metric;

automatically generating a second set of test cases at least in part based on the web service metadata obtained from the web service;

automatically testing the web service using the second set of test cases;

automatically generating second learning data regarding the second set of test cases by associating each test case of the second set of test cases with at least one respective test metric;

automatically comparing the second learning data to the first learning data;

automatically determining, at least in part based on the comparison between the first and the second learning data, that a third set of test cases is to be generated at least in part based on which one of the first or the second set of test cases performs worse relative to the other of the first or the second set of test cases;

responsive to determining that the first set of test cases performs worse relative to the second set of test cases, automatically generating the third set of test cases at least in part based on the first set of test cases, the third set of test cases different from the first set of test cases; and

responsive to determining that the second set of test cases performs worse relative to the first set of test cases, automatically generating the third set of test cases at least in part based on the second set of test cases, the third set of test cases different from the second set of test cases.

2. The method of claim 1 wherein the first set of test cases is automatically generated at least in part based on a name of a function included in the web service metadata obtained from the web service.

3. The method of claim 1 wherein the first set of test cases is automatically generated at least in part based on a name of a variable included in the web service metadata obtained from the web service.

4. The method of claim 1 wherein the first set of test cases is automatically generated at least in part based on a data type of a variable included in the web service metadata obtained from the web service.

5. The method of claim 1 wherein the first set of test cases is automatically generated at least in part based on at least one constraint on a variable included in the web service metadata obtained from the web service.

6. The method of claim 5 wherein the at least one constraint on the variable includes at least one of:

a minimum value, a maximum value, and a string length.

7. The method of claim 1, further comprising:

selecting one of a plurality of predetermined sets of test parameters based on the web service metadata; and

selecting a plurality of test parameters from the selected one of the plurality of predetermined sets of test parameters,

wherein the first set of test cases is automatically generated using the selected plurality of test parameters.

8. The method of claim 7, further comprising:

generating a plurality of random numbers,

wherein the plurality of test parameters are selected from the selected one of the plurality of predetermined sets of test parameters using the plurality of random numbers.

9. The method of claim 1 wherein the first learning data is generated by associating each test case of the first set of test cases with at least one respective test metric, and the second learning data is generated by associating each test case of the second set of test cases with at least one respective test metric, and the third set of test cases is generated at least in part based on the first or the second learning data.

10. The method of claim 1 wherein the first learning data is generated by associating each test case of the first set of test cases with at least one respective first code coverage metric and at least one respective first execution time metric, the second learning data is generated by associating each test case of the second set of test cases with at least one respective second code coverage metric and at least one respective second execution time metric, and the third set of test cases is generated at least in part based on the first or the second learning data.

11. The method of claim 1, further comprising:

evaluating energy selection logic using the first and the second learning data,

wherein a result of the evaluating of the energy selection logic is used to determine whether the third set of test cases is to be generated using the first or the second set of test cases.

12. The method of claim 11 wherein the evaluating of the energy selection logic includes:

calculating the sum of a plurality of test metrics associated with the first set of test cases;

calculating the sum of a plurality of test metrics associated with the second set of test cases; and

calculating a ratio that includes the sum of the plurality of test metrics associated with the first set of test cases and the sum of the plurality of test metrics associated with the second set of test cases.

13. The method of claim 12 wherein the evaluating of the energy selection logic further includes:

calculating the reciprocal of the sum of a plurality of test metrics associated with the first set of test cases; and

calculating the reciprocal of the sum of a plurality of test metrics associated with the second set of test cases,

wherein the ratio also includes the reciprocal of the sum of the plurality of test metrics associated with the first set of test cases and the reciprocal of the sum of the plurality of test metrics associated with the second set of test cases.

14. The method of claim 13 wherein the evaluating of the energy selection logic further includes:

multiplying the ratio by a term that provides a bias for determining that the third set of test cases is to be generated based on one of the first set and the second set of test cases.

15. The method of claim 14 wherein the magnitude of the term that provides the bias for determining that the third set of test cases is to be generated based on one of the first set and the second set of test cases is less than one.

16. The method of claim 1, further comprising:

automatically testing the web service a first time using the third set of test cases;

automatically generating third learning data regarding the testing performed the first time using the third set of test cases;

automatically testing the web service a second time using the third set of test cases;

automatically generating fourth learning data regarding the testing performed the second time using the third set of test cases;

automatically determining at least in part based on the third and the fourth learning data whether a fourth set of test cases is to be generated; and

responsive to determining that the fourth set of test cases is to be generated, automatically generating the fourth set of test cases.

17. The method of claim 16, further comprising:

calculating the sum of a plurality of test metrics included in the third learning data;

calculating the sum of a plurality of test metrics included in the fourth learning data; and

comparing the sum of the plurality of test metrics included in the third learning data to the sum of the plurality of test metrics included in the fourth learning data,

wherein a result of the comparing is used to determine whether the fourth set of test cases is to be generated.

18. The method of claim 16, further comprising:

calculating the sum of a plurality of test metrics included in the fourth learning data; and

calculating the variance of the sum of the plurality of test metrics included in the fourth learning data,

wherein the variance is used to determine whether the fourth set of test cases is to be generated.

19. The method of claim 1, further comprising:

configuring an endpoint used to communicate with the web service,

wherein the web service metadata is obtained from the web service via endpoint.

20. The method of claim 19, further comprising:

receiving an indication of a Uniform Resource Locator (URL) associated with the web service,

wherein the endpoint is configured using the URL.

21. The method of claim 19, further comprising:

receiving an indication of a data communications protocol for communicating with the web service,

wherein the endpoint is configured using the indication of the data communications protocol.

22. A system that tests a web service, the system comprising:

at least one processor;

at least one processor-readable storage medium communicably coupled to the at least one processor, the at least one processor-readable storage medium storing instructions that, when executed by the at least one processor, cause the system to;

obtain web service metadata from the web service;

automatically generate a first set of test cases at least in part based on the web service metadata obtained from the web service;

automatically test the web service using the first set of test cases;

automatically generate first learning data regarding the first set of test cases by associating each test case of the first set of test cases with at least one respective test metric;

automatically generate a second set of test cases at least in part based on the web service metadata obtained from the web service;

automatically test the web service using the second set of test cases;

automatically generate second learning data regarding the second set of test cases by associating each test case of the second set of test cases with at least one respective test metric;

automatically compare the second learning data to the first learning data;

automatically determine, at least in part based on the comparison between the first and the second learning data, that a third set of test cases is to be generated at least in part based on which one of the first or the second set of test cases performs worse relative to the other of the first or the second set of test cases;

responsive to determining that the first set of test cases performs worse relative to the second set of test cases, automatically generate the third set of test cases at least in part based on the first set of test cases, the third set of test cases different from the first set of test cases; and

responsive to determining that the second set of test cases performs worse relative to the first set of test cases, automatically generate the third set of test cases at least in part based on the second set of test cases, the third set of test cases different from the second set of test cases.

23. The system of claim 22 wherein the first set of test cases is automatically generated at least in part based on a name of a function included in the web service metadata obtained from the web service.

24. The system of claim 22 wherein the first set of test cases is automatically generated at least in part based on a name of a variable included in the web service metadata obtained from the web service.

25. The system of claim 22 wherein the first set of test cases is automatically generated at least in part based on a data type of a variable included in the web service metadata obtained from the web service.

26. The system of claim 22 wherein the first set of test cases is automatically generated at least in part based on at least one constraint on a variable included in the web service metadata obtained from the web service.

27. The system of claim 26 wherein the at least one constraint on the variable includes at least one of:

a minimum value, a maximum value, and a string length.

28. The system of claim 22 wherein the instructions, when executed by the at least one processor, cause the system to:

select one of a plurality of predetermined sets of test parameters based on the web service metadata; and

select a plurality of test parameters from the selected one of the plurality of predetermined sets of test parameters,

wherein the first set of test cases is automatically generated using the selected plurality of test parameters.

29. The system of claim 28 wherein the instructions, when executed by the at least one processor, cause the system to:

generate a plurality of random numbers,

wherein the plurality of test parameters are selected from the selected one of the plurality of predetermined sets of test parameters using the plurality of random numbers.

30. The system of claim 22 wherein the first learning data is generated by associating each test case of the first set of test cases with at least one respective test metric, and the second learning data is generated by associating each test case of the second set of test cases with at least one respective test metric, and the third set of test cases is generated at least in part based on the first or the second learning data.

31. The system of claim 22 wherein the first learning data is generated by associating each test case of the first set of test cases with at least one respective first code coverage metric and at least one respective first execution time metric, the second learning data is generated by associating each test case of the second set of test cases with at least one respective second code coverage metric and at least one respective second execution time metric, and the third set of test cases is generated at least in part based on the first or the second learning data.

32. The system of claim 22 wherein the instructions, when executed by the at least one processor, cause the system to:

evaluate energy selection logic using the first and the second learning data,

wherein a result of evaluating the energy selection logic is used to the determine whether the third set of test cases is to be generated using the first or the second set of test cases.

33. The system of claim 32 wherein the instructions, when executed by the at least one processor, cause the system to:

calculate the sum of a plurality of test metrics associated with the first set of test cases;

calculate the sum of a plurality of test metrics associated with the second set of test cases; and

calculate a ratio that includes the sum of the plurality of test metrics associated with the first set of test cases and the sum of the plurality of test metrics associated with the second set of test cases,

wherein the energy selection logic is evaluated using the calculated ratio.

34. The system of claim 33 wherein the instructions, when executed by the at least one processor, cause the system to:

calculate the reciprocal of the sum of a plurality of test metrics associated with the first set of test cases; and

calculate the reciprocal of the sum of a plurality of test metrics associated with the second set of test cases,

wherein the ratio also includes the reciprocal of the sum of the plurality of test metrics associated with the first set of test cases and the reciprocal of the sum of the plurality of test metrics associated with the second set of test cases.

35. The system of claim 33 wherein the instructions, when executed by the at least one processor, cause the system to:

multiply the ratio by a term that provides a bias for determining that the third set of test cases is to be generated based on one of the first set and the second set of test cases.

36. The system of claim 35 wherein the magnitude of the term that provides the bias for determining that the third set of test cases is to be generated based on one of the first set and the second set of test cases is less than one.

37. The system of claim 22 wherein the instructions, when executed by the at least one processor, cause the system to:

automatically test the web service a first time using the third set of test cases;

automatically generate third learning data regarding the testing performed the first time using the third set of test cases;

automatically test the web service a second time using the third set of test cases;

automatically generate fourth learning data regarding the testing performed the second time using the third set of test cases;

automatically determine at least in part based on the third and the fourth learning data whether a fourth set of test cases is to be generated; and

responsive to determining that the fourth set of test cases is to be generated, automatically generate the fourth set of test cases.

38. The system of claim 37 wherein the instructions, when executed by the at least one processor, cause the system to:

calculate the sum of a plurality of test metrics included in the third learning data;

calculate the sum of a plurality of test metrics included in the fourth learning data; and

compare the sum of the plurality of test metrics included in the third learning data to the sum of the plurality of test metrics included in the fourth learning data,

wherein a result of comparing the sum of the plurality of test metrics included in the third learning data to the sum of the plurality of test metrics included in the fourth learning data is used to determine whether the fourth set of test cases is to be generated.

39. The system of claim 37 wherein the instructions, when executed by the at least one processor, cause the system to:

calculate the sum of a plurality of test metrics included in the fourth learning data; and

calculate the variance of the sum of the plurality of test metrics included in the fourth learning data,

wherein the variance is used to determine whether the fourth set of test cases is to be generated.

40. The system of claim 22 wherein the instructions, when executed by the at least one processor, cause the system to:

configure an endpoint used to communicate with the web service,

wherein the web service metadata is obtained from the web service via endpoint.

41. The system of claim 40 wherein the instructions, when executed by the at least one processor, cause the system to:

receive an indication of a Uniform Resource Locator (URL) associated with the web service,

wherein the endpoint is configured using the URL.

42. The system of claim 40 wherein the instructions, when executed by the at least one processor, cause the system to:

receive an indication of a data communications protocol for communicating with the web service,

wherein the endpoint is configured using the indication of the data communications protocol.