Automatically running tests against WEB APIs based on specifications

• US 10,409,711 B2
• Filed: 06/12/2017
• Issued: 09/10/2019
• Est. Priority Date: 06/12/2017
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Dolby, Julian Timothy, Laredo, Jim Alain, Slominski, Aleksander, Wittern, John Erik, Ying, Annie T., Young, Christopher, Zheng, Yunhui
• Primary Examiner(s): Dao, Thuy
• Assistant Examiner(s): Wei, Zheng
• Application Number:
  US15/620,769
• Publication Number:
  US 20180357154A1
• Time in Patent Office: 820 Days
• CPC Class Codes: G06F 11/3664 Environments for testing or debugging software
• CPC Class Codes: G06F 11/3684 for test design, e.g. generating new test cases
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 11/3692 for test results analysis
• CPC Class Codes: G06F 2201/81 Threshold

Abstract

A method and system of determining whether a specification is an accurate representation of an application program interface (API) is provided. The specification is received electronically over a network. Service calls to be tested are identified based on the specification. A test case is created for each of the identified service calls. A sequence is created for the test cases. A test plan is generated based on the created sequence. The generated test plan is executed. Upon identifying an error in response to the executed test plan, a notification is generated, indicating that the specification is not an accurate representation of the API.

Claims

1. A computing device comprising:

a processor;

a network interface coupled to the processor to enable communication over a network;

a storage device coupled to the processor;

a compliance engine software stored in the storage device, wherein an execution of the software by the processor configures the computing device to perform acts comprising;

receiving an electronic specification defining an application program interface (API), over the network;

identifying service calls to be tested based on the specification;

creating a test case for each of the identified service calls;

creating a sequence for the created test cases, for an efficient correlation between the specification and the API, wherein the sequence for the created test cases comprises;

automatically identifying one or more Hypertext Transfer Protocol (HTTP) invocations comprising at least one of;

POST, GET, PUT, PATCH, or DELETE invocation;

automatically ordering any identified POST invocation before any GET, PUT, PATCH, and DELETE invocation; and

automatically ordering any DELETE invocation after any POST, GET, PUT, and PATCH invocation;

generating a test plan based on the created sequence;

executing the generated test plan; and

upon identifying an error in response to the executed test plan, generating a notification indicating that the specification is not an accurate representation of the API.

2. The computing device of claim 1, wherein identifying service calls to be tested comprises parsing a text of the specification to identify Web service calls and one or more input parameters associated with each Web service call.

3. The computing device of claim 1, wherein creating a test case for each of the identified service calls comprises identifying one or more threshold criteria for an error.

4. The computing device of claim 3, wherein the one or more threshold criteria comprises at least one of:

(i) receiving a Hypertext Transfer Protocol (HTTP) error code over the network, or (ii) a discrepancy between an expected data and a received data, in response to the execution of the test case.

5. The computing device of claim 1, wherein generating the test plan comprises creating an executable source code.

6. The computing device of claim 1, wherein generating the test plan comprises translating the sequence of the created test cases in a format that a Web service testing application understands.

7. The computing device of claim 1, wherein executing the generated test plan comprises, for each test case:

sending a Hypertext Transfer Protocol (HTTP) requests to the API;

receiving a response to the HTTP request; and

storing the response in the storage device.

8. The computing device of claim 1, wherein execution of the compliance engine by the processor further configures the computing device to perform an act comprising:

upon identifying no errors in response to the executed test plan, certifying that the specification is an accurate representation of the API.

9. The computing device of claim 1, wherein execution of the compliance engine by the processor further configures the computing device to perform acts comprising, upon identifying an error in response to the executed test plan:

identifying a portion of the specification related to the error; and

indicating the portion of the specification in the generated notification.

10. The computing device of claim 9, wherein execution of the compliance engine by the processor further configures the computing device to perform acts comprising, upon identifying an error in response to the executed test plan, sending the notification to a developer of the API.

11. A non-transitory computer readable storage medium tangibly embodying a computer readable program code having computer readable instructions that, when executed, causes a computer device to carry out a method of determining whether a specification is an accurate representation of an application program interface (API), the method comprising:

receiving the specification electronically, over a network;

identifying service calls to be tested based on the specification;

creating a test case for each of the identified service calls;

creating a sequence for the created test cases, for an efficient correlation between the specification and the API, wherein the sequence for the created test cases comprises;

automatically identifying one or more Hypertext Transfer Protocol (HTTP) invocations comprising at least one of;

POST, GET, PUT, PATCH, or DELETE invocation;

automatically ordering any identified POST invocation before any GET, PUT, PATCH, and DELETE invocation; and

automatically ordering any DELETE invocation after any POST, GET, PUT, and PATCH invocation;

generating a test plan based on the created sequence;

executing the generated test plan; and

upon identifying an error in response to the executed test plan, generating a notification indicating that the specification is not an accurate representation of the API.

12. The non-transitory computer readable storage medium of claim 11, wherein identifying service calls to be tested comprises parsing a text of the specification to identify Web service calls and one or more input parameters associated with each Web service call.

13. The non-transitory computer readable storage medium of claim 11, wherein creating a test case for each of the identified service calls comprises identifying one or more threshold criteria for an error.

14. The non-transitory computer readable storage medium of claim 13, wherein the one or more threshold criteria comprises at least one of:

(i) receiving a Hypertext Transfer Protocol (HTTP) error code over the network, or (ii) a discrepancy between an expected data and a received data, in response to the execution of the test case.

15. The non-transitory computer readable storage medium of claim 11, wherein generating the test plan comprises creating an executable source code.

16. The non-transitory computer readable storage medium of claim 11, wherein executing the generated test plan comprises, for each test case:

sending a Hypertext Transfer Protocol (HTTP) requests to the API;

receiving a response to the HTTP request; and

storing the response in the storage device.

17. The non-transitory computer readable storage medium of claim 11, further comprising, upon identifying no errors in response to the executed test plan, certifying that the specification is an accurate representation of the API.

18. The non-transitory computer readable storage medium of claim 11, further comprising, upon identifying an error in response to the executed test plan:

identifying a portion of the specification related to the error; and

indicating the portion of the specification in the generated notification.