Automated test case generation for applications

• US 9,268,672 B1
• Filed: 05/27/2014
• Issued: 02/23/2016
• Est. Priority Date: 05/27/2014
• Status: Expired due to Fees

• Current Assignee: Dean & Company Strategy Consultants, LLC.
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Gupta, Suresh Chandra
• Primary Examiner(s): Nguyen, Phillip H
• Application Number:
  US14/287,902
• Time in Patent Office: 637 Days
• Field of Search: 717/125, 717/124
• US Class Current: 1/1
• CPC Class Codes: G06F 11/3664 Environments for testing or debugging software
• CPC Class Codes: G06F 11/3684 for test design, e.g. generating new test cases
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites

Abstract

Some implementations include receiving an application binary file for an application to be tested. One or more static analysis operations may be performed on the application binary file to identify application parameters. In some cases, keywords may be associated with individual application parameters, and the keywords may be used to query a test case repository in order to identify test cases. The identified test cases may be used to generate a test plan, and at least a portion of the test plan may be automatically executed in some cases. A test report may be generated that includes a list of test case failures and potential solutions, and the test report may be sent e.g., to a third-party developer or an approval engineer for review.

Claims

1. A system comprising:

one or more processors;

one or more computer-readable media coupled to the one or more processors;

an analysis module maintained on the one or more computer-readable media and executed on the one or more processors to;

receive, for an application to be tested, an application binary file from an electronic device;

perform one or more static analysis operations on the application binary file to determine one or more application parameters, wherein the application parameters represent one or more features that are supported or used by the application, wherein the one or more features include at least one of a camera feature, a microphone feature, a maps feature, an accelerometer feature, a performance feature, or an in-application purchase feature;

determine one or more keywords that are associated with individual application parameters of the one or more application parameters; and

determine one or more test cases based on the one or more keywords; and

a test module maintained on the one or more computer-readable media and executed on the one or more processors to;

receive the one or more test cases;

generate a test plan based on the one or more test cases;

execute at least a portion of the test plan;

generate a test report that includes (i) a list of test cases that failed during execution and (ii) one or more potential solutions associated with individual test cases that failed; and

send the test report to the electronic device.

2. The system as recited in claim 1, wherein performing the one or more static analysis operations on the application binary file includes determining at least one of:

a version of an operating system of the electronic device, one or more software development kit (SDK) versions, one or more user permissions, one or more layout objects, or one or more application programming interfaces (APIs).

3. The system as recited in claim 1, wherein the test module is further executed on the one or more processors to:

receive, via a test plan generation user interface, information associated with at least one of;

one or more devices upon which the application is to be tested;

individual features of the one or more features to be tested;

or

one or more objects to be tested, wherein the one or more objects include at least one of a timer object, a button object, a zoom-in object, a zoom-out object, a scrolling object, a text box object, or an Internet object; and

receiving, via the test plan generation user interface, an indication to generate the test plan based at least in part on the information.

4. The system as recited in claim 1, wherein generating the test report includes:

generating a summary of a failed test case;

generating a list of steps to reproduce the failed test case;

generating an observed result of the failed test case; and

generating an expected result of the failed test case.

5. The system as recited in claim 4, wherein generating the test report further includes:

determining recommendation information that includes a possible cause of the failed test case;

determining software code that is usable by a developer to address the failed test case; and

sending the recommendation information and the sample code to the electronic device for display to the developer.

6. A method comprising:

under control of one or more processors,

receiving, for an application to be tested, an application binary file from an electronic device;

performing one or more static analysis operations on the application binary file to determine one or more application parameters;

determining one or more keywords that are associated with individual application parameters of the one or more application parameters;

determining one or more test cases based on the one or more keywords; and

generating a test plan based on the one or more test cases.

7. The method as recited in claim 6, wherein the one or more application parameters include at least one of:

a user permission to be used by the application;

an application programming interface (API) to be used by the application;

an API version associated with the API to be used by the application;

a software development kit (SDK) version;

a hardware feature of the electronic device to be used by the application;

a software feature of the electronic device to be used by the application;

a layout object to be used by the application;

or

style details to be used by the application.

8. The method as recited in claim 6, further comprising:

executing at least one test case of the one or more test cases; and

generating a test report that includes results associated with execution of the at least one test case.

9. The method as recited in claim 8, wherein the test report identifies a problem encountered during execution of the at least one test case and information regarding a potential solution to the problem.

10. The method as recited in claim 9, wherein the information includes software code usable to address the problem.

11. The method as recited in claim 6, wherein the one or more test cases include at least one test case that is to be executed manually on the electronic device.

12. The method as recited in claim 11, further comprising:

receiving, via a network, information from the electronic device that includes a result of executing the at least one test case; and

storing the result in a test case repository.

13. The method of claim 6, further comprising:

receiving, from the electronic device, information associated with an additional test case to be performed, wherein the electronic device includes a first type of electronic device and wherein the additional test case is to be performed on a second type of electronic device; and

updating the test plan to include the additional test case.

14. The method of claim 6, further comprising sending the test plan to the electronic device, wherein the test plan identifies an expected behavior associated with execution of the one or more test cases.

15. One or more non-transitory computer-readable media maintaining instructions executable by one or more processors to perform operations comprising:

receiving, for an application to be tested, an application binary file;

performing one or more static analysis operations on the application binary file to determine one or more application parameters;

determining one or more keywords that are associated with individual application parameters of the one or more application parameters;

querying a test case repository to identify one or more test cases that are associated with the one or more keywords;

executing, without user input, at least one test case of the one or more test cases; and

generating a test report that includes results associated with execution of the at least one test case.

16. The one or more non-transitory computer-readable media as recited in claim 15, wherein:

the test report includes information associated with a problem encountered during execution of the at least one test case; and

the test report includes information regarding one or more potential solutions to the problem.

17. The one or more non-transitory computer-readable media as recited in claim 16, wherein the information includes software code usable to address the problem.

18. The one or more non-transitory computer-readable media as recited in claim 17, wherein:

the software code is generated by a tester associated with an application store; and

the software code is usable by a third-party developer to address the problem prior to re-submission of an updated application binary file to the application store for testing.

19. The one or more non-transitory computer-readable media as recited in claim 15, the operations further comprising providing a selectable option to delete a particular test case from the one or more test cases.

20. The one or more non-transitory computer-readable media as recited in claim 19, the operations further comprising providing a second selectable option to save a test plan that has been updated to remove the particular test case.