Systematically exploring programs during testing

• US 9,679,090 B1
• Filed: 10/27/2014
• Issued: 06/13/2017
• Est. Priority Date: 10/27/2014
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Marolia, Faizaan Kersi
• Primary Examiner(s): Vu, Tuan
• Application Number:
  US14/524,972
• Time in Patent Office: 960 Days
• CPC Class Codes: G06F 11/34 Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 11/3692 for test results analysis
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 30/20 Design optimisation, verification or simulation optimisation, verification or simulation of circuit designs G06F30/30

Abstract

A testing service is used to test the operation of a program. The testing service systematically explores the display screens generated by the program. The testing service also identifies the elements that are displayed on the screen and programmatically interacts with the elements. Test data including performance metrics and screen captures may be recorded during the testing. Test results associated with the testing of the program are provided to an authorized user.

Claims

1. A non-transitory computer-readable storage medium having computer-executable instructions stored thereupon which, when executed by a computer, cause the computer to:

receive a request at a testing service to test operation of a program;

programmatically perform a first event that simulates a first user interaction on a first screen display associated with the program, wherein the first screen display includes a first element;

determine that a graphical change occurred to the first screen display based at least in part on programmatically performing the first event;

programmatically identify a type of the first element based at least in part on the graphical change to the first screen display, wherein programmatically identifying includes one or more of a graphical analysis of a shape of the first element, or a response caused by the performance of the first event;

determine one or more tests to perform based at least in part on the type of the first element;

cause the first element on the first screen display to be tested using the one or more tests;

programmatically perform a second event that simulates a second user interaction on the first screen display associated with the program;

programmatically identify that the first screen display changed to a second screen display of the program caused by the second event, wherein the second screen display includes a second element;

programmatically identify a type of the second element based at least in part on a shape of the second element;

determine one or more second tests to perform based at least in part on the type of the second element;

cause the second element to be tested using the one or more second tests;

store test results associated with the testing; and

provide the test results in response to the request.

2. The non-transitory computer-readable storage medium of claim 1, wherein the first element on the first screen display and the second element on the second screen display are tested by programmatically performing one or more additional events that simulate one or more further user interactions.

3. The non-transitory computer-readable storage medium of claim 2, wherein programmatically performing the one or more additional events comprises causing one or more events to be received by the program, the one or more events including a tap event, a tilt event, a pan event, a text entry event, a sleep event, a landscape mode event, a portrait mode event, or a device setting change event.

4. The non-transitory computer-readable storage medium of claim 1, wherein storing test results comprises capturing screen displays generated during the testing of the program and storing one or more performance metrics during the testing, the performance metrics comprising one or more of a memory usage, a CPU usage, a network traffic usage, a battery usage, or a graphics usage.

5. The non-transitory computer-readable storage medium of claim 1, wherein providing the test results comprises providing a test script that includes data associated with instructions used to test the program.

6. The non-transitory computer-readable storage medium of claim 1, wherein the testing service is implemented within a service provider network.

7. A system for testing a program, the system comprising:

one or more processor-based computing devices configured to

programmatically identify a shape of a first element included in a first display screen associated with an execution of the program;

programmatically identify a type of the first element based, at least in part, on a graphical analysis of the shape of the first element;

based at least in part on programmatically identifying a type of the first element, cause the first element to be programmatically tested;

programmatically identify a shape of a second element included in the first display screen;

programmatically identify a type of the second element based, at least in part, on a graphical analysis of the shape of the second element;

based at least in part on programmatically identifying a type of the second element, cause the second element to be programmatically tested;

store test results associated with a program; and

provide the test results.

8. The system of claim 7, wherein causing the first element to be tested comprises causing an event to be sent to the program that simulates one or more user interactions with the first element.

9. The system of claim 7, further comprising programmatically identifying one or more other elements included in one or more other screen displays by programmatically identifying, without user interaction, the one or more other screen displays of the program presented in response to sending an event to the program that simulates one or more user interactions with the one or more other elements detected on the one or more other screen displays.

10. The system of claim 7, wherein at least one of one or more test devices or the one or more computing devices are further configured to change a device setting of at least one of the one or more test devices.

11. The system of claim 7, wherein recording the test results comprises capturing screen displays generated during the testing and recording one or more performance metrics during the testing, the metrics including one or more of a memory usage, a CPU usage, a network traffic usage, a battery usage, or a graphics usage of the program.

12. The system of claim 7, wherein providing the test results comprises providing at least one of a test script that includes data associated with instructions used to test the program or a screen graph that indicates the one or more screen displays identified.

13. The system of claim 7, wherein the one or more computing devices are further configured to send data to a developer computing device to display the test results in a graphical user interface (GUI).

14. The system of claim 13, wherein the GUI includes a user interface element for selecting the one or more test devices and a user interface element for defining one or more performance metrics to display in the GUI.

15. The system of claim 13, wherein the GUI provides a display of data that shows test events performed during the testing, a display of one or more performance metrics captured during the testing and a display of one or more screen captures generated during the testing.

16. A computer-implemented method for testing the operation of a program, the method implemented with a computer system having at least one processor coupled with computer-executable instructions, which when executed by the processor, perform the method steps, comprising:

programmatically identifying a first element included in a first screen display associated with an execution of the program based, at least in part, on a programmatic analysis of graphical data associated with a first screen display, wherein the programmatic analysis of the graphical data includes identifying a shape of the first element;

programmatically identifying a type of the first element based at least in part on the shape of the first element;

causing the first element to be tested based at least in part on the type of the first element;

programmatically identifying a second element included in the first screen display based, at least in part, on a programmatic analysis of graphical data associated with the first screen display, wherein the programmatic analysis of the graphical data includes identifying a shape of the second element;

programmatically identifying a type of the second element based at least in part on the shape of the second element;

causing the second element to be tested based at least in part on the type of the second element;

storing test results associated with the first element and the second element; and

providing the test results.

17. The computer-implemented method of claim 16, wherein causing the first element to be tested comprises sending an event to the program that simulates one or more user interactions with the first element.

18. The computer-implemented method of claim 17, further comprising changing a device setting of a computing device during the testing.

19. The computer-implemented method of claim 16, further comprising providing a graphical user interface (GUI) to display the test results, wherein the GUI provides a graphical display of one or more performance metrics and a display of one or more of screen captures.

20. The computer-implemented method of claim 19, wherein the GUI comprises a user interface element for defining one or more performance metrics to display in the GUI and a user interface element for navigating to a next screen capture or a previous screen capture generated during the testing of the program.