System, method, and apparatus for crowd-sourced gathering of application execution events for automatic application testing and replay

• US 10,417,116 B2
• Filed: 07/28/2016
• Issued: 09/17/2019
• Est. Priority Date: 07/28/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Horesh, Lior, Horesh, Raya, Pistoia, Marco, Tripp, Omer
• Primary Examiner(s): Kang, Insun
• Application Number:
  US15/222,067
• Publication Number:
  US 20180032424A1
• Time in Patent Office: 1,146 Days
• Field of Search: None
• CPC Class Codes: G06F 11/302 where the computing system component is a software system
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring
• CPC Class Codes: G06F 11/3644 by instrumenting at runtime
• CPC Class Codes: G06F 11/3684 for test design, e.g. generating new test cases
• CPC Class Codes: G06F 11/3692 for test results analysis
• CPC Class Codes: G06F 2201/865 Monitoring of software
• CPC Class Codes: G06F 8/65 Updates security arrangements therefor G06F21/57

Abstract

A method is disclosed including instrumenting a first version of an application on a plurality of end user devices and receiving execution data for the first version of the application from at least some of the plurality of end user devices. The execution data may be generated by the instrumentation in response to an execution of the first version of the application by the at least some of the end user devices. The method further includes automatically generating execution scripts based on the received execution data. The execution scripts may be configured to reproduce the execution of the first version of the application by the at least some of the end users devices. The method further includes automatically executing at least one of the execution scripts on an updated version of the application.

Claims

1. A method performed by at least one hardware processor, comprising:

instrumenting a first version of an application on a plurality of end user devices associated with end users of the application, wherein the instrumentation comprises injecting instrumentation into executable code of the first version of the application;

receiving execution data for the first version of the application from at least some of the plurality of end user devices, the execution data generated by the at least some of the plurality of end user devices, and the execution data is generated based on an execution of the instrumented executable code of the first version of the application by each of the at least some of the end user devices during real-life use of the application by the end users of the at least some of the end user devices;

automatically generating a plurality of execution scripts based on the received execution data, each execution script configured to reproduce user inputs and user device environment that were used during the real-life execution of the first version of the application by a respective one of the end users devices;

automatically executing at least some of the plurality of execution scripts to use the reproduced user inputs and user device environments on an execution of executable code of an updated version of the application;

monitoring the automatic execution of the at least some of the plurality of execution scripts on the executable code of the updated version of the application to identify an execution point of failure in the execution of the executable code of the first version of the application; and

identifying the execution point of failure from the monitoring by;

instrumenting the updated version of the application;

automatically executing the at least some of the plurality of execution scripts on the instrumented executable code of the updated version of the application to use the reproduced user inputs and user device environments that were used during the automatic execution of the executable code of the updated version of the application;

generating new execution data from the automatic execution of the at least one of the execution scripts on the instrumented executable code of the updated version of the application;

comparing the received execution data from the execution of the instrumented executable code of the first version of the application to the new execution data from the execution of the instrumented executable code of the updated version of the application;

determining whether the execution point of failure occurs in the execution of the executable code of the updated version of the application based on the comparing; and

determining whether the updated version of the application is successful based on whether the execution point of failure occurred in the execution of the executable code of the updated version of the application.

2. The method of claim 1, wherein instrumenting the first version of the application comprises using at least one of a custom class loader and method swizzling.

3. The method of claim 1, wherein the execution data comprises deterministic events.

4. The method of claim 1, wherein the execution data comprises non-deterministic events.

5. The method of claim 1, wherein instrumenting the first version of the application on the plurality of end user devices comprises instrumenting the first version of the application on an arbitrarily large number of end user devices, wherein the arbitrarily large number of end user devices comprises at least five hundred end user devices.

6. A system comprising:

at least one processor; and

a memory storing instructions that, when executed by the at least one processor, cause the at least one processor to;

instrument a first version of an application on a plurality of end user devices associated with end users of the application, wherein the instrumentation comprises injecting instrumentation into executable code of the first version of the application;

receive execution data for the first version of the application from at least some of the plurality of end user devices, the execution data generated by the at least some of the plurality of end user devices, and the execution data is generated based on an execution of the instrumented executable code of the first version of the application by each of the at least some of the end user devices during real-life use of the application by the end users of the at least some of the end user devices;

automatically generate a plurality of execution scripts based on the received execution data, each execution script configured to reproduce user inputs and user device environment that were used during the real-life execution of the first version of the application by a respective one of the end users devices;

automatically execute at least some of the plurality of execution scripts to use the reproduced user inputs and user device environments on an execution of executable code of an updated version of the application;

monitor the automatic execution of the at least some of the plurality of execution scripts on the executable code of the updated version of the application to identify an execution point of failure in the execution of the executable code of the first version of the application; and

identify the execution point of failure from the monitoring by;

instrumenting the updated version of the application;

automatically executing the at least some of the plurality of execution scripts on the instrumented executable code of the updated version of the application to use the reproduced user inputs and user device environments that were used during the automatic execution of the executable code of the updated version of the application;

generating new execution data from the automatic execution of the at least one of the execution scripts on the instrumented executable code of the updated version of the application;

comparing the received execution data from the execution of the instrumented executable code of the first version of the application to the new execution data from the execution of the instrumented executable code of the updated version of the application;

determining whether the execution point of failure occurs in the execution of the executable code of the updated version of the application based on the comparing; and

determining whether the updated version of the application is successful based on whether the execution point of failure occurred in the execution of the executable code of the updated version of the application.

7. The system of claim 6, wherein instrumenting the first version of the application comprises using at least one of a custom class loader and method swizzling.

8. The system of claim 6, wherein the execution data comprises deterministic events.

9. The system of claim 6, wherein the execution data comprises non-deterministic events.

10. An apparatus comprising:

at least one processor; and

memory storing instructions that, when executed by the at least one processor, cause the at least one processor to;

instrument a first version of an application installed on the apparatus, wherein the instrumentation comprises injecting instrumentation into executable code of the first version of the application;

generate first execution data from the first version of the application, the first execution data generated by the instrumentation in response to an execution of the instrumented executable code of the first version of the application by the apparatus;

transmit the first execution data to a server;

receive from the server a plurality of execution scripts, each execution script generated based on respective execution data transmitted to the server from a plurality of end user devices associated with end users of the application, the respective execution data of the plurality of end user devices generated by the plurality of end user devices, and each respective execution data is generated based on an execution of the instrumented executable code of the first version of the application by each of the plurality of end user devices during real-life use of the application by the end users of the at least some of the end user devices, the plurality of execution scripts including an execution script generated based on the first execution data, each execution script configured to reproduce user inputs and user device environment that were used during the real-life execution of the first version of the application by a respective one of the end user devices that generated the execution data;

receive from the server an updated version of the application;

automatically execute at least some of the plurality of execution scripts to use the reproduced user inputs and user device environments on an execution of executable code of the updated version of the application;

monitor the automatic execution of the at least some of the plurality of execution scripts on the executable code of the updated version of the application to identify an execution point of failure in the execution of the executable code of the first version of the application; and

identify the execution point of failure from the monitoring by;

instrumenting the updated version of the application;

automatically executing the at least some of the plurality of execution scripts on the instrumented executable code of the updated version of the application to use the reproduced user inputs and user device environments that were used during the automatic execution of the executable code of the updated version of the application;

generating new execution data from the automatic execution of the at least one of the execution scripts on the instrumented executable code of the updated version of the application;

comparing the received execution data from the execution of the instrumented executable code of the first version of the application to the new execution data from the execution of the instrumented executable code of the updated version of the application;

determining whether the execution point of failure occurs in the execution of the executable code of the updated version of the application based on the comparing; and

determining whether the updated version of the application is successful based on whether the execution point of failure occurred in the execution of the executable code of the updated version of the application.

11. The apparatus of claim 10, wherein instrumenting the first version of the application comprises using at least one of a custom class loader and method swizzling.

12. The apparatus of claim 10, wherein the execution data comprises deterministic events.

13. The apparatus of claim 10, wherein the execution data comprises non-deterministic events.