System and method for facilitating field testing of a test application

• US 10,452,527 B2
• Filed: 12/18/2017
• Issued: 10/22/2019
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Apkudo, Inc.
• Original Assignee: Apkudo, LLC
• Inventors: Matthews, Joshua Scott, Teitelbaum, David Michael
• Primary Examiner(s): Butler, Sarai E
• Application Number:
  US15/844,760
• Publication Number:
  US 20180137040A1
• Time in Patent Office: 673 Days
• Field of Search: 714 381
• CPC Class Codes: G06F 11/2294 by remote test
• CPC Class Codes: G06F 11/3684 for test design, e.g. generating new test cases
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 11/3692 for test results analysis

Abstract

Systems and methods for facilitating field testing of a test application are provided. In certain implementations, one or more metrics related to execution, at a user device, of one or more operations of the test application may be obtained. A determination of whether an error occurred with an operation of the one or more operations may be effectuated based on the one or more metrics. Error information relating to the error may be caused to be transmitted to one or more other user devices, wherein the error information includes information for replicating the error. Replication information relating to an attempt by the first other user device to replicate the error may be received back from at least a first other user device of the one or more other user devices. A determination of whether the first other user device replicated the error may be effectuated based on the replication information.

Claims

1. A system for facilitating field testing of a test application, the system comprising:

a computer system comprising one or more physical processors programmed with computer program instructions which, when executed by the one or more physical processors, cause the computer system to;

obtain one or more metrics related to execution, at a device, of one or more operations of the test application;

determine, based on the one or more metrics, whether an error occurred with an operation of the one or more operations;

cause error information relating to the error to be transmitted to one or more other devices responsive to a determination that the error occurred with the operation;

receive, back from at least a first other device of the one or more other devices, replication information relating to an attempt by the first other device to replicate the error; and

determine whether the first other device replicated the error based on the replication information.

2. The system of claim 1, further comprising:

non-transitory electronic storage media configured to store one or more test scripts relating to the test application, an individual test script comprising the one or more operations to be performed in a set order.

3. The system of claim 2, wherein the computer system is further caused to:

determine that the error occurred with the operation based on a comparison of the one or more metrics and expected results associated with the operation.

4. The system of claim 1, further comprising:

non-transitory electronic storage media configured to store one or more events, an individual event comprising information indicating that an error occurred with one or more of the device or the test application.

5. The system of claim 4, the computer system is further caused to:

determine that the error occurred with the operation based on a comparison of the one or more metrics and the stored one or more events.

6. The system of claim 1, wherein the computer system is further caused to:

transmit, at one or more pre-determined intervals, the one or more metrics to a computing device remote from the device and the one or more other devices.

7. The system of claim 1, wherein the computer system is further caused to:

transmit the error information to the computing device.

8. The system of claim 1, wherein the computer system is further caused to:

obtain at least a first attribute of the first other device responsive to a determination that the first other device replicated the error;

determine that the device shares the first attribute in common with the first other device; and

identify the first attribute as being a possible cause of the error.

9. The system of claim 8, wherein the first attribute comprises at least one of a hardware component, an operating system, or an operating system version.

10. The system of claim 1, wherein the error information includes a link to a test script, wherein the test script is used to automatically attempt to replicate the error.

11. The system of claim 1, wherein the error information includes one or more instructions that causes the first other device to prompt an end user to perform the operation at the first other device to attempt to replicate the error.

12. The system of claim 1, wherein the computer system is further caused to:

receive, from at least a second other device, second error information relating to a second error at the second user device;

determine at least a second operation to execute to attempt to replicate the second error;

cause the second operation to be executed at the device;

obtain one or more second metrics related to the execution of the second operation; and

determine whether the second error was replicated based on the one or more second metrics.

13. The system of claim 12, wherein the computer system is further caused to:

cause second replication information that indicates that the second error was replicated to be transmitted to at least one device remote from the device responsive to a determination that the second error was replicated.

14. The system of claim 13, wherein the at least one device remote from the device comprises at least one of the one or more other devices or a computing device remote from the device and the one or more other devices.

15. The system of claim 1, wherein the computer system is further caused to:

identify the one or more other devices to which to transmit the error information based on a list of one or more devices stored at non-transitory electronic storage media of the system.

16. The system of claim 1, wherein the computer system is further caused to:

cause a request for a list of one or more devices to which the error information should be provided to be transmitted to a computing device remote from the device and the one or more other devices;

obtain the list of one or more devices based on the request; and

identify the one or more other devices to which to transmit the error information based on the list of one or more other devices.

17. The system of claim 1, wherein to cause the error information relating to the error to be transmitted to the one or more other devices, the computer system is further caused to:

cause the error information to be transmitted to a multicast socket of a computing device remote from the device and the one or more other devices.

18. The system of claim 1, wherein the computer system comprises the device, the device comprises the one or more physical processors, and the computer program instructions are part of an operating system of the device.

19. A method of facilitating field testing of a test application, the method being implemented by one or more physical processors executing computer program instructions which, when executed by the one or more physical processors, perform the method, the method comprising:

obtaining, by the one or more physical processors, one or more metrics related to execution, at a device, of one or more operations of the test application;

determining, by the one or more physical processors, based on the one or more metrics, whether an error occurred with an operation of the one or more operations;

causing, by the one or more physical processors, error information relating to the error to be transmitted to one or more other devices responsive to a determination that the error occurred with the operation;

receiving, by the one or more physical processors, back from at least a first other device of the one or more other devices, replication information relating to an attempt by the first other device to replicate the error; and

determining, by the one or more physical processors, whether the first other device replicated the error based on the replication information.

20. A non-transitory electronic storage media storing computer program instructions for facilitating field testing of a test application, the computer program instructions, when executed by one or more physical processors, causes the one or more physical processors to:

obtain one or more metrics related to execution, at a device, of one or more operations of the test application;

determine, based on the one or more metrics, whether an error occurred with an operation of the one or more operations;

cause error information relating to the error to be transmitted to one or more other devices responsive to a determination that the error occurred with the operation;

receive, back from at least a first other device of the one or more other devices, replication information relating to an attempt by the first other device to replicate the error; and

determine whether the first other device replicated the error based on the replication information.