System and method for coordinating field user testing results for a mobile application across various mobile devices

• US 9,367,436 B2
• Filed: 07/01/2015
• Issued: 06/14/2016
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Apkudo, Inc.
• Original Assignee: Apkudo, LLC
• Inventors: Matthews, Joshua Scott, Teitelbaum, David Michael
• Primary Examiner(s): Butler, Sarai
• Application Number:
  US14/789,775
• Publication Number:
  US 20150301929A1
• Time in Patent Office: 349 Days
• Field of Search: 714/38.1
• US Class Current: 1/1
• CPC Class Codes: G06F 11/2294 by remote test
• CPC Class Codes: G06F 11/3684 for test design, e.g. generating new test cases
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 11/3692 for test results analysis

Abstract

Systems and methods for facilitating field testing of a test application are provided. In certain implementations, one or more metrics related to execution, at a mobile device, of one or more operations of the test application may be obtained. A determination of whether an error occurred with an operation of the one or more operations may be effectuated based on the one or more metrics. Error information relating to the error may be caused to be transmitted to one or more other mobile devices, wherein the error information includes information for replicating the error. Replication information relating to an attempt by the first other mobile device to replicate the error may be received back from at least a first other mobile device of the one or more other mobile devices. A determination of whether the first other mobile device replicated the error may be effectuated based on the replication information.

Claims

1. A system for performing field testing of a test application, the system comprising:

a computer system comprising one or more physical processors programmed with computer program instructions which, when executed by the one or more physical processors, cause the computer system to;

obtain one or more metrics related to execution, at a mobile device, of one or more operations of the test application;

determine, based on the one or more metrics, whether an error occurred with an operation of the one or more operations;

cause error information relating to the error to be transmitted to one or more other mobile devices responsive to a determination that the error occurred with the operation, wherein the error information includes information for replicating the error;

receive, back from at least a first other mobile device of the one or more other mobile devices, replication information relating to an attempt by the first other mobile device to replicate the error; and

determine whether the first other mobile device replicated the error based on the replication information.

2. The system of claim 1, further comprising:

non-transitory electronic storage media configured to store one or more test scripts relating to the test application, an individual test script comprising the one or more operations to be performed in a set order.

3. The system of claim 2, wherein the computer system is further caused to:

determine that the error occurred with the operation based on a comparison of the one or more metrics and expected results associated with the operation.

4. The system of claim 1, further comprising:

non-transitory electronic storage media configured to store one or more events, an individual event comprising information indicating that an error occurred with one or more of the mobile device or the test application.

5. The system of claim 4, the computer system is further caused to:

determine that the error occurred with the operation based on a comparison of the one or more metrics and the stored one or more events.

6. The system of claim 1, wherein the computer system is further caused to:

transmit, at one or more pre-determined intervals, the one or more metrics to a computing device remote from the mobile device and the one or more other mobile devices.

7. The system of claim 1, wherein the computer system is further caused to:

transmit the error information to the computing device.

8. The system of claim 1, wherein the computer system is further caused to:

obtain at least a first attribute of the first other mobile device responsive to a determination that the first other mobile device replicated the error;

determine that the mobile device shares the first attribute in common with the first other mobile device; and

identify the first attribute as being a possible cause of the error.

9. The system of claim 8, wherein the first attribute comprises at least one of a hardware component, an operating system, or an operating system version.

10. The system of claim 1, wherein the error information includes a link to a test script, wherein the test script is used to automatically attempt to replicate the error.

11. The system of claim 1, wherein the error information includes one or more instructions that causes the first other mobile device to prompt an end user to perform the operation at the first other mobile device to attempt to replicate the error.

12. The system of claim 1, wherein the computer system is further caused to:

receive, from at least a second other mobile device, second error information relating to a second error at the second other mobile device;

determine at least a second operation to execute to attempt to replicate the second error;

cause the second operation to be executed at the mobile device;

obtain one or more second metrics related to the execution of the second operation; and

determine whether the second error was replicated based on the one or more second metrics.

13. The system of claim 12, wherein the computer system is further caused to:

cause second replication information that indicates that the second error was replicated to be transmitted to a device remote from the mobile device responsive to a determination that the second error was replicated.

14. The system of claim 13, wherein the device remote from the mobile device comprises at least one of the one or more other mobile devices or a computing device remote from the mobile device and the one or more other mobile devices.

15. The system of claim 1, wherein the computer system is further caused to:

identify the one or more other mobile devices to which to transmit the error information based on a list of one or more mobile devices stored at the non-transitory electronic storage media.

16. The system of claim 1, wherein the computer system is further caused to:

cause a request for a list of one or more mobile devices to which the error information should be provided to be transmitted to a computing device remote from the mobile device and the one or more other mobile devices;

obtain the list of one or more mobile devices based on the request; and

identify the one or more other mobile devices to which to transmit the error information based on the list of one or more other mobile devices.

17. The system of claim 1, wherein to cause the error information relating to the error to be transmitted to the one or more other mobile devices, the computer system is further caused to:

cause the error information to be transmitted to a multicast socket of a computing device remote from the mobile device and the one or more other mobile devices.

18. The system of claim 1, wherein the computer system comprises the mobile device, the mobile device comprises the one or more physical processors, and the computer program instructions are part of an operating system of the mobile device.

19. A method of performing field testing of a test application, the method being implemented by a computer system comprising one or more physical processors executing computer program instructions which, when executed by the one or more physical processors, perform the method, the method comprising:

obtaining, by the computer system, one or more metrics related to execution, at a mobile device, of one or more operations of the test application;

determining, by the computer system, based on the one or more metrics, whether an error occurred with an operation of the one or more operations;

causing, by the computer system, error information relating to the error to be transmitted to one or more other mobile devices responsive to a determination that the error occurred with the operation, wherein the error information includes information for replicating the error;

receiving, by the computer system, back from at least a first other mobile device of the one or more other mobile devices, replication information relating to an attempt by the first other mobile device to replicate the error; and

determining, by the computer system, whether the first other mobile device replicated the error based on the replication information.

20. The method of claim 19, further comprising:

storing, by the computer system, at non-transitory electronic storage media of the computer system, one or more test scripts relating to the test application, an individual test script comprising one or more operations to be performed in a set order.

21. The method of claim 20, further comprising:

determining, by the computer system, that the error occurred with the operation based on a comparison of the one or more metrics and expected results associated with the operation.

22. The method of claim 19, further comprising:

storing, by the computer system, at non-transitory electronic storage media of the computer system, one or more events, an individual event comprising information indicating that an error occurred with one or more of the mobile device or the test application.

23. The method of claim 22, further comprising:

determining, by the computer system, that the error occurred with the operation based on a comparison of the one or more metrics and the stored one or more events.

24. The method of claim 19, further comprising:

transmitting, by the computer system, at one or more pre-determined intervals, the one or more metrics to a computing device remote from the mobile device and the one or more other mobile devices.

25. The method of claim 19, further comprising:

transmitting, by the computer system, the error information to the computing device.

26. The method of claim 19, further comprising:

obtaining, by the computer system, at least a first attribute of the first other mobile device responsive to a determination that the first other mobile device replicated the error;

determining, by the computer system, that the mobile device shares the first attribute in common with the first other mobile device; and

identifying, by the computer system, the first attribute as being a possible cause of the error.

27. The method of claim 26, wherein the first attribute comprises at least one of a hardware component, an operating system, or an operating system version.

28. The method of claim 19, further comprising:

receiving, by the computer system, from at least a second other mobile device, second error information relating to a second error at the second other mobile device;

determining, by the computer system, at least a second operation to execute to attempt to replicate the second error;

causing, by the computer system, the second operation to be executed at the mobile device;

obtaining, by the computer system, one or more second metrics related to the execution of the second operation; and

determining, by the computer system, whether the second error was replicated based on the one or more second metrics.

29. The method of claim 19, wherein the computer system comprises the mobile device, the mobile device comprises the one or more physical processors, and the computer program instructions are part of an operating system of the mobile device.

30. A non-transitory electronic storage media storing computer program instructions for performing field testing of a test application, the computer program instructions, when executed by a computer system, causes computer system to:

obtain one or more metrics related to execution, at a mobile device, of one or more operations of the test application;

determine, based on the one or more metrics, whether an error occurred with an operation of the one or more operations;

cause error information relating to the error to be transmitted to one or more other mobile devices responsive to a determination that the error occurred with the operation, wherein the error information includes information for replicating the error;

receive, back from at least a first other mobile device of the one or more other mobile devices, replication information relating to an attempt by the first other mobile device to replicate the error; and

determine whether the first other mobile device replicated the error based on the replication information.