System and method for coordinating field user testing results for a mobile application across various mobile devices

• US 9,075,781 B2
• Filed: 03/15/2013
• Issued: 07/07/2015
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Apkudo, Inc.
• Original Assignee: Apkudo, LLC
• Inventors: Matthews, Joshua Scott, Teitelbaum, David Michael
• Primary Examiner(s): BUTLER, SARAI E
• Application Number:
  US13/837,851
• Publication Number:
  US 20140281714A1
• Time in Patent Office: 844 Days
• Field of Search: 714/27, 714/37
• US Class Current: 1/1
• CPC Class Codes: G06F 11/2294 by remote test
• CPC Class Codes: G06F 11/3684 for test design, e.g. generating new test cases
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 11/3692 for test results analysis

Abstract

A system and method for coordinating field user testing results for a mobile application across various mobile devices is provided. The system may comprise a plurality of mobile devices configured to facilitate field user testing of a test application, and a computing device configured to provide analysis related to field user testing of the test application. A testing module of the mobile device may be configured to: perform one or more operations of the test application; collect a plurality of metrics related to an execution of the one or more operations of the test application; determine whether an error occurred with an operation of the test application based on the collected one or more metrics; and transmit, to one or more other mobile devices, one or more metrics related to the execution of the operation at which the error occurred. The testing module may be part of the operating system.

Claims

1. A mobile device programmed to perform field testing using a test application, comprising:

one or more physical processors programmed with computer program instructions, the computer program instructions, when executed by the one or more physical processors, program the mobile device to;

execute one or more operations of the test application;

obtain one or more metrics related to the execution of the one or more operations;

determine whether an error occurred with an operation, from among the one or more operations, based on the one or more metrics;

cause error information relating to the error to be transmitted to one or more other mobile devices responsive to a determination that the error occurred with the operation, wherein the error information includes information used to replicate the error;

receive, back from at least a first other mobile device among the one or more other mobile devices, replication information relating to an attempt by the first other mobile device to replicate the error; and

determine whether the first other mobile device replicated the error based on the replication information.

2. The mobile device of claim 1, further comprising:

non-transitory electronic storage media configured to store one or more test scripts relating to the test application, an individual test script comprising the one or more operations to be performed in a set order.

3. The mobile device of claim 2, wherein the mobile device is further programmed to determine that the error occurred with the operation based on a comparison of the one or more metrics and expected results associated with the operation.

4. The mobile device of claim 1, further comprising:

non-transitory electronic storage media configured to store one or more events, an individual event comprising information indicating that an error occurred with one or more of the test application and the mobile device.

5. The mobile device of claim 4, wherein the mobile device is further programmed to determine that the error occurred with the operation based on a comparison of the one or more metrics and the stored one or more events.

6. The mobile device of claim 1, wherein the mobile device is further programmed to:

transmit, at one or more pre-determined intervals, the one or more metrics to a computing device remote from the mobile device and the one or more other mobile devices.

7. The mobile device of claim 1, wherein the mobile device is further programmed to:

transmit the error information to the computing device.

8. The mobile device of claim 1, wherein the mobile device is further programmed to:

obtain at least a first attribute of the first other mobile device responsive to a determination that the first other mobile device replicated the error;

determine that the mobile device shares the first attribute in common with the first other mobile device; and

identify the first attribute as being a possible cause of the error.

9. The mobile device of claim 8, wherein the first attribute comprises at least one of a hardware component, an operating system, or an operating system version.

10. The mobile device of claim 1, wherein the error information includes a link to the test script, wherein the test script is used to automatically attempt to replicate the error.

11. The mobile device of claim 1, wherein the error information includes one or more instructions that causes the first other mobile device to prompt an end user to perform the operation at the first other mobile device to attempt to replicate the error.

12. The mobile device of claim 1, wherein the mobile device is further programmed to:

receive, from at least a second other mobile device, second error information relating to a second error at the second other mobile device;

determine at least a second operation to execute to attempt to replicate the second error;

execute the second operation;

collect one or more second metrics related to the execution of the second operation; and

determine whether the second error was replicated based on the one or more second metrics.

13. The mobile device of claim 12, wherein the mobile device is further programmed to:

cause second replication information that indicates that the second error was replicated to be transmitted to a device remote from the mobile device responsive to a determination that the second error was replicated.

14. The mobile device of claim 13, wherein the device remote from the mobile device comprises at least one of the one or more other mobile devices or a computing device remote from the mobile device and the one or more other mobile devices.

15. The mobile device of claim 1, wherein the mobile device is further programmed to:

identify the one or more other mobile devices to which to transmit the error information based on a list of one or more mobile devices stored at the non-transitory electronic storage media.

16. The mobile device of claim 1, wherein the mobile device is further programmed to:

cause a request for a list of one or more mobile devices to which the error information should be provided to be transmitted to a computing device remote from the mobile device and the one or more other mobile devices;

obtain the list of one or more mobile devices based on the request; and

identify the one or more other mobile devices to which to transmit the error information based on the list of one or more other mobile devices.

17. The mobile device of claim 1, wherein to cause the error information relating to the error to be transmitted to the one or more other mobile devices, the mobile device is further programmed to:

cause the error information to be transmitted to a multicast socket of a computing device remote from the mobile device and the one or more other mobile devices.

18. The mobile device of claim 1, wherein the computer program instructions is part of an operating system of the mobile device.

19. A method of performing field testing using a test application, the method being implemented in a mobile device comprising one or more physical processors programmed with computer program instructions that, when executed by the one or more physical processors, program the mobile device to perform the method, the method comprising:

executing, by the mobile device, one or more operations of the test application;

obtaining, by the mobile device, one or more metrics related to the execution of the one or more operations;

determining, by the mobile device, whether an error occurred with an operation, from among the one or more operations, based on the one or more metrics;

causing, by the mobile device, error information relating to the error to be transmitted to one or more other mobile devices responsive to a determination that the error occurred with the operation, wherein the error information includes information used to replicate the error;

receiving, by the mobile device back from at least a first other mobile device among the one or more other mobile devices, replication information relating to an attempt by the first other mobile device to replicate the error; and

determining, by the mobile device, whether the first other mobile device replicated the error based on the replication information.

20. The method of claim 19, further comprising:

storing, at non-transitory electronic storage media of the mobile device, one or more test scripts relating to the test application, an individual test script comprising one or more operations to be performed in a set order.

21. The method of claim 20, further comprising:

determining that the error occurred with the operation based on a comparison of the one or more metrics and expected results associated with the operation.

22. The method of claim 19, further comprising:

storing, at non-transitory electronic storage media of the mobile device, one or more events, an individual event comprising information indicating that an error occurred with one or more of the test application and the mobile device.

23. The method of claim 22, further comprising:

determining that the error occurred with the operation based on a comparison of the one or more metrics and the stored one or more events.

24. The method of claim 19, further comprising:

transmitting, at one or more pre-determined intervals, the one or more metrics to a computing device remote from the mobile device and the one or more other mobile devices.

25. The method of claim 19, further comprising:

transmitting the error information to the computing device.

26. A non-transitory electronic storage media storing computer program instructions for performing field testing using a test application, the computer program instructions, when executed at a mobile device, programs the mobile device to:

execute one or more operations of the test application;

obtain one or more metrics related to the execution of the one or more operations;

determine whether an error occurred with an operation, from among the one or more operations, based on the one or more metrics;

cause error information relating to the error to be transmitted to one or more other mobile devices responsive to a determination that the error occurred with the operation, wherein the error information includes information used to replicate the error;

receive, back from at least a first other mobile device among the one or more other mobile devices, replication information relating to an attempt by the first other mobile device to replicate the error; and

determine whether the first other mobile device replicated the error based on the replication information.

27. The storage media of claim 26, wherein the non-transitory media further stores:

one or more test scripts relating to the test application, an individual test script comprising the one or more operations to be performed in a set order.

28. The storage media of claim 27, wherein the computer program instructions, when executed, further program the mobile device to:

determine that the error occurred with the operation based on a comparison of the one or more metrics and expected results associated with the operation.

29. The storage media of claim 26, wherein the non-transitory media further stores:

one or more events, an individual event comprising information indicating that an error occurred with one or more of the test application and the mobile device.

30. The storage media of claim 29, wherein the computer program instructions, when executed, further program the mobile device to:

determine that the error occurred with the operation based on a comparison of the one or more metrics and the stored one or more events.

31. The storage media of claim 26, wherein the computer program instructions when executed, further program the mobile device to:

transmit, at one or more pre-determined intervals, the one or more metrics to a computing device remote from the mobile device and the one or more other mobile devices.

32. The storage media of claim 26, wherein the computer program instructions when executed, further program the mobile device to:

transmit the error information to the computing device.

33. A computing device that facilitates field testing of mobile devices to which the computing device is communicably coupled, the computing device comprising:

one or more physical processors programmed with computer program instructions, the computer program instructions, when executed by the one or more physical processors, program the computing device to;

obtain, from a first mobile device, error information relating to an error that occurred in relation to execution of an operation at the first mobile device;

obtain, from a second mobile device, replication information relating to an attempt by the second mobile device to replicate the error;

determine whether the second mobile device replicated the error based on the replication information;

determine whether the first mobile device and the second mobile device share in common an attribute responsive to a determination that the first other mobile device replicated the error;

determine that the attribute is a possible cause of the error responsive to a determination that the first mobile device and the second mobile device share in common the attribute.

34. The computing device of claim 33, the computing device further programmed to:

cause the error information to be transmitted to the second mobile device.

35. The computing device of claim 33, wherein the error information is transmitted from the first mobile device to the second mobile device.