Enhancing automated mobile application testing

• US 9,501,390 B1
• Filed: 01/02/2013
• Issued: 11/22/2016
• Est. Priority Date: 01/02/2013
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Nadar, Ashok Thangamani, Talreja, Kamlesh Tekhbahadur, Singh, Lokendra, Chellappa, Sridhar, Surishetty, Pavan Kumar
• Primary Examiner(s): Zhen, Li B
• Assistant Examiner(s): Wheaton, Bradford
• Application Number:
  US13/732,774
• Time in Patent Office: 1,420 Days
• Field of Search: 717/126
• US Class Current: 1/1
• CPC Class Codes: G06F 11/3676 for coverage analysis
• CPC Class Codes: G06F 11/3684 for test design, e.g. generating new test cases
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 11/3692 for test results analysis

Abstract

Disclosed are various embodiments for enhancing automated testing for mobile applications by using an automated testing set. An automated testing pattern set may include multiple patterns that have been determined to yield accurate results in testing applications within a particular application category. The automated testing pattern set may be created for each application category by applying automated random pattern generator tests to applications within the same application category and verifying the accuracy of the random patterns produced by the random pattern generator tests by comparing the results from manual testing. The automated testing pattern set is then created based on patterns ranked according to an assigned accuracy score.

Claims

1. A non-transitory computer-readable medium embodying a program executable in at least one computing device, wherein, when executed, the program causes the at least one computing device to at least:

receive an application for testing;

select a specific application category of a plurality of application categories based at least in part on the application;

determine that an automated testing pattern subset associated with the specific application category is incomplete based at least in part on at least one of a number of other applications tested using the automated testing pattern subset or a number of testing patterns included in the automated testing pattern subset; and

in response to determining that the automated testing pattern subset associated with the specific application category is incomplete;

perform automated random testing on the application, the automated random testing generating a plurality of random testing patterns;

determine a plurality of random testing results based at least in part on the automated random testing of the application, individual ones of the plurality of random testing results corresponding to a respective random testing pattern of the plurality of random testing patterns;

store the plurality of random testing results and the plurality of random testing patterns;

receive a plurality of manual testing results determined from manual testing of the application;

calculate an accuracy score for individual ones of the plurality of random testing patterns based at least in part on a comparison of the plurality of manual testing results with the plurality of random testing results; and

determine whether to add at least one of the plurality of random testing patterns to the automated testing pattern subset associated with the specific application category based at least in part on the accuracy score associated with the individual ones of the plurality of random testing patterns.

2. The non-transitory computer-readable medium of claim 1, wherein, when executed, the program further causes the at least one computing device to at least perform additional automated testing on the application based at least in part on the automated testing pattern subset associated with the specific application category in response to determining that the automated testing pattern subset is complete.

3. The non-transitory computer-readable medium of claim 2, wherein, when executed, the program further causes the at least one computing device to at least send a notification to a user, wherein the notification includes at least one automated testing result.

4. A system, comprising:

at least one computing device; and

at least one application executable in the at least one computing device, wherein, when executed, the at least one application causes the at least one computing device to at least;

select a specific application category of a plurality of application categories based at least in part on an application received for testing;

determine that an automated testing pattern subset associated with the specific application category is incomplete based at least in part on at least one of a number of other applications tested using the automated testing pattern subset or a number of testing patterns included in the automated testing pattern subset; and

in response to determining that the automated testing pattern subset associated with the specific application category is incomplete;

perform automated random testing on the application, the automated random testing generating a plurality of random testing patterns;

determine a plurality of random testing results based at least in part on the automated random testing of the application, individual ones of the plurality of random testing results corresponding to a respective random testing pattern of the plurality of random testing patterns;

receive a plurality of manual testing results determined from manual testing of the application;

calculate an accuracy score for individual ones of the plurality of random testing patterns based at least in part on a comparison of the plurality of manual testing results with the plurality of random testing results; and

determine whether to add at least one of the plurality of random testing patterns to the automated testing pattern subset associated with the specific application category based at least in part on the accuracy score associated with the individual ones of the plurality of random testing patterns.

5. The system of claim 4, wherein, when executed, the at least one application causes the at least one computing device to at least rank the individual ones of the plurality of random testing patterns according to the accuracy score, a highest ranked random testing pattern of the plurality of random testing patterns being determined to be added to the automated testing pattern subset.

6. The system of claim 4, wherein the application is a first application, and when executed, the at least one application causes the at least one computing device to at least:

receive a second application for testing from a developer client device, the second application being associated with the specific application category; and

perform automated testing of the second application by applying the automated testing pattern subset associated with the specific application category to the second application.

7. The system of claim 6, wherein, when executed, the at least one application causes the at least one computing device to at least determine that the second application passed the automated testing.

8. The system of claim 6, wherein, when executed, the at least one application causes the at least one computing device to at least determine that the second application failed the automated testing.

9. The system of claim 6, wherein, when executed, the at least one application causes the at least one computing device to at least send a notification including at least one automated testing result to a client device.

10. The system of claim 4, wherein the automated testing pattern subset is selected from a plurality of automated testing pattern subsets, and individual ones of the automated testing pattern subsets are associated with a corresponding application category of the plurality of application categories.

11. The system of claim 4, wherein a respective random testing pattern of the plurality of random testing patterns is determined to be accurate when a respective random testing result of the plurality of random testing results is the same as at least one of the plurality of manual testing results.

12. The system of claim 11, wherein the accuracy score is incremented when the respective random testing pattern is determined to be accurate.

13. The system of claim 11, wherein the accuracy score is decremented when the respective random testing pattern is determined to be accurate.

14. A method, comprising:

selecting, via at least one computing device, a specific application category of a plurality of application categories based at least in part on an application received for testing;

determining, via the at least one computing device, that an automated testing pattern subset associated with the specific application category is incomplete based at least in part on at least one of a number of other applications tested using the automated testing pattern subset or a number of testing patterns included in the automated testing pattern subset; and

in response to determining that the automated testing pattern subset associated with the specific application category is incomplete;

performing, via the at least one computing device, automated random testing on the application, the automated random testing generating a plurality of random testing patterns;

determining, via at least one computing device, a plurality of random testing results based at least in part on the automated random testing of the application, individual ones of the plurality of random testing results corresponding to a respective random testing pattern of the plurality of random testing patterns;

receiving, via at least one computing device, a plurality of manual testing results determined from manual testing of the application;

calculating, via at least one computing device, an accuracy score for individual ones of the plurality of random testing patterns based at least in part on a comparison of the plurality of manual testing results with the plurality of random testing results; and

determining, via the at least one computing device, whether to add at least one of the plurality of random testing patterns to the automated testing pattern subset associated with the specific application category based at least in part on the accuracy score associated with the individual ones of the plurality of random testing patterns.

15. The method of claim 14, wherein the application is a first application, and the method further comprises:

receiving, via the at least one computing device, a second application for testing from a developer client device, the second application being associated with the specific application category; and

performing, via the at least one computing device, automated testing on the second application using the automated testing pattern subset associated with the specific application category.

16. The method of claim 15, further comprising determining, via the at least one computing device, that the automated testing of the second application creates a failure.

17. The method of claim 16, further comprising notifying, via the at least computing device, the developer client device of the failure.

18. The method of claim 17, further comprising:

receiving, via the at least one computing device, an altered version of the second application from the developer client device, wherein the second application is altered in response to the failure; and

performing, via the at least one computing device, the automated testing of the altered version using the automated testing pattern subset associated with the specific application category.

19. The method of claim 14, wherein the automated testing pattern subset is selected from a plurality of automated testing pattern subsets, and individual ones of the automated testing pattern subsets are associated with a corresponding application category of the plurality of application categories.

20. The method of claim 14, further comprising:

ranking, via the at least one computing device, the individual ones of the plurality of random testing patterns according to the accuracy score; and

wherein the at least one of the plurality of random testing patterns are determined to be added to the automated testing pattern subset based at least in part on the ranking.