Replaying user input actions using screen capture images

• US 9,811,352 B1
• Filed: 08/04/2014
• Issued: 11/07/2017
• Est. Priority Date: 07/11/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Sharifi, Matthew, Petrou, David
• Primary Examiner(s): Kells, Asher
• Application Number:
  US14/451,401
• Time in Patent Office: 1,191 Days
• Field of Search: None
• CPC Class Codes: G06F 16/156 Query results presentation
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/957 Browsing optimisation, e.g. caching or content distillation
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 40/134 Hyperlinking
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06Q 10/1097 Task assignment
• CPC Class Codes: G06V 30/416 Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04W 4/18 Information format or content conversion, e.g. adaptation by the network of the transmitted or received information for the purpose of wireless delivery to users or terminals

Abstract

Systems and methods are provided for automating user input using onscreen content. For example, a method includes receiving a selection of a first image representing a previously captured screen of a mobile device, the first image having a corresponding timestamp, determining a set of stored user input actions that occur prior to the timestamp corresponding to the first image and after a timestamp corresponding to a reference image, the reference image representing another previously captured screen of the mobile device, and providing a user interface element configured to, when selected, initiate a replaying of the set of user input actions on the mobile device, starting from a state corresponding to the reference image.

Claims

1. A mobile device comprising:

at least one processor;

a data store storing;

a plurality of captured user input actions, each input action having a corresponding timestamp; and

a plurality of images, each image representing a screen captured on the mobile device, each image having a corresponding timestamp; and

memory storing instructions that, when executed by the at least one processor, cause the mobile device to;

receive a selection of a first image of the plurality of images,

determine a set of the captured user input actions, the set including user input actions occurring prior to a timestamp corresponding to the first image and after a timestamp corresponding to a reference image, the reference image being one of the plurality of images, and

replay the set of user input actions on the mobile device starting from a state corresponding to the reference image, wherein the replay includes;

performing the user input actions in the set that occur between the timestamp for the reference image and a timestamp for a subsequent image, the performance of the actions resulting in a current screen,

comparing the current screen with the subsequent image, and

responsive to the current screen matching the subsequent image, performing the user input actions in the set that occur between the timestamp for the subsequent image and a next subsequent image.

2. The mobile device of claim 1, wherein the memory further stores instructions that, when executed by the at least one processor, cause the mobile device to:

receive an indication to begin tracking user input; and

capture, in response, new user input actions.

3. The mobile device of claim 1, wherein each of the plurality of user input actions is either a text input action or a non-text input action, and each non-text input action includes an x coordinate, a y coordinate, and an input type.

4. The mobile device of claim 1, wherein replaying the set of user input actions includes:

requesting character input from a user of the mobile device; and

replacing at least one character input action with the character input.

5. The mobile device of claim 1 wherein the reference image is an initial screen for a mobile application that generated the first image.

6. The mobile device of claim 1, wherein the current screen is a virtual screen not visible to a user of the mobile device until the replay terminates.

7. The mobile device of claim 6, wherein replaying the set of user input actions includes, responsive to the current screen failing to match the subsequent image:

terminating the replaying; and

making the current screen visible to the user of the mobile device.

8. The mobile device of claim 1, wherein the plurality of images are images captured between the timestamp of the reference image and the timestamp of the first image, inclusive, and the memory further stores instructions that, when executed by the at least one processor, cause the mobile device to:

send the first image to a server; and

receive the plurality of images from the server, the reference image being the image with the earliest timestamp of the plurality of images.

9. The mobile device of claim 1, wherein the reference image represents an image of a home screen.

10. The mobile device of claim 1, wherein at least some user input actions include a screen coordinate corresponding to a location of the action taken by the user.

11. A method comprising:

receiving a selection of a first image representing a previously captured screen of a mobile device, the first image having a corresponding timestamp;

determining a set of stored user input actions that occur prior to the timestamp corresponding to the first image and after a timestamp corresponding to a reference image, the reference image representing another previously captured screen of the mobile device; and

providing a user interface element configured to, when selected, initiate a replaying of the set of user input actions on the mobile device, starting from a state corresponding to the reference image, wherein the replaying includes;

identifying a plurality of previously captured images, the plurality of previously captured images including the first image and the reference image and including images captured between the timestamp for the reference image and the timestamp for the first image,

performing the user input actions in the set that occur between the timestamp for the reference image and a timestamp for a subsequent image in the plurality of previously captured images, the performance of the actions resulting in a current screen,

comparing the current screen with the subsequent image, and

responsive to the current screen matching the subsequent image, performing the user input actions in the set that occur between the timestamp for the subsequent image and a next subsequent image in the plurality of previously captured images.

12. The method of claim 11, wherein at least two of the user input actions have the same timestamp and together represent a multiple finger input action.

13. The method of claim 11, wherein each of the user input actions in the set is either a text input action or a non-text input action, and each non-text input action has an associated x coordinate, y coordinate, and input type.

14. The method of claim 11, wherein the replaying of the set of user input actions includes:

identifying a next user input action as a text input action;

requesting character input from a user of the mobile device; and

using the character input instead of the next user input action during the replaying.

15. The method of claim 11, wherein the reference image represents an image of an initial screen for a mobile application that generated the first image.

16. The method of claim 11, wherein the current screen is a virtual screen not visible to a user of the mobile device until the replay terminates.

17. The method of claim 16, wherein the replaying of the set of user input actions includes, responsive to determining the current screen fails to match the successive image:

terminating the replaying; and

making the current screen visible to the user of the mobile device.

18. The method of claim 11, wherein identifying the plurality of previously captured images includes:

sending the first image to a server; and

receiving the plurality of previously captured images from the server.

19. The method of claim 11, further comprising:

receiving an indication to begin tracking user input; and

storing, in response, new captured user input actions.

20. The method of claim 19, further comprising:

providing the new captured user input actions to a server, the server using the new captured input actions as input to a machine learning algorithm for predicting future input.

21. The method of claim 11, wherein the reference image represents an image of a home screen.

22. A computer-readable medium storing instructions that, when executed by a processor, cause a computing device to perform operations comprising:

receiving a selection of a first image representing a previously captured screen of a mobile device, the first image having a corresponding timestamp;

determining a set of stored user input actions that occur prior to the timestamp corresponding to the first image and after a timestamp corresponding to a reference image, the reference image representing another previously captured screen of the mobile device;

returning the computing device to a state represented by the reference image; and

performing the set of user input actions starting from the reference image by iteratively;

performing the user input actions in the set that occur between the timestamp for the reference image and a timestamp for a subsequent image, the performance of the actions resulting in a current screen,

comparing the current screen with the subsequent image, and

responsive to the current screen matching the subsequent image, performing the user input actions in the set that occur between the timestamp for the subsequent image and a next subsequent image,

wherein performing the set of user input actions takes the computing device to a state represented by the first image.