Providing a set of user input actions to a mobile device to cause performance of the set of user input actions

• US 10,248,440 B1
• Filed: 10/13/2017
• Issued: 04/02/2019
• Est. Priority Date: 07/11/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Sharifi, Matthew, Petrou, David
• Primary Examiner(s): Kells, Asher D
• Application Number:
  US15/783,390
• Time in Patent Office: 536 Days
• Field of Search: None
• CPC Class Codes: G06F 16/156 Query results presentation
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/957 Browsing optimisation, e.g. caching or content distillation
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 40/134 Hyperlinking
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06Q 10/1097 Task assignment
• CPC Class Codes: G06V 30/416 Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04W 4/18 Information format or content conversion, e.g. adaptation by the network of the transmitted or received information for the purpose of wireless delivery to users or terminals

Abstract

Systems and methods are provided for automating user input using onscreen content. For example, a method includes receiving a selection of a first screen capture image representing a screen captured on a mobile device associated with a user, the first image having a first timestamp. The method also includes determining, using a data store of images of previously captured screens of the mobile device, a reference image from the data store that has a timestamp prior to the first timestamp, identifying a plurality of images in the data store that have respective timestamps between the timestamp for the reference image and the first timestamp, and providing the reference image, the plurality of images, and the first image to the mobile device.

Claims

1. A method comprising:

receiving a selection of a first screen capture image representing a screen captured on a mobile device associated with a user, the first image having a first timestamp;

determining, using a data store of images of previously captured screens of the mobile device, a reference image from the data store that has a timestamp prior to the first timestamp;

identifying a plurality of images in the data store that have respective timestamps between the timestamp for the reference image and the first timestamp;

determining a set of stored user input actions that occur prior to the first timestamp of the first image and after a timestamp corresponding to the reference image; and

providing the reference image, the plurality of images, the first image, and the set of user input actions to the mobile device, wherein providing the reference image, the plurality of images, the first image, and the set of user input actions to the mobile device causes the mobile device to perform the set of user input actions starting from a state corresponding to the reference image, wherein in performing the set of stored user input actions the mobile device automatically performs the actions using a virtual screen that is not visible via the mobile device.

2. The method of claim 1, wherein the reference image represents an image of a home screen for the mobile device.

3. The method of claim 1, wherein the reference image represents an initial image of a mobile application associated with the first image.

4. The method of claim 1, wherein at least two of the user input actions have the same timestamp and together represent a multiple finger input action, and wherein in automatically performing the set of stored user input actions the mobile device performs the multiple finger input action.

5. The method of claim 1, wherein at least one of the user input actions is a non-text input action that has an x coordinate, a y coordinate, and a type.

6. The method of claim 5, wherein the type is swipe action.

7. The method of claim 5, wherein the type is a tap action.

8. The method of claim 5, wherein at least one of the user input actions is a text input action.

9. A method comprising:

receiving a selection of a first screen capture image representing a screen captured on a first mobile device associated with a user, the first image having a first timestamp;

determining, using a data store of images of previously captured screens of the first mobile device, a reference image from the data store that has a timestamp prior to the first timestamp, the reference image representing another previously captured screen on the first mobile device;

identifying a plurality of images in the data store that have respective timestamps between the timestamp for the reference image and the first timestamp;

determining a set of stored user input actions that occur prior to the timestamp corresponding to the first image and after a timestamp corresponding to the reference image; and

providing the set of user input actions, the reference image, the plurality of images, and the first image to a second mobile device associated with the user, wherein providing the set of user input actions, the reference image, the plurality of images, and the first image to the second mobile device causes the second mobile device to automatically perform the set of user input actions starting from a state corresponding to the reference image, wherein in automatically performing the set of stored user input actions the second mobile device automatically performs the actions using a virtual screen that is not visible via the second mobile device.

10. The method of claim 9, wherein at least two of the user input actions have the same timestamp and together represent a multiple finger input action, and wherein in automatically performing the set of stored user input actions the second mobile device performs the multiple finger input action.

11. The method of claim 9, wherein at least one of the user input actions is a non-text input action that has an x coordinate, a y coordinate, and a type.

12. The method of claim 11, wherein the type is a swipe action.

13. The method of claim 11, wherein the type is a tap action.

14. The method of claim 11, wherein at least one of the user input actions is a text input action.