Automating user input from onscreen content

• US 10,592,261 B1
• Filed: 04/01/2019
• Issued: 03/17/2020
• Est. Priority Date: 07/11/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Sharifi, Matthew, Petrou, David
• Primary Examiner(s): Kells, Asher D
• Application Number:
  US16/371,965
• Time in Patent Office: 351 Days
• CPC Class Codes: G06F 16/156 Query results presentation
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/957 Browsing optimisation, e.g. caching or content distillation
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 40/134 Hyperlinking
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06Q 10/1097 Task assignment
• CPC Class Codes: G06V 30/416 Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04W 4/18 Information format or content conversion, e.g. adaptation by the network of the transmitted or received information for the purpose of wireless delivery to users or terminals

Abstract

Systems and methods are provided for automating user input using onscreen content. For example, a method includes receiving a selection of a first screen capture image representing a screen captured on a mobile device associated with a user, the first image having a first timestamp. The method also includes determining, using a data store of images of previously captured screens of the mobile device, a reference image from the data store that has a timestamp prior to the first timestamp, identifying a plurality of images in the data store that have respective timestamps between the timestamp for the reference image and the first timestamp, and providing the reference image, the plurality of images, and the first image to the mobile device.

Claims

1. A method comprising:

receiving a selection of a first image representing a screen captured on a mobile device associated with a user, the first image having a first timestamp;

determining, using a data store of images of previously captured screens of the mobile device, a reference image from the data store that has a prior timestamp that is prior to the first timestamp;

identifying, based on the first image and the reference image, a plurality of user input actions occurring between the prior timestamp and the first timestamp, each of the user input actions having a respective timestamp;

identifying, based on the reference image, a software application installed on the mobile device, the reference image representing a screen of the software application;

launching, on the mobile device, the software application to a state tailored to the reference image; and

causing, based on the respective timestamps of the user input actions, the software application of the mobile device to replay the user input actions,

wherein replaying the user input actions causes the software application to transition from the state that is tailored to the reference image to a state that corresponds to the first image.

2. The method of claim 1, further comprising:

prior to receiving the selection of the first image;

providing, via a display of the mobile device, an assistance window, the assistance window including a preview of the first image; and

receiving user input directed to the assistance window as the selection of the first image, wherein the user input includes a touch input or a spoken input.

3. The method of claim 2, wherein the user input is the spoken input, and wherein the spoken input is received via one or more microphones of the mobile device.

4. The method of claim 2, wherein the assistance window is provided along with a given search result, of a plurality of search results, that are responsive to a search query.

5. The method of claim 2, further comprising:

identifying a habit associated with a user,

wherein the habit associated with the user includes a temporal indication and user behavior associated with the temporal indication, and

wherein the preview of the first image in the assistance window includes the temporal indication and the user behavior of the habit associated with the user.

6. The method of claim 1, wherein identifying the software application installed on the mobile device comprises:

determining that the software application is installed on the mobile device,

wherein launching the software application to a state tailored to the reference image is in response to determining that the software application is installed on the mobile device.

7. The method of claim 1, further comprising:

subsequent to the replaying of the user input actions;

rendering, for display on the mobile device, content that corresponds to the first image.

8. The method of claim 7, further comprising:

subsequent to the rendering of the content that corresponds to the first image;

transmitting, based on user input received at the mobile device, a second image of the screen, including the content that corresponds to the first image, to a second mobile device of an additional user.

9. A computing device, comprising:

at least one processor; and

at least one memory device storing instructions that, when executed, cause the at least one processor to;

receive a selection of a first image representing a screen captured on a computing device associated with a user, the first image having a first timestamp;

determine, using a data store of images of previously captured screens of the computing device, a reference image from the data store that has a prior timestamp that is prior to the first timestamp;

identify, based on the first image and the reference image, a plurality of user input actions occurring between the prior timestamp and the first timestamp, each of the user input actions having a respective timestamp;

identify, based on the reference image, a software application installed on the computing device, the reference image representing a screen of the software application;

launch, on the computing device, the software application to a state that is tailored to the reference image; and

cause, based on the respective timestamps of the user input actions, the software application of the computing device to replay the user input actions,

wherein the instructions that cause the software application to replay each of the user input actions further include instructions that cause the software application to transition from the state that is tailored to the reference image to a state that corresponds to the first image.

10. The computing device of claim 9, wherein the instructions further cause the at least one processor to:

prior to receiving the selection of the first image;

provide, via a display of the computing device, an assistance window, the assistance window including a preview of the first image; and

receive user input directed to the assistance window as the selection of the first image, wherein the user input includes a touch input or a spoken input.

11. The computing device of claim 10, wherein the user input is the spoken input, and wherein the spoken input is received via one or more microphones of the computing device.

12. The computing device of claim 10, wherein the assistance window is provided along with a given search result, of a plurality of search results, that are responsive to a search query.

13. The computing device of claim 10, wherein the instructions further cause the at least one processor to:

identify a habit associated with a user,

wherein the habit associated with the user includes a temporal indication and user behavior associated with the temporal indication, and

wherein the preview of the first image in the assistance window includes the temporal indication and the user behavior of the habit associated with the user.

14. The computing device of claim 9, wherein the instructions that cause the computing device to identify the software application installed on the computing device further comprise instructions to:

determine that the software application is installed on the computing device,

wherein launching the software application to a state tailored to the reference image is in response to determining that the software application is installed on the computing device.

15. The computing device of claim 9, wherein the instructions further cause the at least one processor to:

subsequent to the replaying of the user input actions;

render, for display on the computing device, content that corresponds to the first image.

16. The computing device of claim 15, wherein the instructions further cause the at least one processor to:

subsequent to the rendering of the content that corresponds to the first image;

transmit, based on user input received at the computing device, a second image of the screen, including the content that corresponds to the first image, to a second computing device of an additional user.

17. A mobile device, comprising:

at least one processor; and

at least one memory device storing instructions that, when executed, cause the at least one processor to;

receive, at the mobile device and from a second mobile device, a first image representing a screen captured on the second mobile device,

wherein the mobile device is associated with a user,

wherein the second mobile device is associated with the user or an additional user, and

wherein the first image has a first timestamp;

receive, at the mobile device, a selection of the first image;

determine, using a data store of images of previously captured screenshots, a reference image from the data store that has a prior timestamp that is prior to the first timestamp;

identify, based on the first image and the reference image, a plurality of user input actions occurring between the prior timestamp and the first timestamp, each of the user input actions having a respective timestamp;

identify, based on the reference image, a software application installed on the mobile device, the reference image representing a screen of the software application;

launch, on the mobile device, the software application to a state that is tailored to the reference image; and

cause, based on the respective timestamp for the user input actions, the software application of the mobile device to replay the user input actions,

wherein the instructions that cause the software application to replay the user input actions include instructions that cause the software application to transition from the state that is tailored to the reference image to a state that corresponds to the first image.

18. The mobile device of claim 17, wherein the instructions that cause the mobile device to identify the software application installed on the mobile device further comprise instructions to:

determine that the software application is installed on the mobile device,

wherein launching the software application to a state tailored to the reference image is in response to determining that the software application is installed on the mobile device.

19. The mobile device of claim 17, wherein the instructions further cause the at least one processor to:

subsequent to the replaying of the user input actions;

render, for display on the mobile device, content that corresponds to the first image.

20. The mobile device of claim 19, wherein the instructions further cause the at least one processor to:

subsequent to the rendering of the content that corresponds to the first image;

transmit, based on user input received at the mobile device, a second image of the screen, including the content that corresponds to the first image, to a third mobile device of another additional user.