Annotating relevant content in a screen capture image

• US 9,798,708 B1
• Filed: 08/04/2014
• Issued: 10/24/2017
• Est. Priority Date: 07/11/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Sharifi, Matthew, Petrou, David
• Primary Examiner(s): Kells, Asher
• Application Number:
  US14/451,389
• Time in Patent Office: 1,177 Days
• CPC Class Codes: G06F 16/156 Query results presentation
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/957 Browsing optimisation, e.g. caching or content distillation
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 40/134 Hyperlinking
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06Q 10/1097 Task assignment
• CPC Class Codes: G06V 30/416 Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04W 4/18 Information format or content conversion, e.g. adaptation by the network of the transmitted or received information for the purpose of wireless delivery to users or terminals

Abstract

Systems and methods are provided for highlighting relevant mobile onscreen content. For example, a mobile device can include memory storing instructions that, when executed by at least one processor, cause the mobile device to perform operations including capturing an image of a screen on the mobile device, the screen being displayed on a display of the mobile device, and providing the image to a server. The operations may also include receiving annotation data from the server, the annotation data including a visual cue that corresponds to a portion of the image that includes an entry in a list, the entry being associated with an entity in a graph-based data store relevant to a user of the mobile device, and display the annotation data with a second screen being displayed on the displaying of the mobile device so that the visual cue aligns with the entry in the second screen.

Claims

1. A system comprising:

at least one processor; and

memory storing instructions that, when executed by the at least one processor, cause the system to;

receive an image of a screen captured from a display of a mobile device,

perform text recognition on the image,

determine that the image includes a body of text, wherein the image is determined to include a body of text;

when at least one paragraph of text is identified in the image, or

when a majority of the image is determined to be text, or

when a number of words identified in the image exceeds a threshold value,

identify, responsive to the determination that the image includes a body of text, content-relevant text that summarizes the body of text identified in the image,

generate annotation data, the annotation data including a visual cue that changes an appearance of the content-relevant text, and

provide the annotation data to the mobile device.

2. The system of claim 1, wherein the mobile device displays the annotation data with a screen being displayed on the display of the mobile device so that the visual cue is aligned with the content-relevant text in the display.

3. The system of claim 1, wherein the image is a first image and the annotation data is first annotation data, and the memory further stores instructions that, when executed by the at least one processor, cause the system to:

receive a second image of a second screen captured on the mobile device;

identify at least one entity in the second image relevant to a user of the mobile device by performing recognition on the image;

determine that the entity is associated with a particular structure element of a display structure that repeats, the display structure being identified in the second image;

generate second annotation data for the second image, the second annotation data including a second visual cue that corresponds to the particular structure element; and

provide the second annotation data to the mobile device.

4. The system of claim 3, wherein identifying the at least one entity as relevant includes:

determining a collection the entity belongs to; and

determining that the collection is consistent with an interest of the user.

5. The system of claim 3, wherein identifying the at least one entity as relevant includes:

determining that the entity is a highly ranked entity for the user based on analysis of prior screen captures.

6. The system of claim 3, wherein the display structure is a list, and the structure element is an entry in the list.

7. The system of claim 1, wherein the annotation data is transparent other than the visual cue.

8. A method comprising:

receiving an image of a screen captured from a display of a mobile device;

determining entities in the image by performing recognition and entity detection on the image;

identifying at least one entity of the entities relevant to a user of the mobile device;

determining that the identified entity is associated with a particular structure element of a display structure that repeats, the display structure including structure elements other than those associated with entities identified as relevant to the user, wherein the display structure is identified in the image;

generating annotation data for the image, the annotation data including a visual cue that changes an appearance of the particular structure element when displayed on the mobile device; and

providing the annotation data to the mobile device.

9. The method of claim 8, wherein the mobile device displays the annotation data with a screen being displayed on the display of the mobile device.

10. The method of claim 8, wherein identifying the at least one entity as relevant includes:

determining a type for the entity; and

determining that the type is consistent with an interest of the user.

11. The method of claim 8, wherein identifying the at least one entity as relevant includes:

determining that the entity is a highly ranked entity for the user based on analysis of prior screen captures.

12. The method of claim 8, wherein determining entities in the image includes:

performing image recognition on the image; and

determining that a portion of the image corresponds to an entity.

13. The method of claim 8, wherein the image is a first image and the annotation data is first annotation data, and the method further comprises:

receiving a second image of a second screen captured on the mobile device;

identifying at least one body of text in the second image by performing text recognition on the second image;

determining a portion of the body of text that summarizes the body of text;

generating second annotation data, the second annotation data including a second visual cue that corresponds to the portion of the body of text; and

providing the second annotation data to the mobile device.

14. The method of claim 8, wherein the display structure is a list and the structure element is an entry in the list.

15. A mobile device comprising:

at least one processor; and

memory storing instructions that, when executed by the at least one processor, cause the mobile device to;

capture an image of a screen on the mobile device, the screen being displayed on a display of the mobile device,

provide the image to a server,

receive annotation data, the annotation data including a visual cue that corresponds to a portion of the image that includes all content of a structure element in a display structure that repeats, the structure element being associated with an entity in a graph-based data store relevant to a user of the mobile device and the display structure including structure elements other than the structure element associated with the entity identified as relevant to the user, wherein the visual cue changes an appearance of the structure element when displayed on the mobile device, and

display the annotation data with a second screen being displayed on the display of the mobile device so that the visual cue aligns with the display structure in the second screen.

16. The mobile device of claim 15, the memory further storing instructions that, when executed by the at least one processor, causes the mobile device to:

determine whether the second screen matches the image; and

display the annotation data with the second screen when the second screen matches the image.

17. The mobile device of claim 15, the memory further storing instructions that, when executed by the at least one processor, causes the mobile device to:

capture a second image of a third screen on the mobile device;

provide the second image to the server;

receive second annotation data, the second annotation data including a second visual cue that corresponds to a portion of the second image that represents text that summarizes a body of text in the second image; and

display the second annotation data with a fourth screen being displayed on the display of the mobile device so that the second visual cue aligns with the text that summarizes the body of text on the fourth screen.

18. The mobile device of claim 17, the second annotation data further including a third visual cue that corresponds to a second portion of the second image, the second portion including content that represents an entity in the graph-based data store.

19. The mobile device of claim 15, the annotation data further including a second visual cue that corresponds to a second portion of the image, the second portion representing text that summarizes a body of text in the image.

20. A mobile system comprising:

at least one processor; and

memory storing instructions that, when executed by the at least one processor, cause the mobile system to;

receive an image of a screen captured on the mobile system, the screen being displayed on a display device of the mobile system,

determine entities in the image by performing recognition on the image,

identify at least one entity of the entities relevant to a user of the mobile system,

determine that the identified entity is associated with a particular structure element of a display structure that repeats, the display structure including structure elements other than those associated with entities relevant to the user, wherein the display structure is identified in the image,

generate annotation data, the annotation data including a visual cue that changes an appearance of the particular structure element, and

display the annotation data with a screen being displayed on the display device so that the visual cue aligns with the particular structure element on the screen.

21. The mobile system of claim 20, wherein the display structure is a list and the structure element is an entry in the list.

22. The mobile system of claim 20, wherein memory further stores instructions that, when executed by the at least one processor, causes the mobile system to:

determine that the display structure includes a quantity of entities relevant to the user; and

select a predetermined number of highest ranked entities of the quantity of entities, the predetermined number being less than the quantity,

wherein the annotation data includes a respective visual cue for each structure element that includes one of the predetermined number of highest ranked entities.

23. The mobile system of claim 20 wherein identifying the at least one entity as relevant includes:

determining a type for the entity; and

determining that the type is consistent with an interest of the user.

24. The mobile system of claim 20 wherein identifying the at least one entity as relevant includes:

determining that the entity is a highly ranked entity for the user based on analysis of prior screen captures.

25. The mobile system of claim 20, wherein the image is a first image and the annotation data is first annotation data, and the memory further stores instructions that, when executed by the at least one processor, causes the mobile system to:

receive a second image of a second screen captured on the mobile system;

identify at least one body of text in the second image by performing text recognition on the second image;

determine a portion of the body of text that summarizes the body of text;

generate second annotation data, the second annotation data including a second visual cue that corresponds to the portion of the body of text; and

display the second annotation data in front of a third screen being displayed on the display device, so that the second visual cue aligns with the portion of the body of text in the third screen.

26. The mobile system of claim 25, wherein the body of text is a paragraph.

27. The mobile system of claim 25, wherein the first image is generated by a first mobile application and the second image is generated by a second mobile application.

28. The mobile system of claim 25, wherein the at least one entity is identified as relevant to a user of the mobile system based on past interactions with mobile applications.