Providing an annotation linking related entities in onscreen content

• US 9,582,482 B1
• Filed: 08/04/2014
• Issued: 02/28/2017
• Est. Priority Date: 07/11/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Sharifi, Matthew, Petrou, David
• Primary Examiner(s): Kells, Asher
• Application Number:
  US14/451,396
• Time in Patent Office: 939 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/156 Query results presentation
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/957 Browsing optimisation, e.g. caching or content distillation
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 40/134 Hyperlinking
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06Q 10/1097 Task assignment
• CPC Class Codes: G06V 30/416 Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04W 4/18 Information format or content conversion, e.g. adaptation by the network of the transmitted or received information for the purpose of wireless delivery to users or terminals

Abstract

Systems and methods are provided for providing insight for entities in mobile onscreen content. For example, a method includes receiving, from a mobile device, an indication of selection of a first entity represented by a visual cue in first annotation data for a screen capture image of a screen of the mobile device and determining entities related to the first entity in a graph-based data store. The method may also include identifying a second entity in the screen capture image that is one of the entities related to the first entity, generating second annotation data, the second annotation data including a visual element linking the first entity and the second entity, and providing the second annotation data for display with the screen on the mobile device.

Claims

1. A system comprising:

a graph-based data store of entities connected by edges;

at least one processor; and

memory storing instructions that, when executed by the at least one processor, cause the system to;

receive a selection of a first entity represented by a visual cue in first annotation data for a screen capture image of a screen of a mobile device;

determine entities related to the first entity in the graph-based data store;

identify a second entity in the screen capture image that is one of the entities related to the first entity;

generate second annotation data, the second annotation data including a visual element linking the first entity and the second entity; and

provide the second annotation data for display with the screen on the mobile device.

2. The system of claim 1, wherein the visual element includes text describing the relationship between the first entity and the second entity in the graph-based data store.

3. The system of claim 1, wherein the visual element is configured to be selectable.

4. The system of claim 1, wherein the instructions further including instructions that, when executed by the at least one processor, cause the system to:

identify a plurality of entities in the screen capture image related to the first entity in the graph-based data store; and

generate the second annotation data with a visual element for each of the plurality of entities that links the respective entity of the plurality of entities to the first entity.

5. The system of claim 1, wherein the instructions further including instructions that, when executed by the at least one processor, cause the system to:

receive a selection of the visual element;

generate a text-based description of the relationship between the first entity and the second entity using the graph-based data store; and

include the text-based description in third annotation data provided to the mobile device for display with the screen.

6. The system of claim 1, wherein the instructions further including instructions that, when executed by the at least one processor, cause the system to:

receive a simultaneous selection of the first entity and the second entity;

generate third annotation data that includes a text-based description of the relationship between the second entity and the first entity; and

provide the third annotation data to the mobile device for display with the screen.

7. The system of claim 1, wherein determining entities related to the first entity in the graph-based data store excludes entities related by a path with a length that exceeds a maximum path length.

8. The system of claim 1, wherein the visual cue is an area of actionable content associated with a default action, the visual cue being configured to initiate the default action in response to a first type of user input and to initiate determining related entities in response to a second type of user input.

9. The system of claim 1, wherein the visual element includes an indication of relatedness.

10. A method comprising:

receiving, from a mobile device, an indication of selection of a first entity represented by a visual cue in first annotation data for a screen capture image of a screen of the mobile device;

determining entities related to the first entity in a graph-based data store;

identifying a second entity in the screen capture image that is one of the entities related to the first entity;

generating second annotation data, the second annotation data including a visual element linking the first entity and the second entity; and

providing the second annotation data for display with the screen on the mobile device.

11. The method of claim 10, wherein the visual element includes text describing the relationship between the first entity and the second entity in the graph-based data store.

12. The method of claim 10, further comprising:

identifying a plurality of entities in the screen capture image related to the first entity in the graph-based data store; and

generating the second annotation data with a visual element for each of the plurality of entities that links the respective entity of the plurality of entities to the first entity.

13. The method of claim 12, further comprising:

determining a quantity of the plurality of entities exceeds a threshold;

selecting a subset of the plurality of entities based on a rank for each respective entity with regard to a user of the mobile device; and

generating the second annotation data using entities in the subset of the plurality of entities.

14. The method of claim 10, further comprising:

receiving an indication of simultaneous selection of the first entity and the second entity;

generating a text-based description of the relationship between the first entity and the second entity based on the graph-based data store; and

including the text-based description in third annotation data provided to the mobile device for display with the screen.

15. The method of claim 10, further comprising:

receiving a second selection of the first entity;

identifying a plurality of previously captured screen images from the mobile device that are associated with entities that are related to the first entity in the graph-based data store;

generating a search result that displays at least a respective portion of the plurality of previously captured screen images; and

providing the search result to the mobile device.

16. The method of claim 15, wherein generating the search result includes:

generating a respective visual cue for each portion in the search result, each respective visual cue identifying an entity in the portion that is related to the first entity in the graph-based data store.

17. The method of claim 16, wherein the search result is displayed as a scrollable list.

18. The method of claim 16, wherein the respective portions are selectable and configured to display the entire captured screen image when selected.

19. A mobile device comprising:

a display;

at least one processor; and

memory storing instructions that, when executed by the at least one processor, cause the mobile device to;

receive a selection of a first entity having a selectable visual cue in first annotation data displayed with a screen on the display of the mobile device, the annotation data being generated using an image capture of the screen,

provide the selection of the first entity to a server,

receive second annotation data, the second annotation data including at least one visual element linking the first entity and a second entity identified in the screen, the first entity and the second entity being connected by at least one relationship in a graph-based data store, and

display the second annotation data with the screen on the display of the mobile device.

20. The mobile device of claim 19, the memory further storing instructions that, when executed by the at least one processor, causes the mobile device to:

receive a selection of the second entity and the first entity,

send the selection to the server,

receive second annotation data, the second annotation data including a text description of the at least one relationship between the first entity and the second entity; and

display the second annotation data with the screen on the display of the mobile device.

21. The mobile device of claim 19, the memory further storing instructions that, when executed by the at least one processor, causes the mobile device to:

receive a selection of the visual element,

send the selection to the server,

receive second annotation data, the second annotation data including a text description of the at least one relationship between the first entity and the second entity; and

display the second annotation data with the screen on the display of the mobile device.

22. A mobile device comprising:

a display;

a graph-based data store of entities connected by edges;

at least one processor; and

memory storing instructions that, when executed by the at least one processor, cause the mobile device to;

receive a selection of a first entity represented by a visual cue in first annotation data for a screen capture image, the first annotation data being displayed with a screen on the display of the mobile device,

determine entities related to the first entity in the graph-based data store,

identify a second entity in the screen capture image that is one of the entities related to the first entity,

generate second annotation data, the second annotation data including a visual element linking the first entity and the second entity, and

displaying the second annotation data with the screen on the display of the mobile device.

23. The mobile device of claim 22, wherein the visual element is selectable.

24. The mobile device of claim 23, wherein memory further stores instructions that, when executed by the at least one processor, causes the mobile device to:

receive a selection of the visual element;

generate third annotation data, the third annotation data including a text-based description of the relationship between the first entity and the second entity using the graph-based data store; and

displaying the third annotation data with the screen on the display of the mobile device.

25. The mobile device of claim 22, wherein the visual element includes a description of the relationship between the first entity and the second entity in the data graph.

26. The mobile device of claim 22, wherein the memory further stores instructions that, when executed by the at least one processor, causes the mobile device to:

receive a simultaneous selection of the first entity and a third entity represented by another visual cue in the first annotation data;

determine whether a relationship exists between the first entity and the third entity in the graph-based data store; and

when the relationship exists;

generate third annotation data, the third annotation data including a text-based description of the relationship between the first entity and the second entity using the graph-based data store, and

displaying the third annotation data with the screen on the display of the mobile device.

27. The mobile device of claim 22, wherein the memory further stores instructions that, when executed by the at least one processor, causes the mobile device to:

receive a second selection of the first entity;

identify a plurality of previously captured screen images on the mobile device that are associated with entities that are related to the first entity in the graph-based data store;

generate a search result that displays at least a respective portion of the plurality of previously captured screen images; and

display the search result on the display of the mobile device.

28. The mobile device of claim 27, wherein generating the search result includes:

generating a respective visual cue for each portion in the search result, each respective visual cue identifying an entity in the portion that is related to the first entity in the graph-based data store.