Providing actions for mobile onscreen content

• US 9,824,079 B1
• Filed: 08/04/2014
• Issued: 11/21/2017
• Est. Priority Date: 07/11/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Sharifi, Matthew, Petrou, David
• Primary Examiner(s): Kells, Asher
• Application Number:
  US14/451,385
• Time in Patent Office: 1,205 Days
• CPC Class Codes: G06F 16/156 Query results presentation
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/957 Browsing optimisation, e.g. caching or content distillation
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 40/134 Hyperlinking
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06Q 10/1097 Task assignment
• CPC Class Codes: G06V 30/416 Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04W 4/18 Information format or content conversion, e.g. adaptation by the network of the transmitted or received information for the purpose of wireless delivery to users or terminals

Abstract

Systems and methods identify actionable content in onscreen content and provide at least a default action for the content. For example, a method may include receiving an image of a screen captured from a display of a mobile device, determining areas of actionable content in the image, and determining a respective action for each area of actionable content. The method may also include generating annotation data for the image, the annotation data including a visual cue that corresponds to a first area of actionable content, the visual cue being actionable to initiate the respective action when selected and providing the annotation data for display on the mobile device.

Claims

1. A system comprising:

at least one processor; and

memory storing instructions that, when executed by the at least one processor, cause the system to perform operations comprising;

receiving an image of a screen captured from a display of a mobile device,

determining areas of actionable content in the image,

determining that a density of areas of actionable content in a first portion of the image exceeds a threshold, the first portion including a first area of actionable content and a second area of actionable content,

selecting the first area in response to determining that content represented by the first area has a higher relevancy ranking than content represented by the second area of actionable content,

determining an action for the first area of actionable content,

generating annotation data for the image, the annotation data including a visual cue that corresponds to the first area of actionable content, the visual cue being actionable to initiate the respective action when selected and configured to be displayed with a current screen on the mobile device, and

providing the annotation data for display on the mobile device.

2. The system of claim 1, wherein the system further includes a graph-based data store of entities connected by edges and determining areas of actionable content includes:

performing character recognition on the image to generate text; and

determining that a portion of the text corresponds to an entity in the graph-based data store,

wherein an area of the image corresponding to the portion of the text is the first actionable area.

3. The system of claim 1, wherein each area of actionable content has a type and determining the action includes using the type to determine the action.

4. The system of claim 1, wherein the corresponding visual cue provides an indication of the respective action.

5. The system of claim 1, wherein an appearance of the visual cue is based on a type of the first area.

6. The system of claim 1, the operations further comprising:

determining a respective type for each of a plurality of the areas of actionable content;

determining that the respective type for the first area is a type selected by a user of the mobile device;

determining that the respective type of a second area is a type not selected by the user; and

responsive to the determining, include the first area in and exclude the second area from the annotation data.

7. The system of claim 1, the operations further comprising:

determining a third area of actionable content matches a template, wherein the first area of content and the second area of content correspond to entities in a data graph; and

selecting the third area based on the third area matching the template; and

determine an action associated with the third area,

wherein the annotation data includes a second visual cue that corresponds to the third area of actionable content.

8. The system of claim 1, wherein the annotation data is transparent other than the visual cue.

9. A method comprising:

receiving an image of a screen captured from a display of a mobile device;

determining areas of actionable content in the image;

determining that a density of areas of actionable content in a first portion of the image exceeds a threshold, the first portion including a first area of actionable content, a second area of actionable content, and a third area of actionable content;

selecting the first area in response to determining that content represented by the first area has a higher relevancy ranking than content represented by the second area of actionable content;

determining a first action for the first area of actionable content;

generating annotation data for the image, the annotation data including a first visual cue that corresponds to the first area of actionable content, the first visual cue being actionable to initiate the first action when selected; and

providing the annotation data for display on the mobile device.

10. The method of claim 9, wherein the annotation data includes data used to align the first visual cue with a corresponding area displayed on the display of the mobile device.

11. The method of claim 9, wherein determining areas of actionable content includes:

performing character recognition on the image to generate text; and

determining that a portion of the text corresponds to an entity in a graph-based data store,

wherein an area of the image corresponding to the portion of the text is the first actionable area.

12. The method of claim 11, the first portion also including a third area of actionable content and the method further comprising:

selecting the third area in response to determining that content represented by the third area matches a template, wherein the content represented by the first are and the content represented by the second area correspond to respective entities in a graph-based data store, wherein

the annotation data includes a second visual cue that corresponds to the third area of actionable content.

13. The method of claim 9, wherein determining areas of actionable content includes:

performing image recognition on the image; and

determining that a portion of the image corresponds to a logo, the portion of the image being one of the areas of actionable content.

14. The method of claim 9, wherein determining the respective action for a particular area of actionable content includes:

determining a default action selected by a user of the mobile device for a type of the first area of actionable content.

15. The method of claim 9, further comprising generating an index that includes entries for the areas of actionable content.

16. The method of claim 15, further comprising:

storing the annotation data with the image;

using the index to determine that the areas of actionable content are responsive to a query; and

providing the image and the annotation data to the mobile device.

17. The method of claim 9, wherein the respective actions are selected from a group including opening a URL in a browsing application, launching a phone dialer, launching a calendar application, launching a contacts application, and launching a map application.

18. The method of claim 9, wherein the relevancy ranking is a relevancy ranking with respect to a user of the mobile device.

19. A mobile device comprising:

at least one processor; and

memory storing instructions that, when executed by the at least one processor, cause the mobile device to;

capture an image of a screen on the mobile device, the screen being displayed on a display of the mobile device,

provide the image to a server,

receive annotation data, the annotation data including a quantity of visual cues, each visual cue corresponding to an area of actionable content, wherein the quantity of visual cues is restricted by a threshold for a density of areas of actionable content, wherein the quantity of visual cues includes at least a first visual cue that corresponds to a first area of actionable content and a second visual cue that corresponds to a second area of actionable content, the first visual cue being actionable to initiate a first action when selected, the second visual cue being actionable to initiate a second action when selected, and

display the annotation data with a screen being displayed on the display of the mobile device.

20. The mobile device of claim 19, the memory further storing instructions that, when executed by the at least one processor, causes the mobile device to:

determine whether the screen being displayed matches the image; and

display the annotation data with the screen when the screen being displayed matches the image.

21. The mobile device of claim 19, the memory further storing instructions that, when executed by the at least one processor, causes the mobile device to:

receive a selection of the first visual cue by a user of the mobile device; and

initiate the first action.

22. The mobile device of claim 19, the memory further storing instructions that, when executed by the at least one processor, causes the mobile device to:

store the image and the annotation data in memory;

index the first area of actionable content and associated it with the image and annotation data;

determine, using the index, that the first area of actionable content is responsive to a query; and

display the annotation data with the image in response to the query.

23. The mobile device of claim 19, wherein the first action is a default action selected by a user of the mobile device for a type of the first area of actionable content.

24. A mobile system comprising:

at least one processor; and

memory storing instructions that, when executed by the at least one processor, cause the system to;

receive an image of a screen captured on the mobile system, the screen being displayed on a display device of the mobile system,

determine areas of actionable content in the image,

determine that a density of areas of actionable content in a first portion of the image exceeds a threshold, the first portion including a first area of actionable content and a second area of actionable content,

select the first area based on determining that content represented by the first area has a higher relevancy ranking than content represented by the second area of actionable content,

for at least the first area of actionable content, determine an action associated with the first area,

generate annotation data, the annotation data including a visual cue that corresponds to the first area of actionable content, the visual cue being actionable to initiate the action when selected, and

display the annotation data with a screen being displayed on the display device.

25. The mobile system of claim 24, wherein each area of actionable content has a type and determining the action associated with the first area includes using the type to determine the action.

26. The mobile system of claim 24, wherein memory further stores instructions that, when executed by the at least one processor, causes the system to:

store the image and the annotation data in the memory; and

when the image is selected as a search result, provide the annotation data with the image.

27. The mobile system of claim 24 wherein the first area has a first type of a plurality of types and a visual appearance of the visual cue is associated with the type.

28. The mobile system of claim 24, wherein the memory further stores instructions that, when executed by the at least one processor, causes the mobile system to:

determine a respective type for each of a plurality of the areas of actionable content;

determine that the respective type for the first area is a type selected by a user of the mobile system;

determine that the respective type of a third area is a type not selected by the user; and

responsive to the determining, include the first area in and exclude the third area from the annotation data.

29. The mobile system of claim 24, wherein the first area of actionable content is associated with a logo included in the image.

30. The mobile system of claim 24, wherein the memory further stores instructions that, when executed by the at least one processor, causes the mobile system to:

determine a third area of actionable content matches a template, wherein the first area of content and the second area of content correspond to entities in a data graph;

select the third area based on the third area matching the template; and

determine an action associated with the third area,

wherein the annotation data includes a second visual cue that corresponds to the third area of actionable content.

31. The mobile system of claim 30, wherein the memory further stores instructions that, when executed by the at least one processor, causes the mobile system to:

receive a selection of the visual cue from a user; and

initiating the action in response to the selection.

32. The mobile system of claim 24, wherein the visual cue underlines the first area of actionable content, such that the first area of actionable content appears to be a hyperlink when the annotation data is displayed with the screen.

33. The mobile system of claim 24, wherein the visual cue provides an indication of the action.