Indexing mobile onscreen content

• US 9,886,461 B1
• Filed: 08/04/2014
• Issued: 02/06/2018
• Est. Priority Date: 07/11/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Sharifi, Matthew, Petrou, David
• Primary Examiner(s): Burke, Jeffrey A
• Assistant Examiner(s): Vu, Thong
• Application Number:
  US14/451,390
• Time in Patent Office: 1,282 Days
• Field of Search: 707696, 707748, 707758, 707741, 707739, 382128, 382137, 345428
• CPC Class Codes: G06F 16/156 Query results presentation
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/957 Browsing optimisation, e.g. caching or content distillation
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 40/134 Hyperlinking
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06Q 10/1097 Task assignment
• CPC Class Codes: G06V 30/416 Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04W 4/18 Information format or content conversion, e.g. adaptation by the network of the transmitted or received information for the purpose of wireless delivery to users or terminals

Abstract

Systems and methods for indexing onscreen content on a mobile device are provided. For example, a method can include determining, using an index for a user that associates key items with images of screens captured from a mobile device of the user, that a first key item is responsive to a query initiated at the mobile device, and identifying responsive images, the responsive images being associated with the first key item. The method may also include highlighting, in the responsive images, areas corresponding to the first key item, and providing a search result to the mobile device that includes portions of the responsive images. The portions of the responsive images may be areas corresponding to the first key item in the images.

Claims

1. A system comprising:

at least one processor;

an index that associates, for a user, key items with images of screens captured from a mobile device used by the user, the screens having content generated by a mobile application executing on the mobile device, each image being associated with a mobile application that generated the screen captured in the image; and

memory storing instructions that, when executed by the at least one processor, cause the system to;

determine, using the index, that a first key item is responsive to a text-based query initiated at the mobile device by the user and that the first key item relates to a class of mobile application,

identify responsive images, the responsive images being associated with the first key item in the index,

rank the responsive images, including boosting respective ranks of responsive images generated by mobile applications in the class,

provide a search result to the mobile device that includes a portion of highest ranked responsive images, wherein each portion includes an area of the corresponding image that displays the first key item, and

responsive to receiving a selection of a first portion, open the mobile application that generated the first portion.

2. The system of claim 1, the instructions further including instructions that, when executed by the at least one processor, cause the system to:

receive an image of a screen captured from a display of the mobile device used by the user;

determine a plurality of key items in the image by performing recognition on the image; and

update the index by associating each of the plurality of key items with the image in the index.

3. The system of claim 2, the instructions further including instructions that, when executed by the at least one processor, cause the system to:

calculate respective ranks for the plurality of key items with regard to the image, wherein key items located in areas of the image that are static for the mobile application receive a low rank; and

store the respective ranks in the index.

4. The system of claim 1, wherein the instructions further include instructions that, when executed by the at least one processor, cause the system to:

generate annotation data that includes a visual cue corresponding to the area of each respective portion.

5. The system of claim 1, wherein the index includes images of screens captured from two or more mobile devices, the two or more mobile devices being associated with the user.

6. The system of claim 1, wherein opening the mobile application that generated the first portion includes:

opening the mobile application to a state represented by the image.

7. The system of claim 1, wherein the instructions further include instructions that, when executed by the at least one processor, cause the system to:

responsive to receiving a selection of a second portion from the search result, identify a second image represented by the second portion; and

provide the second image to the mobile device for display on the screen of the mobile device.

8. The system of claim 7, wherein the instructions further include instructions that, when executed by the at least one processor, cause the system to:

identify annotation data for the second image; and

provide the annotation data with the second image to the mobile device.

9. A method comprising:

determining, using an index for a user that associates key items with images of screens captured from a mobile device of the user, that a first key item is responsive to a query initiated at the mobile device and that the first key item relates to a class of mobile applications, each image being associated in the index with a mobile application that generated the screen captured in the image;

identifying responsive images, the responsive images being associated with the first key item;

ranking the responsive images, including boosting respective ranks of responsive images generated by mobile applications in the class;

highlighting, in the responsive images, areas corresponding to the first key item;

providing a search result to the mobile device that includes portions of highest-ranked responsive images; and

responsive to receiving a selection of a first portion, open the mobile application that generated the first portion.

10. The method of claim 9, further comprising:

receiving an image of a screen captured from a display of the mobile device used by the user;

determining a plurality of key items in the image by performing recognition on the image; and

updating the index by associating each of the plurality of key items with the image in the index.

11. The method of claim 10, further comprising:

calculating respective ranks for the plurality of key items with regard to the image; and

storing the respective ranks in the index.

12. The method of claim 9, further comprising:

generating annotation data for the search result, where each portion is an area of actionable content associated with a visual cue, each visual cue being actionable to launch a mobile application associated with the respective image; and

providing the annotation data with the search result to the mobile device.

13. The method of claim 9, wherein the index includes images of screens captured from two or more mobile devices, the two or more mobile devices being associated with a single user.

14. The method of claim 9, wherein the portions of the responsive images represent full images.

15. The method of claim 9, wherein the portions of the responsive images are areas corresponding to the first key item in the images.

16. The method of claim 9, wherein the index includes images of screens captured from two or more mobile devices, the two or more mobile devices being associated with the user.

17. A mobile device comprising:

at least one processor; and

memory storing instructions that, when executed by the at least one processor, cause the mobile device to;

capture an image of a screen on the mobile device, the screen being content generated by a mobile application that is displayed on a display of the mobile device,

provide the image to a server,

send a query to the server the query relating to a class of mobile applications,

receive a search result from the server, the search result including portions of respective screen capture images that are responsive to the query ordered by respective rank, the screen capture images being previously provided by the mobile device to the server, and the respective ranks of images generated by mobile applications in the class represent a boosted rank,

display the search result on the display of the mobile device, and

responsive to receiving a selection of a first portion, open the mobile application that generated the content in first portion.

18. The mobile device of claim 17, the memory further storing instructions that, when executed by the at least one processor, causes the mobile device to:

receive annotation data with the search result, the annotation data including a visual cue with an area of actionable content for each portion of respective screen images in the search result, each visual cue being actionable to launch a respective mobile application associated with the respective portion; and

display the annotation data with the search result.

19. The mobile device of claim 17, the memory further storing instructions that, when executed by the at least one processor, causes the mobile device to:

receive a selection of the portion of a first image from the search result;

identify annotation data for the first image that was included as part of the search result; and

display the annotation data with the image on the display of the mobile device.

20. The mobile device of claim 17, wherein displaying the search result includes displaying the images of the screens as a strip of images.

21. A mobile device comprising:

at least one processor; and

memory storing instructions that, when executed by the at least one processor, cause the mobile device to;

receive an image of a screen captured on the mobile device, the screen being displayed on a display device of the mobile device, and the image including content generated by a mobile application executing when the image was captured,

determine a plurality of key items in the image by performing recognition on the image, at least one of the plurality of key items being an entity in a graph-based data store, wherein key items located in areas of the image that are static for the mobile application executing when the image was captured are excluded from the plurality of key items, and

associate, in an index, each of the plurality of key items with the image and the mobile application that generated the content in the image, and

use the index to provide the image in a search result responsive to a key item of the plurality of key items being responsive to a query.

22. The mobile device of claim 21, wherein the memory further stores instructions that, when executed by the at least one processor, causes the mobile device to, for each of the plurality of key items:

calculate a rank for the key item with regard to the image; and

store the rank in the index.

23. The mobile device of claim 22, wherein the memory further stores instructions that, when executed by the at least one processor, causes the mobile device to:

receive a plurality of images from a plurality of screens captured on the mobile device; and

for each of the plurality of images, determining key items and associating each of the key items with at least one of the respective images,

wherein calculating the rank for the key item includes;

upweighting the rank for the key item when it is associated with images from multiple mobile applications.

24. The mobile device of claim 21, wherein memory further stores instructions that, when executed by the at least one processor, causes the mobile device to:

identify key items in the index that are responsive to a query;

identify responsive images from the index, responsive images being associated with the key items responsive to the query; and

provide a search result that includes at least a portion of the responsive images.

25. The mobile device of claim 24, wherein providing the search result includes:

generating respective annotation data for each responsive image, the respective annotation data including visual cues that correspond to key items in the responsive image; and

displaying each responsive image with the respective annotation data in the search result.

26. The mobile device of claim 24, wherein providing the search result includes:

generating annotation data for the search result, where each responsive image is an area of actionable content associated with a visual cue, each visual cue being actionable to initiate a mobile application from which the responsive image was taken; and

displaying the annotation data with the search result.