Detection and ranking of entities from mobile onscreen content

• US 10,080,114 B1
• Filed: 08/31/2017
• Issued: 09/18/2018
• Est. Priority Date: 07/11/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Sharifi, Matthew, Petrou, David
• Primary Examiner(s): Saini, Amandeep Singh
• Application Number:
  US15/692,682
• Time in Patent Office: 383 Days
• Field of Search: 382305
• CPC Class Codes: G06F 16/156 Query results presentation
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/957 Browsing optimisation, e.g. caching or content distillation
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 40/134 Hyperlinking
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06Q 10/1097 Task assignment
• CPC Class Codes: G06V 30/416 Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04W 4/18 Information format or content conversion, e.g. adaptation by the network of the transmitted or received information for the purpose of wireless delivery to users or terminals

Abstract

Systems and methods are provided for detecting and ranking entities identified in screen content displayed on a mobile device. For example, a method includes receiving an image captured from a mobile device display for a mobile application and determining a window that includes a chronological set of images, the images each representing a respective screen captured from a display of a mobile device and having an associated timestamp. The method also includes identifying entities appearing in images in a first portion of the window using text for images in a remaining portion of the window as context to disambiguate ambiguous entity references.

Claims

1. A mobile device comprising:

at least one processor; and

memory storing;

a data store of entities,

recognized items identified by performing recognition on each of a plurality of images of screens captured on the mobile device, each recognized item being associated with an image of the plurality of images, and

instructions that, when executed by the at least one processor, cause the mobile device to;

select a set of images from the plurality of images, the set representing a chronological window of time, each image in the set having a timestamp within the chronological window, and

identify entities from the data store in the recognized items of images in a first portion of the window using recognized items for images in a remaining portion of the window as context to disambiguate ambiguous entity references.

2. The mobile device of claim 1, wherein the first portion is a middle portion and the remaining portion includes a front portion of images having a timestamp prior to timestamps for images in the middle portion and a back portion of images having a timestamp after the timestamps for the images in the middle portion.

3. The mobile device of claim 1, wherein the memory further stores, for each of the plurality of images, a mobile application associated with the image and selecting the set of images includes:

using the associated mobile applications to determine the set.

4. The mobile device of claim 1, wherein the memory further stores, for each of the plurality of images, a mobile application associated with the image and selecting the set of images includes:

determining that a next chronological image is associated with a mobile application that differs from a mobile application associated with a last image in the set; and

closing the set in response to the determining.

5. The mobile device of claim 4, wherein the first portion is a last portion and the remaining portion includes images having a timestamp prior to timestamps for images in the last portion.

6. The mobile device of claim 1, wherein the memory further stores, for each of the plurality of images, a mobile application associated with the image and selecting the set of images includes:

determining that a next chronological image is associated with a first mobile application that differs from a second mobile application associated with a last image in the set;

determining that the first mobile application launched the second mobile application; and

including the next chronological image in the set.

7. The mobile device of claim 1, wherein the instructions further include instructions that, when executed by the at least one processor, cause the mobile device to:

determine, from the entities in the first portion, at least one entity that is an outlier;

and

store, in a user profile in the memory, the entities identified in the first portion except for the at least one entity.

8. The mobile device of claim 1, wherein the instructions further include instructions that, when executed by the at least one processor, cause the mobile device to:

calculate a rank for a first entity of the entities identified in the first portion based on an amount of time the first entity appears in the images.

9. The mobile device of claim 8, wherein calculating the rank includes:

determining that a location of the first entity remains constant across images; and

downgrading the rank in response to the determining.

10. A computer system comprising:

at least one processor; and

memory storing instructions that, when executed by the at least one processor, cause the system to;

receive, from a mobile device, an image of a screen displayed on the mobile device,

identify recognized items for the image by performing recognition on the image,

repeat the receiving and identifying for a plurality of images,

select a set of images from the plurality of images, the set representing a chronological window of time, each image in the set having a timestamp within the chronological window, and

identify entities appearing in the recognized items for images in a first portion of the window using the recognized items for images in a remaining portion of the window as context to disambiguate ambiguous entity references.

11. The system of claim 10, wherein the memory further stores instructions that, when executed by the at least one processor, causes the system to:

determine that a first candidate entity and a second candidate entity each correspond to a same recognized item for a first image in the first portion of the window;

determine that the first candidate entity shares a category with a mobile application associated with the first image; and

select the first candidate entity over the second candidate entity based on sharing the category.

12. The system of claim 11, wherein the second candidate entity has a higher prior probability than the first candidate entity.

13. The system of claim 10, wherein the memory further stores instructions that, when executed by the at least one processor, causes the system to:

determine that a next chronological image is associated with a mobile application that differs from a mobile application associated with a last image in the set; and

close the set in response to the determining.

14. The system of claim 10, wherein the memory further stores instructions that, when executed by the at least one processor, causes the system to:

determine that a next chronological image is associated with a first mobile application that differs from a second mobile application associated with a last image in the set;

determine that the first mobile application launched the second mobile application; and

include the next chronological image in the set.

15. The system of claim 10, wherein the instructions further include instructions that, when executed by the at least one processor, cause the system to:

calculate a rank for a first entity of the entities identified in the first portion based on an amount of time the first entity appears in the images; and

store the first entity with the rank in the memory.

16. The system of claim 15, wherein calculating the rank includes:

determining that a location of the first entity remains constant across images; and

downgrading the rank in response to the determining.

17. A method comprising:

selecting a set of images from a plurality of images of mobile application user interfaces captured while the mobile application was active, the set representing a chronological window of time, each image in the set having a timestamp within the chronological window;

identifying recognized items in the set of images by performing recognition on each of the images, each recognized item being associated with an image of the set of images; and

identifying entities from an entity data store in the recognized items of images in a first portion of the window using recognized items for images in a remaining portion of the window as context to disambiguate ambiguous entity references.

18. The method of claim 17, further comprising:

calculating a rank for a first entity of the entities identified in the first portion based on an amount of time the first entity appears in the images.

19. The method of claim 18, wherein calculating the rank includes:

determining that a location of the first entity remains constant across images; and

downgrading the rank in response to the determining.

20. The method of claim 17, wherein the images are captured responsive to a touch event and responsive to a transition from one application to another.