Entity disambiguation in a mobile environment

• US 10,652,706 B1
• Filed: 03/14/2019
• Issued: 05/12/2020
• Est. Priority Date: 07/11/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Sharifi, Matthew, Petrou, David
• Primary Examiner(s): Saini, Amandeep
• Application Number:
  US16/353,522
• Time in Patent Office: 425 Days
• Field of Search: 382305
• CPC Class Codes: G06F 16/156 Query results presentation
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/957 Browsing optimisation, e.g. caching or content distillation
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 40/134 Hyperlinking
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06Q 10/1097 Task assignment
• CPC Class Codes: G06V 30/416 Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04W 4/18 Information format or content conversion, e.g. adaptation by the network of the transmitted or received information for the purpose of wireless delivery to users or terminals

Abstract

Systems and methods are provided for disambiguating entities in a mobile environment and using the disambiguated entity to perform actions. An example method includes identifying an ambiguous entity reference in a screen shot of an interface generated by a first mobile application executing on a mobile device, determining a chronological window of content captured prior to the screen shot, and identifying a plurality of entities appearing in the chronological window of content. The method also includes disambiguating the ambiguous entity reference using the plurality of entities appearing in the chronological window and using the disambiguated entity to perform an action in a second mobile application executing on the mobile device.

Claims

1. A method comprising:

identifying an ambiguous entity reference in a screen shot of an interface generated by a first mobile application executing on a mobile device;

determining a chronological window of content captured prior to the screen shot;

identifying a plurality of entities appearing in the chronological window of content;

disambiguating the ambiguous entity reference using the plurality of entities appearing in the chronological window; and

using the disambiguated entity to perform an action in a second mobile application executing on the mobile device.

2. The method of claim 1, wherein the screen shot has a first timestamp and the chronological window of content includes content associated with respective timestamps that occur prior to the first timestamp.

3. The method of claim 1, wherein the chronological window of content represents a plurality of screen capture events.

4. The method of claim 1, wherein the chronological window of content represents a predetermined quantity of screen capture events.

5. The method of claim 1, wherein the chronological window of content represents a time period.

6. The method of claim 1, wherein the screen shot is presented in a carousel of screen capture images.

7. The method of claim 1, wherein recognized content in the chronological window of content is stitched together to form a document.

8. The method of claim 1, further comprising using a category of the first mobile application with the plurality of entities appearing in the chronological window to disambiguate the ambiguous entity reference.

9. A mobile device comprising:

at least one processor; and

memory storing instructions that, when executed by the at least one processor, cause the mobile device to;

identify an ambiguous entity reference in a screen shot of an interface generated by a first mobile application executing on the mobile device,

determine a chronological window of content captured prior to the screen shot,

identify a plurality of entities appearing in the chronological window of content,

disambiguate the ambiguous entity reference using the plurality of entities appearing in the chronological window, and

use the disambiguated entity to perform an action in a second mobile application executing on the mobile device.

10. The mobile device of claim 9, wherein the memory further stores an index that includes at least some of the plurality of entities appearing in the chronological window.

11. The mobile device of claim 9, wherein the memory further stores instructions that, when executed by the at least one processor, causes the mobile device to:

determine that a first candidate entity and a second candidate entity each correspond to the ambiguous entity reference;

determine that the first candidate entity shares a category with the first mobile application; and

boost a probability of the first candidate entity based on sharing the category.

12. The mobile device of claim 11, wherein the second candidate entity has a higher prior probability than the first candidate entity.

13. The mobile device of claim 9, wherein the memory further stores instructions that, when executed by the at least one processor, causes the mobile device to:

determine that the disambiguated entity is associated with at least two possible actions; and

provide the at least two actions for selection,

wherein the action performed in the second mobile application is the action selected by a user of the mobile device.

14. The mobile device of claim 9, wherein the chronological window of content represents a plurality of screen capture events.

15. The mobile device of claim 9, wherein the chronological window of content represents a predetermined quantity of screen capture events.

16. The mobile device of claim 9, wherein the chronological window of content represents a time period.

17. The mobile device of claim 9, wherein the screen shot is presented in a carousel of screen capture images.

18. A method comprising:

identifying an ambiguous entity reference in a screen shot of an interface generated by a first mobile application executing on a mobile device;

determining a category of the first mobile application;

disambiguating the ambiguous entity reference using the category of the first mobile application; and

using the disambiguated entity to perform an action in a second mobile application executing on the mobile device.

19. The method of claim 18, further comprising:

determining a chronological window of content captured prior to the screen shot; and

identifying a plurality of entities appearing in the chronological window of content,

wherein the plurality of entities are used with the category of the first mobile application to disambiguate the ambiguous entity reference.

20. The method of claim 18, wherein the screen shot is presented in a carousel of screen capture images.