CONTEXTUAL VOICE SEARCH SUGGESTIONS

• US 20170371885A1
• Filed: 06/27/2016
• Published: 12/28/2017
• Est. Priority Date: 06/27/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Aggarwal, Vikram, Ramchandran, Aparna Rajan
• Granted Patent: US 11,232,136 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 16/3322 using system suggestions G06F16/3325 takes precedence
• CPC Class Codes: G06F 16/334 Query execution G06F16/335 takes precedence
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 9/453 Help systems
• CPC Class Codes: G06V 30/224 of printed characters having additional code marks or containing code marks

Abstract

Methods, systems, and apparatus for receiving user input that invokes digital assistant functionality; obtaining screen data indicating content displayed on a screen of the computing device; determining a classification for an entity referenced in the content indicated by the screen data; determining a suggested request that refers to the entity based on the determined classification; and providing the suggested request in response to receiving the user input that invokes the digital assistant functionality.

Claims

1. A method performed by one or more computing devices, the method comprising:

receiving, by the one or more computing devices, user input that invokes digital assistant functionality;

obtaining, by the one or more computing devices, screen data indicating content displayed on a screen of the computing device;

determining, by the one or more computing devices, a classification for an entity referenced in the content indicated by the screen data;

determining, by the one or more computing devices, a suggested request that refers to the entity based on the determined classification; and

providing, by the one or more computing devices, the suggested request in response to receiving the user input that invokes the digital assistant functionality.

2. The method of claim 1, wherein determining the classification comprises selecting an entity type for the entity from among a plurality of entity types.

3. The method of claim 1, wherein determining the suggested request comprises:

determining a pronoun for the entity; and

generating a suggested query that includes the pronoun to refer to the entity referenced in the content indicated by the screen data.

4. The method of claim 3, determining the pronoun comprises:

accessing mapping data that indicates pronouns for a plurality of entities; and

selecting a pronoun that the mapping data indicates corresponding to the entity.

5. The method of claim 1, wherein determining the suggested request comprises:

identifying one or more query types associated with the classification;

selecting a particular query type from the one or more query types; and

determining a suggested query of the particular query type.

6. The method of claim 1, further comprising determining one or more attributes associated with the entity type, and determining a query that asks about one of the attributes.

7. The method of claim 1, wherein determining the suggested request comprises determining a suggested request to initiate communication with a person or to navigate to a location.

8. The method of claim 1, further comprising recognizing one or more terms displayed on the screen of the computing device using optical character recognition on the screen data;

and

wherein determining the classification comprises determining a semantic classification for the recognized one or more terms.

9. The method of claim 8, wherein recognizing the one or more terms comprises recognizing multiple different terms;

wherein the method further comprises ranking the recognized multiple different terms; and

wherein determining the suggested request comprises determining a suggested request that refers to the entity corresponding to the highest-ranking term.

10. The method of claim 8, further comprising determining text formatting characteristics of the multiple different terms indicated by the screen data;

wherein ranking the recognized multiple different terms comprises ranking the multiple different terms based at least in part on the determined formatting characteristics.

11. The method of claim 8, wherein ranking the recognized multiple different terms comprises ranking the multiple different terms based at least in part on (i) classifications assigned to the multiple different terms, (ii) sizes at which the multiple different terms are displayed, (iii) frequencies that the multiple different terms are displayed on the screen, (iv) quantities of actions that can be generated for the multiple different terms, or (v) measures of previous suggestions relating to the multiple different terms that were provided and acted upon by a user.

12. The method of claim 1, further comprising determining that the screen references only a single entity or only a single entity of a particular entity type;

and

wherein providing the suggested request is performed in response to determining only a single entity or only a single entity of a particular entity type.

13. A system comprising:

one or more computing devices and more or more storage devices storing instructions that are operable, when executed by the one or more computing devices, to cause the one or more computing devices to perform operations comprising;

receiving, by the one or more computing devices, user input that invokes digital assistant functionality;

obtaining, by the one or more computing devices, screen data indicating content displayed on a screen of the computing device;

determining, by the one or more computing devices, a classification for an entity referenced in the content indicated by the screen data;

determining, by the one or more computing devices, a suggested request that refers to the entity based on the determined classification; and

providing, by the one or more computing devices, the suggested request in response to receiving the user input that invokes the digital assistant functionality.

14. The system of claim 13, wherein determining the classification comprises selecting an entity type for the entity from among a plurality of entity types.

15. The system of claim 13, wherein determining the suggested request comprises:

determining a pronoun for the entity; and

generating a suggested query that includes the pronoun to refer to the entity referenced in the content indicated by the screen data.

16. The system of claim 15, determining the pronoun comprises:

accessing mapping data that indicates pronouns for a plurality of entities; and

selecting a pronoun that the mapping data indicates corresponding to the entity.

17. The system of claim 13, wherein determining the suggested request comprises:

identifying one or more query types associated with the classification;

selecting a particular query type from the one or more query types; and

determining a suggested query of the particular query type.

18. The system of claim 13, further comprising determining one or more attributes associated with the entity type, and determining a query that asks about one of the attributes.

19. The system of claim 13, wherein determining the suggested request comprises determining a suggested request to initiate communication with a person or to navigate to a location.

20. One or more non-transitory computer-readable storage media encoded with a software program, the program comprising instructions that, if executed by one or more computing devices, cause the one or more computing devices to perform operations comprising:

receiving, by the one or more computing devices, user input that invokes digital assistant functionality;

obtaining, by the one or more computing devices, screen data indicating content displayed on a screen of the computing device;

determining, by the one or more computing devices, a classification for an entity referenced in the content indicated by the screen data;

determining, by the one or more computing devices, a suggested request that refers to the entity based on the determined classification; and

providing, by the one or more computing devices, the suggested request in response to receiving the user input that invokes the digital assistant functionality.