Image search query predictions by a keyboard

• US 10,664,157 B2
• Filed: 08/24/2016
• Issued: 05/26/2020
• Est. Priority Date: 08/03/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Kong, Nicholas Chi-Yuen, Millius, Sebastian, Greenberg, Alexa
• Primary Examiner(s): Xia, Xuyang
• Application Number:
  US15/246,091
• Publication Number:
  US 20180039406A1
• Time in Patent Office: 1,371 Days
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/3323 using document space presentation or visualization, e.g. category, hierarchy or range presentation and selection
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 3/0237 using prediction or retrieval techniques
• CPC Class Codes: G06F 3/03547 Touch pads, in which fingers can move on a surface
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces

Abstract

A computing device is described that includes at least one processor and a memory including instructions that when executed cause the at least one processor to output, for display, a graphical keyboard comprising a plurality of keys, and determine, based on an indication of a selection of one or more keys from the plurality of keys, text of an electronic communication. The instructions, when executed, further cause the at least one processor to generate, based at least in part on the text, an image based search query, and output, for display, within the graphical keyboard, a graphical indication to indicate that the computing device generated the image based search query.

Claims

1. A method comprising:

outputting, by a keyboard application executing at a computing device, for display, a graphical keyboard comprising a plurality of keys;

determining, by the keyboard application, based on an indication of a selection of one or more keys from the plurality of keys, text of an electronic communication;

outputting, by the keyboard application, for display within an input region of the graphical keyboard, the text;

generating, by the keyboard application, based at least in part on the text, one or more words suggested for text entry;

outputting, by the keyboard application, for display within a suggestion region of the graphical keyboard, the one or more words suggested for text entry;

responsive to determining an end of the text associated with the electronic communication, generating, by the keyboard application, based at least in part on the text, an image based search query;

responsive to generating the image based search query, outputting, by the keyboard application, for display within the graphical keyboard and in place of the suggestion region, a first graphical indication to indicate that the keyboard application generated the image based search query;

responsive to receiving an indication of a selection of the first graphical indication, initiating, by the keyboard application, based on the image based search query, an image based search; and

outputting, by the keyboard application, for display within the graphical keyboard, a second graphical indication of one or more image based search results obtained from the image based search.

2. The method of claim 1, further comprising:

determining, by the keyboard application, the end of the text in response to determining that a last key of the selection of one or more keys corresponds to a punctuation key associated with a punctuation character.

3. The method of claim 1, further comprising:

determining, by the keyboard application, the end of the text in response to determining that a last key of the selection of one or more keys corresponds to a send key of the graphical keyboard to send the electronic communication.

4. The method of claim 1, wherein generating the image based search query comprises:

determining, by the keyboard application, based on at least a portion of the text, one or more words;

determining, by the keyboard application, based on the one or more words and from a local model of searchable entities, a score assigned to a searchable entity indicating a probability that the searchable entity is relevant to the one or more words, the local model being associated with a current location of the computing device;

responsive to determining the score assigned to the searchable entity satisfies a threshold, identifying, by the keyboard application, the searchable entity; and

generating, by the keyboard application, based on the searchable entity, the image based search query.

5. The method of claim 4, wherein the local model is a first local model, the method further comprising:

responsive to determining a change in the current location of the computing device from a first location to a second location;

obtaining, by the keyboard application, from a remote computing system, a second model of searchable entities, the second model being associated with the second location; and

replacing, by the keyboard application, the first local model with the second model.

6. The method of claim 1, wherein generating the image based search query comprises:

determining, by the keyboard application, based on at least a portion of the text, one or more words;

determining, by the keyboard application, based on the one or more words and from a global model of searchable entities, a score assigned to a searchable entity indicating a probability that the searchable entity is relevant to the one or more words, the global model being associated with a geographical region that includes respective locations associated with two or more models of searchable entities;

responsive to determining the score assigned to the searchable entity satisfies a threshold, identifying, by the computing device, the searchable entity; and

generating, by the keyboard application, based on the searchable entity, the image based search query.

7. The method of claim 1, wherein generating the image based search query comprises:

determining, by the keyboard application, based on at least a portion of the text, one or more words;

determining, by the keyboard application, based on the one or more words and from at least one of a global model of searchable entities or a local model of searchable entities, a score assigned to a searchable entity indicating a probability that the searchable entity is relevant to the one or more words, the global model being associated with a geographical region that includes respective locations associated with two different local models of searchable entities including the local model of searchable entities;

modifying, by the keyboard application, based on contextual information and from a machine learned model, the score assigned to the searchable entity; and

after modifying the score assigned to the searchable entity and responsive to determining the score assigned to the searchable entity satisfies a threshold, generating, by the keyboard application, based on the searchable entity, the image based search query,

wherein the contextual information includes information associated with an electronic conversation that includes the electronic communication and one or more other electronic communications that have been sent or received by the computing device.

8. The method of claim 1, wherein the one or more image based search results include at least one dynamic image and the at least one dynamic image is output for display as a rendered animation of the at least one dynamic image.

9. The method of claim 1, wherein generating the image based search query comprises:

identifying, by the keyboard application, based at least in part on the text, a searchable entity;

determining, by the keyboard application, based at least in part on the searchable entity, a curated category of image based search associated with the searchable entity; and

generating, by the keyboard application, from the curated category of image based search, the image based search query, wherein the searchable entity comprises a trigger phrase, and wherein the curated category of image based search associated with the searchable entity is further determined based at least in part on a current context of the computing device.

10. The method of claim 1, wherein outputting the second graphical indication comprises:

outputting, for display within the graphical keyboard and in place of the plurality of keys of the graphical keyboard, the second graphical indication of the one or more image based search results obtained from the image based search.

11. A method comprising:

invoking, by a first application executing at a computing device, for receiving first text input at an input region of a graphical user interface of the first application, a keyboard application that is configured to provide the graphical user interface with a graphical keyboard comprising a plurality of keys, the plurality of keys including a search key and one or more character keys;

responsive to detecting a user selection of the search key, outputting, by the keyboard application, for display adjacent to the input region of the graphical user interface and within the graphical keyboard, a search box for receiving additional text input that forms the basis for an image based search query;

determining, by the keyboard application, based on an indication of a selection of one or more of the character keys from the plurality of keys, the additional text input that forms the basis for the image based search query;

generating, by the keyboard application, based at least in part on the additional text input, an image based search query;

responsive to detecting a selection of a return key from the plurality of keys, performing an image based search based on the image based search query, wherein the search key is different than the return key; and

outputting, by the keyboard application, for display adjacent to the search box and in place of the one or more character keys of the graphical keyboard, a carousel of image based search results returned from the image based search performed based on the image based search query and selectable text associated with additional image based search results.

12. The method of claim 11, further comprising:

responsive to detecting a user selection of a particular curated category of image based searches, replacing the image based search results that are output for display in the carousel with different search results that are associated with the particular curated category.

13. The method of claim 11, further comprising:

after detecting a user selection of a particular image based search result from the image based search results that are output for display in the carousel, inserting, by the keyboard application, the particular image based search result in the input region of the graphical user interface,

wherein;

the search key is positioned between the input region of the graphical user interface and the one or more character keys;

the search key is further positioned next to a suggestion region of the graphical keyboard; and

outputting the search box for display comprises replacing, by the keyboard application, the suggestion region with the search box.

14. The method of claim 11, wherein the image based search results comprise animated images or videos and the method further comprises:

while displaying the image based search results in the carousel, automatically playing, by the keyboard application, the animated images or videos from within the carousel.

15. The method of claim 11, the method further comprising:

responsive to detecting a user selection of a particular curated category of image based searches, sending, by the keyboard application, to the first application, the particular image based search result for display by the first application in the input region of the graphical user interface.

16. The method of claim 15, the method further comprising:

sending, by the first application, to a different computing device, data representing the particular image based search result.

17. A non-transitory computer-readable storage medium storing instructions for a first application and a keyboard application, wherein, when executed by one or more processors, the instructions for the keyboard application cause the one or more processors to:

output, for receiving first text input at an input region of a graphical user interface of the first application, a graphical user interface with a graphical keyboard comprising a plurality of keys, the plurality of keys including a search key and one or more character keys;

output, responsive to detecting a user selection of the search key, for display adjacent to the input region of the graphical user interface and within the graphical keyboard, a search box for receiving additional text input that forms the basis for an image based search query;

determine, based on an indication of a selection of one or more of the character keys from the plurality of keys, the additional text input that forms the basis for the image based search query;

generate, based at least in part on the additional text input, an image based search query;

responsive to detecting a selection of a return key from the plurality of keys, initiate an image based search based on the image based search query, wherein the search key is different than the return key; and

output, for display adjacent to the search box and in place of the one or more character keys of the graphical keyboard, a carousel of image based search results returned from the image based search performed based on the image based search query and selectable text associated with additional image based search results.