Query modification based on non-textual resource context

• US 9,830,391 B1
• Filed: 06/24/2014
• Issued: 11/28/2017
• Est. Priority Date: 06/24/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Bakir, Gokhan H., Behzadi, Behshad
• Primary Examiner(s): Boccio, Vincent F
• Application Number:
  US14/313,559
• Time in Patent Office: 1,253 Days
• Field of Search: 707723, 707707, 707759, 707767
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/3322 using system suggestions G06F16/3325 takes precedence
• CPC Class Codes: G06F 16/3325 Reformulation based on results of preceding query
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, modifying queries based on non-textual content. In one aspect, a method includes receiving, from a user device, a query including a plurality of terms; determining active non-textual data displayed in an application environment on the user device; determining, from the non-textual textual data, modification data for the query; generating a set of modified queries based on the query and the modification parameters; scoring the modified queries according to one or more scoring criteria; selecting one of the modified queries based on the scoring; and providing, to the user device, search results responsive to the selected modified query.

Claims

1. A method implemented in a data processing apparatus, the method comprising:

receiving, at a data processing apparatus including one or more hardware processors, a query from a user device, the query including a plurality of terms input by a user, the user device displaying plurality of active non-textual data elements in an active window of an application environment when the user input the query;

determining, by the data processing apparatus, that the query is an ambiguous query and that the user device is displaying the active non-textual data elements that were displayed when the query was input and in response;

identifying, by the data processing apparatus, the active non-textual data displayed in the application environment on the user device;

automatically determining, by the data processing apparatus, for each of the active non-textual data elements, modification data from the active non-textual data element for the query;

automatically generating, by the data processing apparatus, for each modification data for each active non-textual data element, a set of modified queries based on the query and the modification data;

automatically scoring, by the data processing apparatus, the modified queries according to one or more scoring criteria;

automatically selecting, by the data processing apparatus, one of the modified queries based on the scoring; and

providing, by the data processing apparatus, to the user device, search results responsive to the selected modified query.

2. The method of claim 1, wherein determining the plurality of active non-textual data elements displayed in an active window of an application environment on the user device comprises:

determining an application context to which the query belongs, the application context corresponding to an active application instantiated on the user device; and

determining non-textual data elements displayed in the active window of the application environment of the application.

3. The method of claim 2, wherein determining the plurality of active non-textual data elements displayed in an active window of an application environment of the application comprises determining one or more images displayed in the application environment and selected in the application environment.

4. The method of claim 3, wherein determining, from the plurality of active non-textual data elements, modification data for the query comprises:

determining, from the selected images, labels that describe subject matter of the one or more images; and

for each label, generating a modified query based on the query and the label.

5. The method of claim 4, wherein generating a modified query based on the query and the label comprises concatenating the modified query with the label.

6. The method of claim 3, wherein determining, from the plurality of active non-textual data elements, modification data for the query comprises:

determining, from the one or more selected images, entity text describing entities depicted in the one or more images; and

for each entity, generating a modified query based on the query and the entity.

7. The method of claim 6, wherein generating a modified query based on the query and the entity text describing the entity comprises revising one or more terms of the query based on the entity text describing the entity.

8. The method of claim 6, wherein generating a modified query based on the query and the entity text describing the entity comprises:

determining a similarity of the text of the query and the entity text to each of a plurality of prior queries stored in a query log; and

selecting, form the query log, prior queries for which the similarity measure meets a threshold similarity value.

9. The method of claim 2, wherein determining the plurality of active non-textual data elements displayed in an active window of an application environment of the application comprises determining video frames displayed in the application environment and selected in the application environment.

10. The method of claim 9, wherein determining, from the plurality of active non-textual data elements, modification data for the query comprises:

processing a video frame for structured video data;

detecting structured video data;

deriving text from the structured video data; and

generating a modified query based on the query and the text.

11. The method of claim 10, wherein the structure data includes QR codes.

12. The method of claim 10, wherein deriving text from the structured video data comprises detecting entity text that describes an entity.

13. The method of claim 9, wherein determining, from the plurality of active non-textual data elements, modification data for the query comprises:

performing optical character recognition on the one or more video frames to derive text from the one or more video frames; and

generating one or more modified queries based on the query and the text.

14. The method of claim 13, wherein deriving text from the one or more video frames comprises detecting entity text from the text that describes an entity.

15. The method of claim 14, wherein generating a modified query based on the query and the entity text describing the entity comprises revising one or more terms of the query based on the entity text describing the entity.

16. The method of claim 14, wherein generating a modified query based on the query and the entity text describing the entity comprises:

determining a similarity of the text of the query and the entity text to each of a plurality of prior queries stored in a query log; and

selecting, form the query log, prior queries for which the similarity measure meets a threshold similarity value.

17. The method of claim 1, wherein scoring the modified queries according to one or more scoring criteria comprises, for each modified query:

receiving, from a search process, search data describing relevance scores of resources that are determined to be relevant to the modified query; and

generating, from the search data, a score for the modified query.

18. A system, comprising:

a data processing apparatus, including one or more hardware processors; and

one or more data stores in communication with the data processing apparatus and storing instructions that upon execution causes the data processing apparatus to perform operations comprising;

receiving, from a user device displaying a plurality of active non-textual data elements in an active window of an application environment, a query including a plurality of terms, the query input when the user device was displaying the active window;

determining, by the data processing apparatus, that the query is an ambiguous query and that the user device is displaying the active non-textual data elements that were displayed when the query was input and in response;

identifying, by the data processing apparatus, active non-textual data displayed in the application environment on the user device;

automatically determining, by the data processing apparatus, for each of the active non-textual data elements, modification data from the active non-textual data element for the query;

automatically generating, by the data processing apparatus, for each modification data for each active non-textual data element, a set of modified queries based on the query and the modification data;

automatically scoring, by the data processing apparatus, the modified queries according to one or more scoring criteria;

automatically selecting, by the data processing apparatus, one of the modified queries based on the scoring; and

providing, by the data processing apparatus, to the user device, search results responsive to the selected modified query.

19. A non-transitory computer storage medium encoded with instructions that when executed by one or more data processing apparatus cause the one or more data processing apparatus to perform operations comprising:

receiving, from a user device displaying a plurality of active non-textual data elements in an active window of an application environment, a query including a plurality of terms, the query input when the user device was displaying the active window;

determining, by the data processing apparatus, that the query is an ambiguous query and that the user device is displaying the active non-textual data elements that were displayed when the query was input and in response;

identifying, by the data processing apparatus, the active non-textual data displayed in the application environment on the user device;

automatically determining, by the data processing apparatus, for each of the active non-textual textual data elements, modification data from the active non-textual data element for the query;

automatically generating, by the data processing apparatus, for each modification data for each active non-textual data element, a set of modified queries based on the query and the modification data;

automatically scoring, by the data processing apparatus, the modified queries according to one or more scoring criteria;

automatically selecting, by the data processing apparatus, one of the modified queries based on the scoring; and

providing, by the data processing apparatus, to the user device, search results responsive to the selected modified query.