Mapping images to search queries

• US 10,489,410 B2
• Filed: 04/18/2016
• Issued: 11/26/2019
• Est. Priority Date: 04/18/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Sharifi, Matthew, Petrou, David, Sharma, Abhanshu
• Primary Examiner(s): Ruiz, Angelica
• Application Number:
  US15/131,178
• Publication Number:
  US 20170300495A1
• Time in Patent Office: 1,317 Days
• CPC Class Codes: G06F 16/24522 Translation of natural language queries to structured queries
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/90335 Query processing

Abstract

Methods, systems, and apparatus for receiving a query image, receiving one or more entities that are associated with the query image, identifying, for one or more of the entities, one or more candidate search queries that are pre-associated with the one or more entities, generating a respective relevance score for each of the candidate search queries, selecting, as a representative search query for the query image, a particular candidate search query based at least on the generated respective relevance scores and providing the representative search query for output in response to receiving the query image.

Claims

1. A computer-implemented method comprising:

receiving a query image from a user device;

performing a visual recognition process on the query image to identify non-textual image features in the query image relating to objects present in the image and obtain textual image labels that describe the non-textual image features related to objects present in the image;

accessing data associating terms with entities of a set of known entities and data associating candidate search queries with entities of the set of known entities;

identifying, using the textual image labels obtained for the non-textual image features and the data associating terms with the set of known entities, one or more entities that are associated with the obtained textual image labels describing the non-textual image features related to objects present in the query image, wherein identifying the one or more entities includes comparing the textual image labels to the terms associated with the set of known entities, wherein each known entity in the set of known entities is associated with a respective set of terms;

identifying, for one or more of the entities and from the data associating candidate search queries with the entities of the set of known entities, one or more candidate search queries, wherein the one or more candidate search queries are textual search queries and wherein the one or more candidate search queries are different than the terms associated with the one or more entities;

generating a respective relevance score for each of the candidate search queries;

selecting, as a representative search query for the query image, a particular candidate search query based at least on the generated respective relevance scores; and

providing the representative search query for output in response to receiving the query image.

2. The method of claim 1, wherein generating a respective relevance score for each of the candidate search queries comprises, for each candidate search query:

determining whether a context of the query image matches the candidate search query;

based on the determined match, generating a respective relevance score for the candidate search query.

3. The method of claim 2, wherein determining whether the context of the query image matches the candidate search query comprises determining whether the query image has an associated location that matches the candidate search query.

4. The method of claim 1, further comprising:

receiving a natural language query; and

generating a respective relevance score for each of the candidate search queries based at least on the received natural language query.

5. The method of claim 1, wherein generating a respective relevance score for each of the candidate search queries comprises, for each candidate search query:

generating a search results page using the candidate search query;

analyzing the generated search results page to determine a measure indicative of how interesting and useful the search results page is; and

based on the determined measure, generating a respective relevance score for the candidate search query.

6. The method of claim 1, wherein generating a respective relevance score for each of the candidate search queries comprises, for each candidate search query:

determining a popularity of the candidate search query; and

based on the determined popularity, generating a respective relevance score for the candidate search query.

7. The method of claim 1, wherein receiving one or more entities that are associated with the query image comprises:

obtaining one or more query image labels; and

identifying, for one or more of the query image labels, one or more entities that are pre-associated with the one or more query image labels.

8. The method of claim 7, wherein the one or more query image labels comprise fine-grained image labels.

9. The method of claim 7, wherein the one or more query image labels comprise coarse-grained image labels.

10. The method of claim 7, further comprising generating a respective label score for each of the query image labels.

11. The method of claim 10, wherein a respective label score for a query image label is based at least on a topicality of the query image label.

12. The method of claim 10, wherein a respective label score for a query image label is based at least on how specific the label is.

13. The method of claim 10, wherein a respective label score for a query image label is based at least on a reliability of a backend by which the query image label is obtained from and a calibrated backend confidence score.

14. The method of claim 10, wherein selecting a particular candidate search query based at least on the candidate query scores further comprises selecting a particular candidate search query based at least on the candidate query scores and the label scores.

15. The method of claim 14, wherein selecting a particular candidate search query based at least on the candidate query scores and the label scores comprises:

determining an aggregate score between each label score and associated candidate query score;

ranking the determined aggregate scores; and

selecting a particular candidate search query that corresponds to a highest ranked score.

16. The method of claim 1, wherein selecting a particular candidate search query based at least on the candidate query scores comprises:

ranking the relevance scores for the candidate search queries; and

selecting a particular candidate search query that corresponds to a highest ranked score.

17. The method of claim 16, wherein providing the representative search query for output in response to receiving the query image further comprises providing a predetermined number of candidate search queries that correspond to the predetermined number of highest ranked scores for output in response to receiving the query image.

18. The method of claim 1, further comprising:

generating a search results page using the representative search query; and

providing the generated search results page for output in response to receiving the query image.

19. A system comprising:

one or more computers and more or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

receiving a query image from a user device;

performing a visual recognition process on the query image to identify non-textual image features in the query image relating to objects present in the image and obtain textual image labels that describe the non-textual image features related to objects present in the image;

accessing data associating terms with entities of a set of known entities and data associating candidate search queries with entities of the set of known entities;

identifying, using the textual image labels obtained for the non-textual image features and the data associating terms with the set of known entities, one or more entities that are associated with the obtained textual image labels describing the non-textual image features related to objects present in the query image, wherein identifying the one or more entities includes comparing the textual image labels to the terms associated with the set of known entities, wherein each known entity in the set of known entities is associated with a respective set of terms;

identifying, for one or more of the entities and from the data associating candidate search queries with the entities of the set of known entities, one or more candidate search queries, wherein the one or more candidate search queries are textual search queries and wherein the one or more candidate search queries are different than the terms associated with the one or more entities;

generating a respective relevance score for each of the candidate search queries;

selecting, as a representative search query for the query image, a particular candidate search query based at least on the generated respective relevance scores; and

providing the representative search query for output in response to receiving the query image.

20. A computer-readable storage device encoded with a computer program, the program comprising instructions that, if executed by one or more computers, cause the one or more computers to perform operations comprising:

receiving a query image from a user device;

performing a visual recognition process on the query image to identify non-textual image features in the query image relating to objects present in the image and obtain textual image labels that describe the non-textual image features related to objects present in the image;

accessing data associating terms with entities of a set of known entities and data associating candidate search queries with entities of the set of known entities;

identifying, using the textual image labels obtained for the non-textual image features and the data associating terms with the set of known entities, one or more entities that are associated with the obtained textual image labels describing the non-textual image features related to objects present in the query image, wherein identifying the one or more entities includes comparing the textual image labels to the terms associated with the set of known entities, wherein each known entity in the set of known entities is associated with a respective set of terms;

identifying, for one or more of the entities and from the data associating candidate search queries with the entities of the set of known entities, one or more candidate search queries, wherein the one or more candidate search queries are textual search queries and wherein the one or more candidate search queries are different than the terms associated with the one or more entities;

generating a respective relevance score for each of the candidate search queries;

selecting, as a representative search query for the query image, a particular candidate search query based at least on the generated respective relevance scores; and

providing the representative search query for output in response to receiving the query image.