Identifying textual terms in response to a visual query

• US 9,372,920 B2
• Filed: 01/13/2015
• Issued: 06/21/2016
• Est. Priority Date: 08/08/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Bengio, Samy, Petrou, David
• Primary Examiner(s): Beausoliel, Jr., Robert
• Assistant Examiner(s): Pham, Michael
• Application Number:
  US14/596,081
• Publication Number:
  US 20150193528A1
• Time in Patent Office: 525 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/3334 Selection or weighting of terms from queries, including natural language queries
• CPC Class Codes: G06F 16/338 Presentation of query results
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information

Abstract

A method, system, and computer readable storage medium is provided for identifying textual terms in response to a visual query is provided. A server system receives a visual query from a client system. The visual query is responded to as follows. A set of image feature values for the visual query is generated. The set of image feature values is mapped to a plurality of textual terms, including a weight for each of the textual terms in the plurality of textual terms. The textual terms are ranked in accordance with the weights of the textual terms. Then, in accordance with the ranking the textual terms, one or more of the ranked textual terms are sent to the client system.

Claims

1. A computer-implemented method comprising:

receiving a query image;

obtaining a set of image features that are associated with the query image;

obtaining one or more image feature values for the set of image features;

providing one or more of the image feature values to multiple image relevance models that are each associated with a different query term, each image relevance model being trained to output a score that reflects a relevance of a given query image, from which the image feature values were obtained, to the query term associated with the image relevance model;

obtaining, from each of the multiple image relevance models, the score that reflects the relevance of the query image to the query term associated with the image relevance model;

selecting a subset of the query terms that are associated with the multiple image relevance models based at least on the scores; and

providing, for output, one or more of the query terms of the subset of the query terms.

2. The computer-implemented method of claim 1, wherein obtaining the one or more image feature values comprises obtaining a vector of image feature values.

3. The computer-implemented method of claim 2, further comprising:

for each query term, obtaining a weight for the query term by applying the vector of image feature values to a respective image relevance vector for the query term, wherein each component of the image relevance vector indicates a relative importance of each corresponding component in the vector of image feature values in determining whether the query term is relevant,

wherein selecting the subset of the query terms is based on the respective weight for each query term.

4. The computer-implemented method of claim 3, further comprising mapping the vector of image feature values to one or more query term and weight pairs.

5. The computer-implemented method of claim 2, further comprising, for each query term, applying the vector of image features values to a respective image relevance model, the respective image relevance model including the respective image relevance vector.

6. The computer-implemented method of claim 2, further comprising:

identifying a matrix of image relevance vectors; and

multiplying the vector of image features values by the matrix of image relevance vectors, wherein each row of the matrix of image relevance vectors corresponds to a respective query term of the set of query terms.

7. The computer-implemented method of claim 6, further comprising mapping each vector of image feature values to a respective query term and weight pair.

8. The computer-implemented method of claim 6, further comprising, for each query term, obtaining the weight for the query term based on multiplying the vector of image features values by the matrix of image relevance vectors.

9. The computer-implemented method of claim 8, further comprising ranking each query term of the subset of the query terms based on the respective weight.

10. A system comprising:

one or more computers and one or more storage devices storing instructions that are configured to, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

receiving a query image;

obtaining a set of image features that are associated with the query image;

obtaining one or more image feature values for the set of image features;

providing one or more of the image feature values to multiple image relevance models that are each associated with a different query term, each image relevance model being trained to output a score that reflects a relevance of a given query image, from which the image feature values were obtained, to the query term associated with the image relevance model;

obtaining, from each of the multiple image relevance models, the score that reflects the relevance of the query image to the query term associated with the image relevance model;

selecting a subset of the query terms that are associated with the multiple image relevance models based at least on the scores; and

providing, for output, one or more of the query terms of the subset of the query terms.

11. The system of claim 10, wherein obtaining the one or more image feature values comprises obtaining a vector of image feature values.

12. The system of claim 11, the operations further comprising:

for each query term, obtaining a weight for the query term by applying the vector of image feature values to a respective image relevance vector for the query term, wherein each component of the image relevance vector indicates a relative importance of each corresponding component in the vector of image feature values in determining whether the query term is relevant,

wherein selecting the subset of the query terms is based on the respective weight for each query term.

13. The system of claim 12, the operations further comprising mapping the vector of image feature values to one or more query term and weight pairs.

14. The system of claim 12, the operations further comprising, for each query term, applying the vector of image features values to a respective image relevance model, the respective image relevance model including the respective image relevance vector.

15. The system of claim 12, the operations further comprising:

identifying a matrix of image relevance vectors; and

multiplying the vector of image features values by the matrix of image relevance vectors, wherein each row of the matrix of image relevance vectors corresponds to a respective query term of the set of query terms.

16. The system of claim 15, the operations further comprising mapping each vector of image feature values to a respective query term and weight pair.

17. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

receiving a query image;

obtaining a set of image features that are associated with the query image;

obtaining one or more image feature values for the set of image features;

providing one or more of the image feature values to multiple image relevance models that are each associated with a different query term, each image relevance model being trained to output a score that reflects a relevance of a given query image, from which the image feature values were obtained, to the query term associated with the image relevance model;

obtaining, from each of the multiple image relevance models, the score that reflects the relevance of the query image to the query term associated with the image relevance model;

selecting a subset of the query terms that are associated with the multiple image relevance models based at least on the scores; and

providing, for output, one or more of the query terms of the subset of the query terms.

18. The computer-readable medium of claim 17, wherein obtaining the one or more image feature values comprises obtaining a vector of image feature values.

19. The computer-readable medium of claim 18, the operations further comprising:

for each query term, obtaining a weight for the query term by applying the vector of image feature values to a respective image relevance vector for the query term, wherein each component of the image relevance vector indicates a relative importance of each corresponding component in the vector of image feature values in determining whether the query term is relevant,

wherein selecting the subset of the query terms is based on the respective weight for each query term.

20. The computer-readable medium of claim 19, the operations further comprising mapping the vector of image feature values to one or more query term and weight pairs.