Identifying Textual Terms in Response to a Visual Query

• US 20140046935A1
• Filed: 08/08/2012
• Published: 02/13/2014
• Est. Priority Date: 08/08/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Bengio, Samy, Petrou, David
• Granted Patent: US 8,935,246 B2
• Time in Patent Office: Days
• US Class Current: 707/723
• CPC Class Codes: G06F 16/3334 Selection or weighting of terms from queries, including natural language queries
• CPC Class Codes: G06F 16/338 Presentation of query results
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information

Abstract

A method, system, and computer readable storage medium is provided for identifying textual terms in response to a visual query is provided. A server system receives a visual query from a client system. The visual query is responded to as follows. A set of image feature values for the visual query is generated. The set of image feature values is mapped to a plurality of textual terms, including a weight for each of the textual terms in the plurality of textual terms. The textual terms are ranked in accordance with the weights of the textual terms. Then, in accordance with the ranking the textual terms, one or more of the ranked textual terms are sent to the client system.

Claims

1. A method, for identifying textual terms in response to a visual query comprising:

on a server system having one or more processors and memory storing one or more programs for execution by the one or more processors;

receiving a visual query from a client system;

responding to the visual query by;

generating a set of image feature values for the visual query;

mapping the set of image feature values to a plurality of textual terms, including a weight for each of the textual terms in the plurality of textual terms;

ranking the textual terms in accordance with the weights of the textual terms; and

in accordance with the ranking the textual terms, sending one or more of the ranked textual terms to the client system.

2. The method of claim 1, including:

wherein the mapping utilizes a set of image relevance models, each model corresponding to a predefined textual term.

3. The method of claim 1, wherein:

the set of image feature values for the visual query comprises an image features vector of the image feature values; and

the mapping includes multiplying the image features vector by a matrix of image relevance models, each row of the matrix corresponding to a predefined textual term.

4. The method of claim 1, wherein the set of image feature values includes:

color histogram values, intensity values, and an edge statistic.

5. The method of claim 1, wherein responding to the visual query further comprises:

sending to the client system one or more images associated with the ranked textual terms.

6. The method of claim 5, wherein the one or more images associated with the ranked textual terms have image feature values similar to the image feature values identified for the visual query.

7. The method of claim 6, further comprising:

in response to user selection of a term of the ranked textual terms, performing a textual query search on the selected term.

8. The method of claim 1, further comprising:

sending to the client system an interactive results document comprising one or more visual identifiers of respective sub-portions of the visual query, and for each visual identifier in at least a subset of the visual identifiers at least one user selectable link comprising a respective textual term of the ranked textual terms.

9. A server system, comprising:

one or more processors; and

memory storing one or more programs for execution by the one or more processors;

the one or more programs comprising instructions for;

receiving a visual query from a client system;

responding to the visual query by;

generating a set of image feature values for the visual query;

mapping the set of image feature values to a plurality of textual terms, including a weight for each of the textual terms in the plurality of textual terms;

ranking the textual terms in accordance with the weights of the textual terms; and

in accordance with the ranking the textual terms, sending one or more of the ranked textual terms to the client system.

10. The server system of claim 9, wherein instructions for mapping include instructions for utilizing a set of image relevance models, each model corresponding to a predefined textual term.

11. The server system of claim 9, wherein:

the set of image feature values for the visual query comprises an image features vector of the image feature values; and

the instructions for mapping include instructions for multiplying the image features vector by a matrix of image relevance models, each row of the matrix corresponding to a predefined textual term.

12. The server system of claim 9, wherein the set of image feature values includes:

color histogram values, intensity values, and an edge statistic.

13. The server system of claim 9, wherein responding to the visual query further comprises instructions for:

sending to the client system one or more images associated with the ranked textual terms.

14. The server system of claim 13, wherein the one or more images associated with the ranked textual terms have image feature values similar to the image feature values identified for the visual query.

15. The server system of claim 9, further comprising instructions for:

sending to the client system an interactive results document comprising one or more visual identifiers of respective sub-portions of the visual query, and for each visual identifier in at least a subset of the visual identifiers at least one user selectable link comprising a respective textual term of the ranked textual terms.

16. A non-transitory computer readable storage medium storing one or more programs configured for execution by a computer, the one or more programs comprising instructions for:

receiving a visual query from a client system;

responding to the visual query by;

generating a set of image feature values for the visual query;

mapping the set of image feature values to a plurality of textual terms, including a weight for each of the textual terms in the plurality of textual terms;

ranking the textual terms in accordance with the weights of the textual terms; and

in accordance with the ranking the textual terms, sending one or more of the ranked textual terms to the client system.

17. The non-transitory computer readable storage medium of claim 16, wherein the instructions for mapping include instructions for utilizing a set of image relevance models, each model corresponding to a predefined textual term.

18. The non-transitory computer readable storage medium of claim 16, wherein:

the set of image feature values for the visual query comprises an image features vector of the image feature values; and

the instructions for mapping include instructions for multiplying the image features vector by a matrix of image relevance models, each row of the matrix corresponding to a predefined textual term.

19. The non-transitory computer readable storage medium of claim 16, wherein the set of image feature values includes:

color histogram values, intensity values, and an edge statistic.

20. The non-transitory computer readable storage medium of claim 16, wherein responding to the visual query further comprises instructions for:

sending to the client system one or more images associated with the ranked textual terms.

21. The non-transitory computer readable storage medium of claim 20, wherein the one or more images associated with the ranked textual terms have image feature values similar to the image feature values identified for the visual query.

22. The non-transitory computer readable storage medium of claim 16, further comprising instructions for:

sending to the client system an interactive results document comprising one or more visual identifiers of respective sub-portions of the visual query, and for each visual identifier in at least a subset of the visual identifiers at least one user selectable link comprising a respective textual term of the ranked textual terms.