Visual restrictions for image searches

• US 9,400,809 B2
• Filed: 04/30/2014
• Issued: 07/26/2016
• Est. Priority Date: 08/02/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Ioffe, Sergey
• Primary Examiner(s): Beausoliel, Jr., Robert
• Assistant Examiner(s): Khakhar, Nirav K
• Application Number:
  US14/265,546
• Publication Number:
  US 20150169646A1
• Time in Patent Office: 818 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content

Abstract

A method and apparatus are provided for performing an image search based on a search query having a portion P₁ and a portion P₂. Based on the first search query, a second search query is generated that includes a portion P₃ and the portion P₂ such that the second search query is broader in scope than the first search query, while still retaining the portion P₂ of the first query. A first image search is then performed for the first search query to obtain a first set of search results and a second image search is performed for the second search query to obtain a second set of search results. Consequently, an image from the first set of search results is selected for presentation to a user, wherein the selection is based on content of the second set of search results.

Claims

1. A computer-implemented method comprising:

receiving a first search query that includes one or more query terms that identify a viewpoint in which an object is to appear in images that are identified in response to the first search query;

in response to receiving the first search query that includes one or more query terms that identify the viewpoint in which the object is to appear in images that are identified in response to the first search query, generating a second search query that includes (i) one or more query terms that refer to a category including the object, and (ii) one or more query terms that identify the viewpoint in which the object is to appear in images that are identified in response to the first search query;

obtaining (i) a first set of one or more images that are identified as responsive to the first search query, and (ii) a second set of one or more images that are identified as responsive to the second search query; and

selecting one or more of the images of the first set based at least on an indication of visual similarity between the selected one or more images and the one or more images of the second set of one or more images.

2. The method of claim 1, wherein generating a second search query that includes (i) one or more query terms that refer to a category including the object, and (ii) one or query terms that identify the viewpoint in which the object is to appear in images that are identified in response to the first search query, comprises:

determining the category associated with the object based at least on the first search query; and

determining one or more query terms that are not included in the first search query and that refer to the category.

3. The method of claim 2, wherein determining the category associated with the object based at least on the first search query, comprises:

identifying the category associated with one or more query terms associated with the object in the first search query based at least on one or more terms in the first search query that refer to the object and that are different than the one or more query terms that identify the viewpoint in which the object is to appear in images.

4. The method of claim 1, comprising generating the indication of visual similarity based on:

determining a function based on the second set of one or more images and the first query; and

determining to generate the indication of visual similarity based on evaluating the function using the selected one or more images.

5. The method of claim 1, comprising generating the indication of visual similarity based on:

training an image classifier using the one or more images of the second set of one or more images; and

determining to generate the indication of similarity based on classifying the selected one or more images using the trained image classifier.

6. The method of claim 1, comprising generating the indication of visual similarity based on:

generating first representations of the selected one or more images;

generating a second representation based on the second set of one or more images;

identifying visual differences between the first representations and the second representation; and

determining to generate the indication of similarity based on the identified visual differences.

7. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

receiving a first search query that includes one or more query terms that identify a viewpoint in which an object is to appear in images that are identified in response to the first search query;

in response to receiving the first search query that includes one or more query terms that identify the viewpoint in which the object is to appear in images that are identified in response to the first search query, generating a second search query that includes (i) one or more query terms that refer to a category including the object, and (ii) one or more query terms that identify the viewpoint in which the object is to appear in images that are identified in response to the first search query;

obtaining (i) a first set of one or more images that are identified as responsive to the first search query, and (ii) a second set of one or more images that are identified as responsive to the second search query; and

selecting one or more of the images of the first set based at least on an indication of visual similarity between the selected one or more images and the one or more images of the second set of one or more images.

8. The system of claim 7, wherein generating a second search query that includes (i) one or more query terms that refer to a category including the object, and (ii) one or query terms that identify the viewpoint in which the object is to appear in images that are identified in response to the first search query, comprises:

determining the category associated with the object based at least on the first search query; and

determining one or more query terms that are not included in the first search query and that refer to the category.

9. The system of claim 8, wherein determining the category associated with the object based at least on the first search query, comprises:

identifying the category associated with one or more query terms associated with the object in the first search query based at least on one or more terms in the first search query that refer to the object and that are different than the one or more query terms that identify the viewpoint in which the object is to appear in images.

10. The system of claim 7, the operations comprising generating the indication of visual similarity based on:

determining a function based on the second set of one or more images and the first query; and

determining to generate the indication of visual similarity based on evaluating the function using the selected one or more images.

11. The system of claim 7, the operations comprising generating the indication of visual similarity based on:

training an image classifier using the one or more images of the second set of one or more images; and

determining to generate the indication of similarity based on classifying the selected one or more images using the trained image classifier.

12. The system of claim 7, the operations comprising generating the indication of visual similarity based on:

generating first representations of the selected one or more images;

generating a second representation based on the second set of one or more images;

identifying visual differences between the first representations and the second representation; and

determining to generate the indication of similarity based on the identified visual differences.

13. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

receiving a first search query that includes one or more query terms that identify a viewpoint in which an object is to appear in images that are identified in response to the first search query;

in response to receiving the first search query that includes one or more query terms that identify the viewpoint in which the object is to appear in images that are identified in response to the first search query, generating a second search query that includes (i) one or more query terms that refer to a category including the object, and (ii) one or more query terms that identify the viewpoint in which the object is to appear in images that are identified in response to the first search query;

obtaining (i) a first set of one or more images that are identified as responsive to the first search query, and (ii) a second set of one or more images that are identified as responsive to the second search query; and

selecting one or more of the images of the first set based at least on an indication of visual similarity between the selected one or more images and the one or more images of the second set of one or more images.

14. The medium of claim 13, wherein generating a second search query that includes (i) one or more query terms that refer to a category including the object, and (ii) one or query terms that identify the viewpoint in which the object is to appear in images that are identified in response to the first search query, comprises:

determining the category associated with the object based at least on the first search query; and

determining one or more query terms that are not included in the first search query and that refer to the category.

15. The medium of claim 14, wherein determining the category associated with the object based at least on the first search query, comprises:

identifying the category associated with one or more query terms associated with the object in the first search query based at least on one or more terms in the first search query that refer to the object and that are different than the one or more query terms that identify the viewpoint in which the object is to appear in images.

16. The medium of claim 13, the operations comprising generating the indication of visual similarity based on:

determining a function based on the second set of one or more images and the first query; and

determining to generate the indication of visual similarity based on evaluating the function using the selected one or more images.

17. The medium of claim 13, the operations comprising generating the indication of visual similarity based on:

training an image classifier using the one or more images of the second set of one or more images; and

determining to generate the indication of similarity based on classifying the selected one or more images using the trained image classifier.