User interface for presenting search results for multiple regions of a visual query

• US 9,087,059 B2
• Filed: 08/04/2010
• Issued: 07/21/2015
• Est. Priority Date: 08/07/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Petrou, David, Power, Theodore
• Primary Examiner(s): Yen, Syling
• Application Number:
  US12/850,513
• Publication Number:
  US 20110035406A1
• Time in Patent Office: 1,812 Days
• Field of Search: 707/706, 707/722, 707/736, 707/758, 707/759, 707/764, 707/765, 707/766, 707/769, 707/781, 707/999.002, 707/3, 707/5
• US Class Current: 1/1
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user
• CPC Class Codes: G06F 16/438 Presentation of query results

Abstract

A visual query such as a photograph, screen shot, scanned image, or video frame is submitted to a visual query search system from a client system. The search system processes the visual query by sending it to a plurality of parallel search systems, each implementing a distinct visual query search process. A plurality of results is received from the parallel search systems. Utilizing the search results, an interactive results document is created and sent to the client system. The interactive results document has at least one visual identifier for a sub-portion of the visual query with a selectable link to at least one search result for that sub-portion. The visual identifier may be a bounding box around the respective sub-portion, or a semi-transparent label over the respective sub-portion. Optionally, the bounding box or label is color coded by type of result.

Claims

1. A computer-implemented method comprising:

obtaining, from each of at least first and second image search components to which a query image is submitted, a set of resources that the image search component identifies as matching the query image based on applying the query image to an image-based search process that is specific to the image search component;

selecting (i) a particular resource that the first image search component identifies as matching the query image, and (ii) a particular resource that the second image search component identifies as matching the query image;

determining (i) a region of the query image that is associated with the particular resource that the first image search component identifies as matching the query image, and (ii) a region of the query image that is associated with the particular resource that the second image search component identifies as matching the query image; and

generating an image search results page that includes, (i) in a first region of the search results page that is associated with the first image search component, a first interactive-search result that includes a thumbnail image associated with the region of the query image that is determined to be associated with the particular resource that the first image search component identifies as matching the query image, wherein the first interactive-search result is linked to the particular resource that the first image search component identifies as matching the query image, (ii) in a different, second region of the search results page that is associated with the second image search component, a second interactive-search result that includes a thumbnail image associated with the region of the query image that is determined to be associated with the particular resource that the second image search component identifies as matching the query image, wherein the second interactive-search result is linked to the particular resource that the second image search component identifies as matching the query image, and (iii) a third region of the search results page that is associated with the query image.

2. The computer-implemented method of claim 1, wherein the region of the query image that is determined to be associated with the particular resource that the first image search component identifies as matching the query image includes a bounding box that outlines a boundary of an identifiable entity in a sub-portion of the query image.

3. The computer-implemented method of claim 2, wherein the bounding box is associated with an activation region of a user selectable link, the user selectable link associated with one or more search results corresponding to the sub-portion of the query image.

4. The computer-implemented method of claim 2, further comprising:

identifying a type of the identifiable entity in the sub-portion of the query image; and

formatting a visual display of the bounding box based on the identified type of the identifiable entity in the sub-portion of the query image.

5. The computer-implemented method of claim 1, wherein the region of the query image that is determined to be associated with the particular resource that the first image search component includes a textual element, the method further comprising:

obtaining, from a term query search component, an additional set of resources based on the textual element; and

augmenting the image search result page to include the additional set of resources.

6. The computer-implemented method of claim 1, wherein the region of the query image that is determined to be associated with the particular resource that the first image search component includes a facial image, the method further comprising:

identifying one or more attributes associated with a person matching the facial image; and

augmenting the image search results page to include the one or more attributes associated with the person matching the facial image.

7. The computer-implemented method of claim 1, wherein generating the image search results page further includes:

generating the image search result page to include a search results list associated with (i) the particular resource that the first image search component identifies as matching the query image and (ii) the particular resource that the second image search component identifies as matching the query image;

receiving a selection of a user selectable link associated with one of (i) the particular resource that the first image search component identifies as matching the query image and (ii) the particular resource that the second image search component identifies as matching the query image; and

identifying a search result of the search results list corresponding to the selection of the user selectable link.

8. The computer-implemented method of claim 1, further comprising:

receiving a selection of a user selectable link associated with one of (i) the particular resource that the first image search component identifies as matching the query image and (ii) the particular resource that the second image search component identifies as matching the query image; and

based on the selection of the user selectable link, providing a subset of search results that are associated with one of (i) the particular resource that the first image search component identifies as matching the query image and (ii) the particular resource that the first image search component identifies as matching the query image.

9. The computer-implemented method of claim 1, wherein the first image search component and the second image search component are substantially the same.

10. The computer-implemented method of claim 1, wherein the first image search component and the second image search component differ.

11. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

obtaining, from each of at least first and second image search components to which a query image is submitted, a set of resources that the image search component identifies as matching the query image based on applying the query image to an image-based search process that is specific to the image search component;

selecting (i) a particular resource that the first image search component identifies as matching the query image, and (ii) a particular resource that the second image search component identifies as matching the query image;

determining (i) a region of the query image that is associated with the particular resource that the first image search component identifies as matching the query image, and (ii) a region of the query image that is associated with the particular resource that the second image search component identifies as matching the query image; and

generating an image search results page that includes, (i) in a first region of the search results page that is associated with the first image search component, a first interactive-search result that includes a thumbnail image associated with the region of the query image that is determined to be associated with the particular resource that the first image search component identifies as matching the query image, wherein the first interactive-search result is linked to the particular resource that the first image search component identifies as matching the query image, (ii) in a different second region of the search results page that is associated with the second image search component, a second interactive-search result that includes a thumbnail image associated with the region of the query image that is determined to be associated with the particular resource that the second image search component identifies as matching the query image, wherein the second interactive-search result is linked to the particular resource that the second image search component identifies as matching the query image, and (iii) a third region of the search results page that is associated with the query image.

12. The system of claim 11, wherein the region of the query image that is determined to be associated with the particular resource that the first image search component identifies as matching the query image includes a bounding box that outlines a boundary of an identifiable entity in a sub-portion of the query image.

13. The system of claim 12, wherein the bounding box is associated with an activation region of a user selectable link, the user selectable link associated with one or more search results corresponding to the sub-portion of the query image.

14. The system of claim 12, the operations further comprising:

identifying a type of the identifiable entity in the sub-portion of the query image; and

formatting a visual display of the bounding box based on the identified type of the identifiable entity in the sub-portion of the query image.

15. The system of claim 11, wherein the region of the query image that is determined to be associated with the particular resource that the first image search component includes a textual element, the method further comprising:

obtaining, from a term query search component, an additional set of resources based on the textual element; and

augmenting the image search result page to include the additional set of resources.

16. The system of claim 11, wherein the region of the query image that is determined to be associated with the particular resource that the first image search component includes a facial image, the method further comprising:

identifying one or more attributes associated with a person matching the facial image; and

augmenting the image search results page to include the one or more attributes associated with the person matching the facial image.

17. The system of claim 11, wherein generating the image search results page further includes:

generating the image search result page to include a search results list associated with (i) the particular resource that the first image search component identifies as matching the query image and (ii) the particular resource that the second image search component identifies as matching the query image;

receiving a selection of a user selectable link associated with one of (i) the particular resource that the first image search component identifies as matching the query image and (ii) the particular resource that the second image search component identifies as matching the query image; and

identifying a search result of the search results list corresponding to the selection of the user selectable link.

18. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

obtaining, from each of at least first and second image search components to which a query image is submitted, a set of resources that the image search component identifies as matching the query image based on applying the query image to an image-based search process that is specific to the image search component;

selecting (i) a particular resource that the first image search component identifies as matching the query image, and (ii) a particular resource that the second image search component identifies as matching the query image;

determining (i) a region of the query image that is associated with the particular resource that the first image search component identifies as matching the query image, and (ii) a region of the query image that is associated with the particular resource that the second image search component identifies as matching the query image; and

generating an image search results page that includes, (i) in a first region of the search results page that is associated with the first image search component, a first interactive-search result that includes a thumbnail image associated with the region of the query image that is determined to be associated with the particular resource that the first image search component identifies as matching the query image, wherein the first interactive-search result is linked to the particular resource that the first image search component identifies as matching the query image, (ii) in a different, second region of the search results page that is associated with the second image search component, a second interactive-search result that includes a thumbnail image associated with the region of the query image that is determined to be associated with the particular resource that the second image search component identifies as matching the query image, wherein the second interactive-search result is linked to the particular resource that the second image search component identifies as matching the query image, and (iii) a third region of the search results page that is associated with the query image.

19. The non-transitory computer-readable medium of claim 18, wherein the region of the query image that is determined to be associated with the particular resource that the first image search component identifies as matching the query image includes a bounding box that outlines a boundary of an identifiable entity in a sub-portion of the query image.

20. The non-transitory computer-readable medium of claim 19, wherein the bounding box is associated with an activation region of a user selectable link, the user selectable link associated with one or more search results corresponding to the sub-portion of the query image.