User Interface for Presenting Search Results for Multiple Regions of a Visual Query

• US 20110035406A1
• Filed: 08/04/2010
• Published: 02/10/2011
• Est. Priority Date: 08/07/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Petrou, David, Power, Theodore
• Granted Patent: US 9,087,059 B2
• Time in Patent Office: Days
• US Class Current: 707/769
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user
• CPC Class Codes: G06F 16/438 Presentation of query results

Abstract

A visual query such as a photograph, screen shot, scanned image, or video frame is submitted to a visual query search system from a client system. The search system processes the visual query by sending it to a plurality of parallel search systems, each implementing a distinct visual query search process. A plurality of results is received from the parallel search systems. Utilizing the search results, an interactive results document is created and sent to the client system. The interactive results document has at least one visual identifier for a sub-portion of the visual query with a selectable link to at least one search result for that sub-portion. The visual identifier may be a bounding box around the respective sub-portion, or a semi-transparent label over the respective sub-portion. Optionally, the bounding box or label is color coded by type of result.

Claims

1. A computer-implemented method of processing a visual query comprising:

at a server system having one or more processors and memory storing one or more programs for execution by the one or more processors;

receiving a visual query from a client system;

processing the visual query by sending the visual query to a plurality of parallel search systems for simultaneous processing, wherein ones of the plurality of search systems implement a respective visual query search process of a plurality of visual query search processes;

receiving a plurality of search results from one or more of the plurality of parallel search systems;

creating an interactive results document comprising one or more visual identifiers of respective sub-portions of the visual query, and for each visual identifier at least one user selectable link to at least one of the search results; and

sending the interactive results document to the client system.

2. The computer-implemented method of claim 1, wherein at least one search result comprises data related to the corresponding sub-portion of the visual query.

3. The computer-implemented method of claim 1, wherein the one or more visual identifiers comprise one or more bounding boxes around respective sub-portions of the visual query.

4. The computer-implemented method of claim 3, wherein each of the bounding boxes outlines the respective sub-portion of the visual query.

5. The computer-implemented method of claim 3, wherein at least one bounding box is includes one or more smaller bounding boxes.

6. The computer-implemented method of claim 3, wherein each of the bounding boxes includes a user selectable link to one or more search results, where the user selectable link has an activation region corresponding to the sub-portion of the visual query surrounded by the bounding box.

7. The computer-implemented method of claim 1, wherein a respective user selectable link to one or more of the search results has an activation region, the activation region corresponding to the sub-portion of the visual query that is associated with a corresponding visual identifier.

8. The computer-implemented method of claim 1, further comprising:

sending text of a respective sub-portion to a text based query processing system.

9. The computer-implemented method of claim 1, wherein, when the sub-portion of the visual query corresponding to a respective visual identifier contains text comprising one or more terms, search results corresponding to the respective visual identifier include results from a term query search on at least one of the terms in the text.

10. The computer-implemented method of claim 1, wherein, when the sub-portion of the visual query corresponding to a respective visual identifier contains a person'"'"'s face, search results corresponding to the respective visual identifier include one or more of:

name, handle, contact information, account information, address information, current location of a related mobile device associated with the person who'"'"'s face is contained in the selectable sub-portion other images of the person who'"'"'s face is contained in the selectable sub-portion, and potential image matches for the person'"'"'s face.

11. The computer-implemented method of claim 1, wherein, when the sub-portion of the visual query corresponding to a respective visual identifier contains a product, search results corresponding to the respective visual identifier include one or more of:

product information, a product review, an option to initiate purchase of the product, an option to initiate a bid on the product, a list of similar products, and a list of related products.

12. The computer-implemented method of claim 1, wherein a respective visual identifier of the one or more visual identifiers is formatted for presentation in a visually distinctive manner in accordance with a type of recognized entity in the respective sub-portion of the visual query.

13. The computer-implemented method of claim 12, wherein the respective visual identifier is formatted for presentation in a visually distinctive manner selected from the group consisting of overlay color, overlay pattern, label background color, label background pattern, label font color, and border color.

14. The computer-implemented method of claim 1, wherein a respective visual identifier of the one or more visual identifiers comprises a label consisting of at least one term associated with the respective sub-portion of the visual query, wherein the label is formatted for presentation in the interactive results document on or near the respective sub-portion.

15. The computer-implemented method of claim 1, wherein the sending further comprises sending a subset of the plurality of search results in a search results list for presentation with the interactive results document.

16. The computer-implemented method of claim 15, further comprising:

receiving a user selection of the at least one user selectable link; and

identifying a search result in the search results list corresponding to the selected link.

17. A server system, for processing a visual query, comprising:

one or more central processing units for executing programs;

memory storing one or more programs be executed by the one or more central processing units;

the one or more programs comprising instructions for;

receiving a visual query from a client system;

processing the visual query by sending the visual query to a plurality of parallel search systems for simultaneous processing, wherein ones of the plurality of search systems implement a respective visual query search process of a plurality of visual query search processes;

receiving a plurality of search results from the one or more of the plurality of parallel search systems;

creating an interactive results document comprising one or more visual identifiers of respective sub-portions of the visual query, and for each visual identifier at least one user selectable link to at least one of the search results; and

sending the interactive results document to the client system.

18. The system of claim 17, wherein the one or more visual identifiers comprise one or more bounding boxes around respective sub-portions of the visual query.

19. The system of claim 18, wherein each of the bounding boxes outlines the respective sub-portion of the visual query.

20. The system of claim 18, wherein each of the bounding boxes includes a user selectable link to one or more search results, where the user selectable link has an activation region corresponding to the sub-portion of the visual query surrounded by the bounding box.

21. The system of claim 17, wherein a respective visual identifier of the one or more visual identifiers comprises a label consisting of at least one term associated with the respective sub-portion of the visual query, wherein the label is formatted for presentation in the interactive results document on or near the respective sub-portion.

22. A non-transitory computer readable storage medium storing one or more programs configured for execution by a computer, the one or more programs comprising instructions for:

receiving a visual query from a client system;

processing the visual query by sending the visual query to a plurality of parallel search systems for simultaneous processing, wherein ones of the plurality of search systems implement a respective visual query search process of a plurality of visual query search processes;

receiving a plurality of search results from the one or more of the plurality of parallel search systems;

creating an interactive results document comprising one or more visual identifiers of respective sub-portions of the visual query, and for each visual identifier at least one user selectable link to at least one of the search results; and

sending the interactive results document to the client system.

23. The computer readable storage medium of claim 22, wherein the one or more visual identifiers comprise one or more bounding boxes around respective sub-portions of the visual query.

24. The computer readable storage medium of claim 23, wherein each of the bounding boxes outlines the respective sub-portion of the visual query.

25. The computer readable storage medium of claim 23, wherein each of the bounding boxes includes a user selectable link to one or more search results, where the user selectable link has an activation region corresponding to the sub-portion of the visual query surrounded by the bounding box.

26. The computer readable storage medium of claim 22, wherein a respective visual identifier of the one or more visual identifiers comprises a label consisting of at least one term associated with the image in the respective sub-portion of the visual query, wherein the label is formatted for presentation in the interactive results document on or near the respective sub-portion.