Architecture for responding to visual query

• US 10,534,808 B2
• Filed: 02/18/2014
• Issued: 01/14/2020
• Est. Priority Date: 08/07/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Petrou, David
• Primary Examiner(s): Lin, Shew Fen
• Application Number:
  US14/183,001
• Publication Number:
  US 20140164406A1
• Time in Patent Office: 2,156 Days
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/5838 using colour

Abstract

A visual query such as a photograph, a screen shot, a scanned image, a video frame, or an image created by a content authoring application is submitted to a visual query search system. The search system processes the visual query by sending it to a plurality of parallel search systems, each implementing a distinct visual query search process. These parallel search systems may include but are not limited to optical character recognition (OCR), facial recognition, product recognition, bar code recognition, object-or-object-category recognition, named entity recognition, and color recognition. Then at least one search result is sent to the client system. In some embodiments, when the visual query is an image containing a text element and a non-text element, at least one search result includes an optical character recognition result for the text element and at least one image-match result for the non-text element.

Claims

1. A computer-implemented method comprising:

receiving a query image from an application on a user computing device;

performing a type recognition pre-processing of the query image;

providing the query image to multiple image search engines for processing in parallel based on a plurality of types recognized in the pre-processing of the query image,

wherein, for each image search engine to which the query image is submitted, the image search engine is configured to identify zero or more resources that match the query image, from among a collection of resources that are associated with the image search engine, by applying at least a respective portion of the query image, based on type value of pre-processing results, to an image-based search process that is unique to the image search engine;

in response to providing the query image to the multiple image search engines for processing in parallel, obtaining a set of two or more resources that the multiple image search engines identify as matching the query image;

communicating the two or more resources embedded in an interactive results document to the application for display, the interactive results document comprising bounding boxes around each of one or more sub-portions of the query image, the sub-portions each representing an identified entity of the query image, and each of the bounding boxes being associated with one or more resources related to corresponding sub-portion;

presenting, to the application for display, one or more visual identifiers denoting each identified entity in each of the bounding boxes;

receiving an input of a selection of a particular bounding box on the interactive results document; and

displaying the one or more resources related to the corresponding sub-portion associated with the selected bounding box.

2. The computer-implemented method of claim 1, wherein providing the query image to the multiple image search engines further comprises:

identifying the one or more sub-portions of the query image;

for each sub-portion of the one or more sub-portions, providing the sub-portion of the query image to a respective image search engine of the multiple image search engines.

3. The computer-implemented method of claim 2, wherein providing the sub-portion of the query image to the respective image search engine of the multiple image search engines further comprises providing each sub-portion of the query image to a different image search engine of the multiple image search engines.

4. The computer-implemented method of claim 1, comprising:

identifying one or more textual elements of the query image;

providing the one or more textual elements of the query image to a term query search engine;

in response to providing the one or more textual elements of the query image to the term query search engine, obtaining one or more results from the term query search engine based on the one or more textual elements of the query image; and

augmenting the data identifying the one or more resources of the subset of resources that the multiple image search engines identify as matching the query image with the one or more results from the term query search engine.

5. The computer-implemented method of claim 1, comprising:

identifying a likelihood that the query image includes a recognizable pattern;

comparing the likelihood that the query image includes the recognizable pattern to a threshold;

based on comparing the likelihood that the query image includes the recognizable pattern to the threshold, determining that the likelihood is greater than the threshold; and

based on determining that the likelihood is greater than the threshold, providing the query image to a particular image search engine of the multiple image search engines that is associated with the pattern.

6. The computer-implemented method of claim 5, wherein the recognizable pattern is of a particular type of pattern, and wherein the method further comprises:

providing the particular type of pattern to a user providing the query image;

receiving, from the user, an indication that the query image is associated with the particular type of pattern; and

based on receiving the indication that the query image is associated with the particular type of pattern, providing the query image to the particular image search engine of the multiple image search engines that is associated with the particular type.

7. The computer-implemented method of claim 1, comprising:

identifying the one or more sub-portions of the query image that each correspond to a particular type;

for each sub-portion, determining a confidence value indicating a level of confidence that the sub-portion of the query image corresponds to the particular type; and

for each sub-portion, providing the sub-portion and the corresponding confidence value to a particular image search engine of the multiple image search compartments that is associated with the sub-portion.

8. The computer-implemented method of claim 7, wherein providing the sub-portion and the corresponding confidence value to the particular image search engine of the multiple image search compartments that is associated with the sub-portion further comprises providing the sub-portion to the particular image search engine prior to providing the query image to remaining image search engines of the multiple image search engines.

9. The computer-implemented method of claim 8, wherein obtaining the set of two or more resources that the multiple image search engines identify as matching the query image further comprises obtaining the set of two or more resources based on remaining sub-portions of the query image from the remaining image search engines.

10. The computer-implemented method of claim 1, wherein obtaining the set of two or more resources that the multiple image search engines identify as matching the query image further comprises:

receiving, from a user, an indication that the query image is associated with a particular type of image; and

filtering the subset of resources to include only resources that are associated with the particular type of image.

11. The computer-implemented method of claim 1, wherein the plurality of comprises a barcode.

12. The computer-implemented method of claim 1, wherein the plurality of types comprises one or more colors.

13. The computer-implemented method of claim 1, wherein the plurality of types comprises text.

14. The computer-implemented method of claim 1, wherein the plurality of types comprises a face.

15. The computer-implemented method of claim 1, further comprising requesting an input from a user to verify the one or more recognized types.

16. The computer-implemented method of claim 1, wherein the one or more visual indicators is a text label.

17. The computer-implemented method of claim 1, wherein the one or more visual indicators is an overlay pattern.

18. The computer-implemented method of claim 1, wherein the one or more visual indicators is an overlay color.

19. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

receiving a query image;

performing a type recognition pre-processing of the query image;

providing the query image to multiple image search engines for processing in parallel based on one or more a plurality of types recognized in the pre-processing of the query image,

wherein, for each image search engine to which the query image is submitted, the image search engine is configured to identify zero or more resources that match the query image, from among a collection of resources that are associated with the image search engine, by applying at least a respective portion of the query image, based on type value of pre-processing results, to an image-based search process that is unique to the image search engine;

in response to providing the query image to the multiple image search engines for processing in parallel, obtaining a set of two or more resources that the multiple image search engines identify as matching the query image;

communicating the two or more resources embedded in an interactive results document to the application for display, the interactive results document comprising bounding boxes around each of one or more sub-portions of the image, the sub-portions each representing an identified entity of the image, and each of the bounding boxes being associated with one or more resources related to the corresponding sub-portion;

presenting, to the application for display, one or more visual identifiers denoting each identified entity in each of the bounding boxes;

receiving an input of a selection of a particular bounding box on the interactive results document; and

displaying the one or more resources related to the corresponding sub-portion associate with the selected bounding box.

20. The system of claim 19, wherein providing the query image to the multiple image search engines further comprises:

identifying the one or more sub-portions of the query image;

for each sub-portion of the one or more sub-portions, providing the sub-portion of the query image to a respective image search engine of the multiple image search engines.

21. The system of claim 20, wherein providing the sub-portion of the query image to the respective image search engine of the multiple image search engines further comprises providing each sub-portion of the query image to a different image search engine of the multiple image search engines.

22. The system of claim 19, the operations further comprising:

identifying one or more textual elements of the query image;

providing the one or more textual elements of the query image to a term query search engine;

in response to providing the one or more textual elements of the query image to the term query search engine, obtaining one or more results from the term query search engine based on the one or more textual elements of the query image; and

augmenting the data identifying the one or more resources of the subset of resources that the multiple image search engines identify as matching the query image with the one or more results from the term query search engine.

23. The system of claim 19, the operations further comprising:

identifying a likelihood that the query image includes a recognizable pattern;

comparing the likelihood that the query image includes the recognizable pattern to a threshold;

based on comparing the likelihood that the query image includes the recognizable pattern to the threshold, determining that the likelihood is greater than the threshold; and

based on determining that the likelihood is greater than the threshold, providing the query image to a particular image search engine of the multiple image search engines that is associated with the pattern.

24. The system of claim 23, wherein the recognizable pattern is of a particular type of pattern, and the operations further comprising:

providing the particular type of pattern to a user providing the query image;

receiving, from the user, an indication that the query image is associated with the particular type of pattern; and

based on receiving the indication that the query image is associated with the particular type of pattern, providing the query image to the particular image search engine of the multiple image search engines that is associated with the particular type.

25. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

receiving a query image;

performing a type recognition pre-processing of the query image;

providing the query image to multiple image search engines for processing in parallel based on one or more a plurality of types recognized in the pre-processing of the query image,

wherein, for each image search engine to which the query image is submitted, the image search engine is configured to identify zero or more resources that match the query image, from among a collection of resources that are associated with the image search engine, by applying at least a respective portion of the query image, based on type value of pre-processing results, to an image-based search process that is unique to the image search engine;

in response to providing the query image to the multiple image search engines for processing in parallel, obtaining a set of two or more resources that the multiple image search engines identify as matching the query image;

communicating the two or more resources embedded in an interactive results document to the application for display, the interactive result document comprising bounding boxes around each of one or more sub-portions of the query image, the sub-portions each representing an identified entity of the query image, and each of the bounding boxes being associated with one or more resources related to the corresponding sub-portion;

presenting, to the application for display, one or more visual identifiers denoting each identified entity in each of the bounding boxes;

receiving an input of a selection of a particular bounding box on the interactive results document; and

displaying the one or more resources related to the corresponding sub-portion associated with the selected bounding box.

26. The computer-readable medium of claim 25, wherein providing the query image to the multiple image search engines further comprises:

identifying the one or more sub-portions of the query image;

for each sub-portion of the one or more sub-portions, providing the sub-portion of the query image to a respective image search engine of the multiple image search engines.

27. The computer-readable medium of claim 26, wherein providing the sub-portion of the query image to the respective image search engine of the multiple image search engines further comprises providing each sub-portion of the query image to a different image search engine of the multiple image search engines.

28. The computer-readable medium of claim 25, the operations further comprising:

identifying one or more textual elements of the query image;

providing the one or more textual elements of the query image to a term query search engine;

in response to providing the one or more textual elements of the query image to the term query search engine, obtaining one or more results from the term query search engine based on the one or more textual elements of the query image; and

augmenting the data identifying the one or more resources of the subset of resources that the multiple image search engines identify as matching the query image with the one or more results from the term query search engine.