Architecture for Responding to Visual Query

• US 20140164406A1
• Filed: 02/18/2014
• Published: 06/12/2014
• Est. Priority Date: 08/07/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Petrou, David
• Granted Patent: US 10,534,808 B2
• Time in Patent Office: Days
• US Class Current: 707/754
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/5838 using colour

Abstract

A visual query such as a photograph, a screen shot, a scanned image, a video frame, or an image created by a content authoring application is submitted to a visual query search system. The search system processes the visual query by sending it to a plurality of parallel search systems, each implementing a distinct visual query search process. These parallel search systems may include but are not limited to optical character recognition (OCR), facial recognition, product recognition, bar code recognition, object-or-object-category recognition, named entity recognition, and color recognition. Then at least one search result is sent to the client system. In some embodiments, when the visual query is an image containing a text element and a non-text element, at least one search result includes an optical character recognition result for the text element and at least one image-match result for the non-text element.

Claims

1-24. -24. (canceled)

25. A computer-implemented method comprising:

receiving a query image;

providing the query image to multiple image search components, wherein, for each image search component to which the query image is submitted, the image search component is configured to identify zero or more resources that match the query image, from among a collection of resources that are associated with the image search component, by applying at least a respective portion of the query image to an image-based search process that is unique to the image search component;

in response to providing the query image to the multiple image search components, obtaining a set of resources that the multiple image search components identify as matching the query image;

selecting a subset of the set of resources that the multiple image search components identify as matching the query image; and

providing, for output, data identifying one or more resources of the subset of resources that the multiple image search components identify as matching the query image.

26. The computer-implemented method of claim 25, wherein providing the query image to the multiple image search components further comprises:

identifying one or more sub-portions of the query image;

for each sub-portion of the one or more sub-portions, providing the sub-portion of the query image to a respective image search component of the multiple search components.

27. The computer-implemented method of claim 26, wherein providing the sub-portion of the query image to the respective image search component of the multiple search components further comprises providing each sub-portion of the query image to a different image search component of the multiple search components.

28. The computer-implemented method of claim 25, further comprising:

identifying one or more textual elements of the query image;

providing the one or more textual elements of the query image to a term query search component;

in response to providing the one or more textual elements of the query image to the term query search component, obtaining one or more results from the term query search component based on the one or more textual elements of the query image; and

augmenting the data identifying the one or more resources of the subset of resources that the multiple image search components identify as matching the query image with the one or more results from the term query search component.

29. The computer-implemented method of claim 25, further comprising:

identifying a probability that the query image includes a recognizable pattern;

comparing the probability that the query image includes the recognizable pattern to a threshold;

based on comparing the probability that the query image includes the recognizable pattern to the threshold, determining that the probability is greater than the threshold; and

based on determining that the probability is greater than the threshold, providing the query image to a particular image search component of the multiple image search components that is associated with the pattern.

30. The computer-implemented method of claim 29, wherein the recognizable pattern is of a particular type of pattern, and wherein the method further comprises:

providing the particular type of pattern to a user providing the query image;

receiving, from the user, an indication that the query image is associated with the particular type of pattern; and

based on receiving the indication that the query image is associated with the particular type of pattern, providing the query image to the particular image search component of the multiple image search components that is associated with the particular type.

31. The computer-implemented method of claim 25, further comprising:

identifying one or more sub-portions of the query image that each correspond to a particular type;

for each sub-portion, determining a confidence value indicating a level of confidence that the sub-portion of the query image corresponds to the particular type; and

for each sub-portion, providing the sub-portion and the corresponding confidence value to a particular image search component of the multiple image search compartments that is associated with the sub-portion.

32. The computer-implemented method of claim 31, wherein providing the sub-portion and the corresponding confidence value to the particular image search component of the multiple image search compartments that is associated with the sub-portion further comprises providing the sub-portion to the particular search component prior to providing the query image to remaining image search components of the multiple image search components.

33. The computer-implemented method of claim 32, wherein obtaining the set of resources that the multiple image search components identify as matching the query image further comprises obtaining the set of resources based on remaining sub-portions of the query image from the remaining image search components.

34. The computer-implemented method of claim 25, wherein selecting the subset of the set of resources that the multiple image search components identify as matching the query image further comprises:

receiving, from the user, an indication that the query image is associated with a particular type of image; and

filtering the subset of resources to include only resources that are associated with the particular type of image.

35. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

receiving a query image;

providing the query image to multiple image search components, wherein, for each image search component to which the query image is submitted, the image search component is configured to identify zero or more resources that match the query image, from among a collection of resources that are associated with the image search component, by applying at least a respective portion of the query image to an image-based search process that is unique to the image search component;

in response to providing the query image to the multiple image search components, obtaining a set of resources that the multiple image search components identify as matching the query image;

selecting a subset of the set of resources that the multiple image search components identify as matching the query image; and

providing, for output, data identifying one or more resources of the subset of resources that the multiple image search components identify as matching the query image.

36. The system of claim 35, wherein providing the query image to the multiple image search components further comprises:

identifying one or more sub-portions of the query image;

for each sub-portion of the one or more sub-portions, providing the sub-portion of the query image to a respective image search component of the multiple search components.

37. The system of claim 36, wherein providing the sub-portion of the query image to the respective image search component of the multiple search components further comprises providing each sub-portion of the query image to a different image search component of the multiple search components.

38. The system of claim 35, the operations further comprising:

identifying one or more textual elements of the query image;

providing the one or more textual elements of the query image to a term query search component;

in response to providing the one or more textual elements of the query image to the term query search component, obtaining one or more results from the term query search component based on the one or more textual elements of the query image; and

augmenting the data identifying the one or more resources of the subset of resources that the multiple image search components identify as matching the query image with the one or more results from the term query search component.

39. The system of claim 35, the operations further comprising:

identifying a probability that the query image includes a recognizable pattern;

comparing the probability that the query image includes the recognizable pattern to a threshold;

based on comparing the probability that the query image includes the recognizable pattern to the threshold, determining that the probability is greater than the threshold; and

based on determining that the probability is greater than the threshold, providing the query image to a particular image search component of the multiple image search components that is associated with the pattern.

40. The system of claim 39, wherein the recognizable pattern is of a particular type of pattern, and the operations further comprising:

providing the particular type of pattern to a user providing the query image;

receiving, from the user, an indication that the query image is associated with the particular type of pattern; and

based on receiving the indication that the query image is associated with the particular type of pattern, providing the query image to the particular image search component of the multiple image search components that is associated with the particular type.

41. A computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

receiving a query image;

providing the query image to multiple image search components, wherein, for each image search component to which the query image is submitted, the image search component is configured to identify zero or more resources that match the query image, from among a collection of resources that are associated with the image search component, by applying at least a respective portion of the query image to an image-based search process that is unique to the image search component;

in response to providing the query image to the multiple image search components, obtaining a set of resources that the multiple image search components identify as matching the query image;

selecting a subset of the set of resources that the multiple image search components identify as matching the query image; and

providing, for output, data identifying one or more resources of the subset of resources that the multiple image search components identify as matching the query image.

42. The computer-readable medium of claim 41, wherein providing the query image to the multiple image search components further comprises:

identifying one or more sub-portions of the query image;

for each sub-portion of the one or more sub-portions, providing the sub-portion of the query image to a respective image search component of the multiple search components.

43. The computer-readable medium of claim 42, wherein providing the sub-portion of the query image to the respective image search component of the multiple search components further comprises providing each sub-portion of the query image to a different image search component of the multiple search components.

44. The computer-readable medium of claim 41, the operations further comprising:

identifying one or more textual elements of the query image;

providing the one or more textual elements of the query image to a term query search component;

in response to providing the one or more textual elements of the query image to the term query search component, obtaining one or more results from the term query search component based on the one or more textual elements of the query image; and

augmenting the data identifying the one or more resources of the subset of resources that the multiple image search components identify as matching the query image with the one or more results from the term query search component.