Architecture for responding to visual query
First Claim
1. A computer-implemented method comprising:
- receiving a query image from an application on a user computing device;
performing a type recognition pre-processing of the query image;
providing the query image to multiple image search engines for processing in parallel based on a plurality of types recognized in the pre-processing of the query image,wherein, for each image search engine to which the query image is submitted, the image search engine is configured to identify zero or more resources that match the query image, from among a collection of resources that are associated with the image search engine, by applying at least a respective portion of the query image, based on type value of pre-processing results, to an image-based search process that is unique to the image search engine;
in response to providing the query image to the multiple image search engines for processing in parallel, obtaining a set of two or more resources that the multiple image search engines identify as matching the query image;
communicating the two or more resources embedded in an interactive results document to the application for display, the interactive results document comprising bounding boxes around each of one or more sub-portions of the query image, the sub-portions each representing an identified entity of the query image, and each of the bounding boxes being associated with one or more resources related to corresponding sub-portion;
presenting, to the application for display, one or more visual identifiers denoting each identified entity in each of the bounding boxes;
receiving an input of a selection of a particular bounding box on the interactive results document; and
displaying the one or more resources related to the corresponding sub-portion associated with the selected bounding box.
2 Assignments
0 Petitions
Accused Products
Abstract
A visual query such as a photograph, a screen shot, a scanned image, a video frame, or an image created by a content authoring application is submitted to a visual query search system. The search system processes the visual query by sending it to a plurality of parallel search systems, each implementing a distinct visual query search process. These parallel search systems may include but are not limited to optical character recognition (OCR), facial recognition, product recognition, bar code recognition, object-or-object-category recognition, named entity recognition, and color recognition. Then at least one search result is sent to the client system. In some embodiments, when the visual query is an image containing a text element and a non-text element, at least one search result includes an optical character recognition result for the text element and at least one image-match result for the non-text element.
195 Citations
28 Claims
-
1. A computer-implemented method comprising:
-
receiving a query image from an application on a user computing device; performing a type recognition pre-processing of the query image; providing the query image to multiple image search engines for processing in parallel based on a plurality of types recognized in the pre-processing of the query image, wherein, for each image search engine to which the query image is submitted, the image search engine is configured to identify zero or more resources that match the query image, from among a collection of resources that are associated with the image search engine, by applying at least a respective portion of the query image, based on type value of pre-processing results, to an image-based search process that is unique to the image search engine; in response to providing the query image to the multiple image search engines for processing in parallel, obtaining a set of two or more resources that the multiple image search engines identify as matching the query image; communicating the two or more resources embedded in an interactive results document to the application for display, the interactive results document comprising bounding boxes around each of one or more sub-portions of the query image, the sub-portions each representing an identified entity of the query image, and each of the bounding boxes being associated with one or more resources related to corresponding sub-portion; presenting, to the application for display, one or more visual identifiers denoting each identified entity in each of the bounding boxes; receiving an input of a selection of a particular bounding box on the interactive results document; and displaying the one or more resources related to the corresponding sub-portion associated with the selected bounding box. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18)
-
-
19. A system comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising; receiving a query image; performing a type recognition pre-processing of the query image; providing the query image to multiple image search engines for processing in parallel based on one or more a plurality of types recognized in the pre-processing of the query image, wherein, for each image search engine to which the query image is submitted, the image search engine is configured to identify zero or more resources that match the query image, from among a collection of resources that are associated with the image search engine, by applying at least a respective portion of the query image, based on type value of pre-processing results, to an image-based search process that is unique to the image search engine; in response to providing the query image to the multiple image search engines for processing in parallel, obtaining a set of two or more resources that the multiple image search engines identify as matching the query image; communicating the two or more resources embedded in an interactive results document to the application for display, the interactive results document comprising bounding boxes around each of one or more sub-portions of the image, the sub-portions each representing an identified entity of the image, and each of the bounding boxes being associated with one or more resources related to the corresponding sub-portion; presenting, to the application for display, one or more visual identifiers denoting each identified entity in each of the bounding boxes; receiving an input of a selection of a particular bounding box on the interactive results document; and displaying the one or more resources related to the corresponding sub-portion associate with the selected bounding box. - View Dependent Claims (20, 21, 22, 23, 24)
-
25. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:
-
receiving a query image; performing a type recognition pre-processing of the query image; providing the query image to multiple image search engines for processing in parallel based on one or more a plurality of types recognized in the pre-processing of the query image, wherein, for each image search engine to which the query image is submitted, the image search engine is configured to identify zero or more resources that match the query image, from among a collection of resources that are associated with the image search engine, by applying at least a respective portion of the query image, based on type value of pre-processing results, to an image-based search process that is unique to the image search engine; in response to providing the query image to the multiple image search engines for processing in parallel, obtaining a set of two or more resources that the multiple image search engines identify as matching the query image; communicating the two or more resources embedded in an interactive results document to the application for display, the interactive result document comprising bounding boxes around each of one or more sub-portions of the query image, the sub-portions each representing an identified entity of the query image, and each of the bounding boxes being associated with one or more resources related to the corresponding sub-portion; presenting, to the application for display, one or more visual identifiers denoting each identified entity in each of the bounding boxes; receiving an input of a selection of a particular bounding box on the interactive results document; and displaying the one or more resources related to the corresponding sub-portion associated with the selected bounding box. - View Dependent Claims (26, 27, 28)
-
Specification