Architecture for responding to a visual query

• US 20110125735A1
• Filed: 08/04/2010
• Published: 05/26/2011
• Est. Priority Date: 08/07/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Petrou, David
• Granted Patent: US 9,135,277 B2
• Time in Patent Office: Days
• US Class Current: 707/723
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/5838 using colour

Abstract

A visual query such as a photograph, a screen shot, a scanned image, a video frame, or an image created by a content authoring application is submitted to a visual query search system. The search system processes the visual query by sending it to a plurality of parallel search systems, each implementing a distinct visual query search process. These parallel search systems may include but are not limited to optical character recognition (OCR), facial recognition, product recognition, bar code recognition, object-or-object-category recognition, named entity recognition, and color recognition. Then at least one search result is sent to the client system. In some embodiments, when the visual query is an image containing a text element and a non-text element, at least one search result includes an optical character recognition result for the text element and at least one image-match result for the non-text element.

Claims

1. A computer-implemented method of processing a visual query comprising:

at a server system having one or more processors and memory storing one or more programs for execution by the one or more processors;

receiving a visual query from a client system;

processing the visual query by sending the visual query to a plurality of parallel search systems for simultaneous processing, wherein ones of the plurality of search systems implement a respective visual query search process of a plurality of visual query search processes, wherein the plurality of visual query search processes includes at least;

optical character recognition (OCR), facial recognition, and a first query-by-image process other than OCR and facial recognition;

receiving a plurality of search results from one or more of the plurality of parallel search systems; and

sending at least one of the plurality of search results to the client system.

2. The computer-implemented method of claim 1, including:

when at least two of the received search results meet predefined criteria, ranking the received search results that meet the predefined criteria; and

sending at least one search result of the ranked search results to the client system.

3. The computer-implemented method of claim 1, wherein the first query-by-image process is selected from the set consisting of:

product recognition, bar code recognition, landmark recognition, place recognition assisted by geo-location information, similar image recognition, object-or-object-category recognition, named entity recognition, and color recognition.

4. The computer-implemented method of claim 1, wherein the visual query is selected from the group consisting of:

a photograph, a screen shot, a scanned image, a video frame, a plurality of video frames.

5. The computer-implemented method of claim 1, wherein the client system is selected from a group consisting of:

a mobile device and a desktop device.

6. The computer-implemented method of claim 1, wherein, when the visual query is an image containing a text element and a non-text element, the at least one search result includes an optical character recognition result for the text element and at least one image-match result for the non-text element.

7. The computer-implemented method of claim 1, wherein, when the visual query is an image containing a text element and a non-text element, and the at least one search result comprises an interactive results document comprising a first visual identifier for the text element with a link to a search result produced by an optical character recognition process, and a second visual identifier for the non-text element with a link to a search result produced by an image-match process.

8. The computer-implemented method of claim 1, further comprising:

combining at least two of the plurality of search results into a compound search result.

9. A search engine system, for processing a visual query, comprising:

one or more central processing units for executing programs;

memory storing one or more programs to be executed by the one or more central processing units;

the one or more programs comprising instructions for;

receiving a visual query from a client system;

processing the visual query by sending the visual query to a plurality of parallel search systems for simultaneous processing, wherein ones of the plurality of search systems implement a respective visual query search process of a plurality of visual query search processes, wherein the plurality of visual query search processes includes at least;

optical character recognition (OCR), facial recognition, and a first query-by-image process other than OCR and facial recognition;

receiving a plurality of search results from one or more of the plurality of parallel search systems; and

sending at least one of the plurality of search results to the client system.

10. The system of claim 9, including instructions for:

when at least two of the received search results meet predefined criteria, ranking the received search results that meet the predefined criteria; and

sending at least one search result of the ranked search results to the client system.

11. The system of claim 9, wherein the first query-by-image process is selected from the set consisting of:

product recognition, bar code recognition, landmark recognition, place recognition assisted by geo-location information, similar image recognition, object-or-object-category recognition, named entity recognition, and color recognition.

12. The system of claim 9, wherein the visual query is selected from the group consisting of:

a photograph, a screen shot, a scanned image, a video frame, a plurality of video frames.

13. The system of claim 9, wherein the client system is selected from a group consisting of:

a mobile device and a desktop device.

14. The system of claim 9, wherein, when the visual query is an image containing a text element and a non-text element, the at least one search result includes an optical character recognition result for the text element and at least one image-match result for the non-text element.

15. The system of claim 9, wherein, when the visual query is an image containing a text element and a non-text element, and the at least one search result comprises an interactive results document comprising a first visual identifier for the text element with a link to a search result produced by an optical character recognition process, and a second visual identifier for the non-text element with a link to a search result produced by an image-match process.

16. The system of claim 9, further comprising instructions for:

combining at least two of the plurality of search results into a compound search result.

17. A non-transitory computer readable storage medium for processing a visual query storing one or more programs configured for execution by a computer, the one or more programs comprising instructions for:

receiving a visual query from a client system;

processing the visual query by sending the visual query to a plurality of parallel search systems for simultaneous processing, wherein ones of the plurality of search systems implement a respective visual query search process of a plurality of visual query search processes, wherein the plurality of visual query search processes includes at least;

optical character recognition (OCR), facial recognition, and a first query-by-image process other than OCR and facial recognition;

receiving a plurality of search results from one or more of the plurality of parallel search systems; and

sending at least one of the plurality of search results to the client system.

18. The computer readable storage medium of claim 17, including instructions for:

when at least two of the received search results meet predefined criteria, ranking the received search results that meet the predefined criteria; and

sending at least one search result of the ranked search results to the client system.

19. The computer readable storage medium of claim 17, wherein the first query-by-image process is selected from the set consisting of:

product recognition, bar code recognition, landmark recognition, place recognition assisted by geo-location information, similar image recognition, object-or-object-category recognition, named entity recognition, and color recognition.

20. The computer readable storage medium of claim 17, wherein the visual query is selected from the group consisting of:

a photograph, a screen shot, a scanned image, a video frame, a plurality of video frames.

21. The computer readable storage medium of claim 17, wherein the client system is selected from a group consisting of:

a mobile device and a desktop device.

22. The computer readable storage medium of claim 17, wherein, when the visual query is an image containing a text element and a non-text element, the at least one search result includes an optical character recognition result for the text element and at least one image-match result for the non-text element.

23. The computer readable storage medium of claim 17, wherein, when the visual query is an image containing a text element and a non-text element, and the at least one search result comprises an interactive results document comprising a first visual identifier for the text element with a link to a search result produced by an optical character recognition process, and a second visual identifier for the non-text element with a link to a search result produced by an image-match process.

24. The computer readable storage medium of claim 17, further comprising instructions for:

combining at least two of the plurality of search results into a compound search result.