Identifying Matching Canonical Documents in Response to a Visual Query

• US 20110129153A1
• Filed: 08/06/2010
• Published: 06/02/2011
• Est. Priority Date: 12/02/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Casey, Matthew R., Popat, Ashok C., Petrou, David
• Granted Patent: US 9,183,224 B2
• Time in Patent Office: Days
• US Class Current: 382/182
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06V 10/768 using context analysis, e.g. recognition aided by known co-occurring patterns
• CPC Class Codes: G06V 10/993 Evaluation of the quality of the acquired pattern
• CPC Class Codes: G06V 30/418 Document matching, e.g. of document images

Abstract

A server system receives a visual query from a client system. The visual query is an image containing text such as a picture of a document. At the receiving server or another server, optical character recognition (OCR) is performed on the visual query to produce text recognition data representing textual characters. Each character in a contiguous region of the visual query is individually scored according to its quality. The quality score of a respective character is influenced by the quality scores of neighboring or nearby characters. Using the scores, one or more high quality strings of characters are identified. Each high quality string has a plurality of high quality characters. A canonical document containing the one or more high quality textual strings is retrieved. At least a portion of the canonical document is sent to the client system.

Claims

1. A computer-implemented method of processing a visual query comprising:

on a server system having one or more processors and memory storing one or more programs for execution by the one or more processors;

receiving a visual query from a client system;

performing optical character recognition (OCR) on the visual query to produce text recognition data representing a textual characters, including a plurality of textual characters in a contiguous region of the visual query;

scoring each textual character in the plurality of textual characters;

identifying, in accordance with the scoring, one or more high quality textual strings, each comprising a plurality of high quality textual characters from among the plurality of textual characters in the contiguous region of the visual query;

identifying, in accordance with the one or more high quality textual strings, a canonical source document matching the visual query;

retrieving the canonical source document containing the one or more high quality textual strings; and

sending at least a portion of the canonical source document to the client system.

2. The method of claim 1, wherein the portion of the canonical source document is an image segment of the canonical source document.

3. The method of claim 1, wherein the portion of the canonical source document is a machine readable text segment of the canonical source document.

4. The method of claim 1, wherein the identifying high quality strings includes:

scoring a plurality of words each in accordance with the textual character scores of the textual characters comprising a respective word; and

identifying, in accordance with the word scores, one or more high quality textual strings, each comprising a plurality of high quality words.

5. The method of claim 1, wherein scoring of a respective textual character comprises scoring the textual character as either a high quality textual character or a low quality textual character.

6. The method of claim 1, wherein scoring of a respective textual character includes generating a language-conditional character probability for the respective character indicating how well the character and a set of characters that precede the character in the text segment concord with a language model.

7. The method of claim 1, wherein the scoring of a respective textual character is based on both an OCR quality score of the textual character alone and a scoring of one or more neighboring textual characters.

8. The method of claim 1, wherein the sending includes sending the original query, a canonical source document image segment, and a canonical source document machine readable text segment for simultaneous presentation.

9. The method of claim 2, wherein the image segment presented visually matches text and non-text elements of the visual query.

10. The method of claim 1, wherein the retrieving a canonical source document further includes:

calculating a quality score of a respective high quality textual string of the one or more high quality textual strings;

retrieving an image version of the canonical source document if the quality score is below a predetermined value; and

retrieving a machine readable text version of the canonical source document if the quality score is at or above a predetermined value.

11. A server system, for processing a visual query, comprising:

one or more central processing units for executing programs;

memory storing one or more programs be executed by the one or more central processing units;

the one or more programs comprising instructions for;

receiving a visual query from a client system;

performing optical character recognition (OCR) on the visual query to produce text recognition data representing a textual characters, including a plurality of textual characters in a contiguous region of the visual query;

scoring each textual character in the plurality of textual characters;

identifying, in accordance with the scoring, one or more high quality textual strings, each comprising a plurality of high quality textual characters from among the plurality of textual characters in the contiguous region of the visual query;

identifying, in accordance with the one or more high quality textual strings, a canonical source document matching the visual query;

retrieving the canonical source document containing the one or more high quality textual strings; and

sending at least a portion of the canonical source document to the client system.

12. The system of claim 11, wherein the instructions for identifying high quality strings include instructions for:

scoring a plurality of words each in accordance with the textual character scores of the textual characters comprising a respective word; and

identifying, in accordance with the word scores, one or more high quality textual strings, each comprising a plurality of high quality words.

13. The system of claim 11, wherein the instructions for scoring of a respective textual character comprises instructions for scoring the textual character as either a high quality textual character or a low quality textual character.

14. The system of claim 11, wherein the instructions for scoring of a respective textual character includes instructions for generating a language-conditional character probability for the respective character indicating how well the character and a set of characters that precede the character in the text segment concord with a language model.

15. The system of claim 11, wherein the instructions for scoring of a respective textual character is based on both an OCR quality score of the textual character alone and a scoring of one or more neighboring textual characters.

16. The system of claim 11, wherein the instructions for retrieving a canonical source document further includes instructions for:

calculating a quality score of a respective high quality textual string of the one or more high quality textual strings;

retrieving an image version of the canonical source document if the quality score is below a predetermined value; and

retrieving a machine readable text version of the canonical source document if the quality score is at or above a predetermined value.

17. A non-transitory computer readable storage medium storing one or more programs configured for execution by a computer, the one or more programs comprising instructions for:

receiving a visual query from a client system;

performing optical character recognition (OCR) on the visual query to produce text recognition data representing a textual characters, including a plurality of textual characters in a contiguous region of the visual query;

scoring each textual character in the plurality of textual characters;

identifying, in accordance with the scoring, one or more high quality textual strings, each comprising a plurality of high quality textual characters from among the plurality of textual characters in the contiguous region of the visual query;

identifying, in accordance with the one or more high quality textual strings, a canonical source document matching the visual query;

retrieving the canonical source document containing the one or more high quality textual strings; and

sending at least a portion of the canonical source document to the client system.

18. The computer readable storage medium of claim 17, wherein the instructions for identifying high quality strings include instructions for:

scoring a plurality of words each in accordance with the textual character scores of the textual characters comprising a respective word; and

identifying, in accordance with the word scores, one or more high quality textual strings, each comprising a plurality of high quality words.

19. The computer readable storage medium of claim 17, wherein the instructions for scoring of a respective textual character comprises instructions for scoring the textual character as either a high quality textual character or a low quality textual character.

20. The computer readable storage medium of claim 17, wherein the instructions for scoring of a respective textual character includes instructions for generating a language-conditional character probability for the respective character indicating how well the character and a set of characters that precede the character in the text segment concord with a language model.

21. The computer readable storage medium of claim 17, wherein the instructions for scoring of a respective textual character is based on both an OCR quality score of the textual character alone and a scoring of one or more neighboring textual characters.

22. The computer readable storage medium of claim 17, wherein the instructions for retrieving a canonical source document further includes instructions for:

calculating a quality score of a respective high quality textual string of the one or more high quality textual strings;

retrieving an image version of the canonical source document if the quality score is below a predetermined value; and

retrieving a machine readable text version of the canonical source document if the quality score is at or above a predetermined value.