Identifying matching canonical documents consistent with visual query structural information

• US 9,087,235 B2
• Filed: 07/29/2014
• Issued: 07/21/2015
• Est. Priority Date: 12/02/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Petrou, David, Popat, Ashok C., Casey, Matthew R.
• Primary Examiner(s): Werner, Brian P
• Application Number:
  US14/445,420
• Publication Number:
  US 20140334746A1
• Time in Patent Office: 357 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/133 Evaluation of quality of the acquired characters
• CPC Class Codes: G06V 30/262 using context analysis, e.g. lexical, syntactic or semantic context
• CPC Class Codes: G06V 30/413 Classification of content, e.g. text, photographs or tables

Abstract

A server system receives a visual query from a client system, performs optical character recognition (OCR) on the visual query to produce text recognition data representing textual characters, including a plurality of textual characters in a contiguous region of the visual query. The server system also produces structural information associated with the textual characters in the visual query. Textual characters in the plurality of textual characters are scored. The method further includes identifying, in accordance with the scoring, one or more high quality textual strings, each comprising a plurality of high quality textual characters from among the plurality of textual characters in the contiguous region of the visual query. A canonical document that includes the one or more high quality textual strings and that is consistent with the structural information is retrieved. At least a portion of the canonical document is sent to the client system.

Claims

1. A method performed by data processing apparatus, the method comprising:

receiving, from a device, an image query that includes an image;

identifying textual characters in a region of the image and structural information associated with the textual characters in the region of the image, the structural information specifying a position of at least one of the textual characters with respect to one or more reference point elements in the image of the image query;

retrieving, using one or more of the textual characters and the structural information, a canonical document that includes the one or more textual characters at a location in the canonical document that is consistent with the structural information; and

sending, to the device, at least a portion of the canonical document.

2. The method of claim 1, wherein the one or more reference point elements include at least one of a margin of a document depicted in the image, an edge of the image, or a line break.

3. The method of claim 1, wherein the structural information specifies at least one of relative position of the textual characters, relative sizes of the textual characters, an ordering of the textual characters, a count of the textual characters, or a font category of the textual characters.

4. The method of claim 1, further comprising identifying one or more high quality text strings in the region of the image, wherein:

the one or more high quality text strings include the one or more textual characters; and

retrieving the canonical document comprises retrieving the canonical document using the one or more high quality text strings.

5. The method of claim 4, wherein identifying the one or more high quality text strings comprises:

determining a text quality score for each of the textual characters;

determining a score for each of a plurality of words in the region of the image, the score for each particular word being based on the score for each textual character included in the particular word; and

classifying each word as a high quality text string or a low quality text string based on the score for the word.

6. The method of claim 5, wherein the text quality score for a particular textual character is based on a quality score for the particular text character and a quality score for at least one neighboring character that is located next to the particular text character.

7. The method of claim 5, further comprising determining that the quality score for at least one of the one or more high quality text strings is less than a predetermined value, wherein sending at least a portion of the canonical document to the device comprises sending an image version of the at least a portion of the canonical document in response to determining that the quality score for at least one of the one or more high quality text strings is less than the predetermined value.

8. The method of claim 5, further comprising determining that the quality score for at least one of the one or more high quality text strings is greater than a predetermined value, wherein sending at least a portion of the canonical document to the device comprises sending a text version of the at least a portion of the canonical document in response to determining that the quality score for at least one of the one or more high quality text strings is greater than the predetermined value.

9. A system comprising:

a data processing apparatus;

a memory storage apparatus in data communication with the data processing apparatus, the memory storage apparatus storing instructions executable by the data processing apparatus and that upon such execution cause the data processing apparatus to perform operations comprising;

receiving, from a device, an image query that includes an image;

identifying textual characters in a region of the image and structural information associated with the textual characters in the region of the image, the structural information specifying a position of at least one of the textual characters with respect to one or more reference point elements in the image of the image query;

retrieving, using one or more of the textual characters and the structural information, a canonical document that includes the one or more textual characters at a location in the canonical document that is consistent with the structural information; and

sending, to the device, at least a portion of the canonical document.

10. The system of claim 9, wherein the one or more reference point elements include at least one of a margin of a document depicted in the image, an edge of the image, or a line break.

11. The system of claim 9, wherein the structural information specifies at least one of relative position of the textual characters, relative sizes of the textual characters, an ordering of the textual characters, a count of the textual characters, or a font category of the textual characters.

12. The system of claim 9, wherein the operations further comprise identifying one or more high quality text strings in the region of the image, wherein:

the one or more high quality text strings include the one or more textual characters; and

retrieving the canonical document comprises retrieving the canonical document using the one or more high quality text strings.

13. The system of claim 12, wherein identifying the one or more high quality text strings comprises:

determining a text quality score for each of the textual characters;

determining a score for each of a plurality of words in the region of the image, the score for each particular word being based on the score for each textual character included in the particular word; and

classifying each word as a high quality text string or a low quality text string based on the score for the word.

14. The system of claim 13, wherein the text quality score for a particular textual character is based on a quality score for the particular text character and a quality score for at least one neighboring character that is located next to the particular text character.

15. The system of claim 13, wherein the operations further comprise determining that the quality score for at least one of the one or more high quality text strings is less than a predetermined value, wherein sending at least a portion of the canonical document to the device comprises sending an image version of the at least a portion of the canonical document in response to determining that the quality score for at least one of the one or more high quality text strings is less than the predetermined value.

16. The system of claim 13, wherein the operations further comprise determining that the quality score for at least one of the one or more high quality text strings is greater than a predetermined value, wherein sending at least a portion of the canonical document to the device comprises sending a text version of the at least a portion of the canonical document in response to determining that the quality score for at least one of the one or more high quality text strings is greater than the predetermined value.

17. A computer storage device encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:

receiving, from a device, an image query that includes an image;

identifying textual characters in a region of the image and structural information associated with the textual characters in the region of the image, the structural information specifying a position of at least one of the textual characters with respect to one or more reference point elements in the image of the image query;

retrieving, using one or more of the textual characters and the structural information, a canonical document that includes the one or more textual characters at a location in the canonical document that is consistent with the structural information; and

sending, to the device, at least a portion of the canonical document.

18. The computer storage device of claim 17, wherein the one or more reference point elements include at least one of a margin of a document depicted in the image, an edge of the image, or a line break.

19. The computer storage device of claim 17, wherein the structural information specifies at least one of relative position of the textual characters, relative sizes of the textual characters, an ordering of the textual characters, a count of the textual characters, or a font category of the textual characters.

20. The computer storage device of claim 17, wherein the operations further comprise identifying one or more high quality text strings in the region of the image, wherein:

the one or more high quality text strings include the one or more textual characters; and

retrieving the canonical document comprises retrieving the canonical document using the one or more high quality text strings.