SELECTIVE DISPLAY OF OCR'ED TEXT AND CORRESPONDING IMAGES FROM PUBLICATIONS ON A CLIENT DEVICE

• US 20100188419A1
• Filed: 02/05/2009
• Published: 07/29/2010
• Est. Priority Date: 01/28/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Ratnakar, Viresh, Popat, Ashok, Haugen, Frances
• Granted Patent: US 8,373,724 B2
• Time in Patent Office: Days
• US Class Current: 345/619
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 11/206 Drawing of charts or graphs
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/127 with the intervention of an operator
• CPC Class Codes: G09G 5/14 Display of multiple viewports

Abstract

Text is extracted from a source image of a publication using an Optical Character Recognition (OCR) process. A document is generated containing text segments of the extracted text. The document includes a control module that responds to user interactions with the displayed document. Responsive to a user selection of a displayed text segment, a corresponding image segment from the source image containing the text is retrieved and rendered in place of the selected text segment. The user can select again to toggle the display back to the text segment. Each text segment can be tagged with a garbage score indicating its quality. If the garbage score of a text segment exceeds a threshold value, the corresponding image segment can be automatically displayed instead.

Claims

1. A computer-implemented method for displaying a publication, comprising:

receiving a document including text segments generated responsive to an Optical Character Recognition (OCR) process performed on a source image of the publication;

displaying one or more of the text segments on a display of a client device used by a user;

responsive to a selection of a displayed text segment by the user, retrieving an image segment of the source image corresponding to the selected text segment; and

displaying the image segment in place of the selected text segment on the display of the client device.

2. The computer-implemented method of claim 1, further comprising:

responsive to a selection of the image segment by the user, displaying the text segment in place of the selected image segment on the display of the client device.

3. The computer-implemented method of claim 1, wherein the document further includes a garbage score indicating a quality of an associated text segment, and wherein displaying one or more of the text segments comprises:

comparing the garbage score of the associated text segment with a threshold value;

responsive to the garbage score being below the threshold value, displaying the associated text segment; and

responsive to the garbage score exceeding the threshold value, displaying an image segment corresponding to the associated text segment in place of the associated text segment on the display of the client device.

4. The computer-implemented method of claim 1, wherein the document includes positional information relating the included text segments to image segments of the source image, wherein retrieving the image segment further comprises:

identifying positional information in the document for the displayed text segment; and

transmitting a request for the image segment, the request including the identified positional information.

5. The computer-implemented method of claim 4, wherein the request for the image segment is transmitted to a remote server and the image segment is retrieved from the remote server.

6. The computer-implemented method of claim 4, wherein the positional information describes a region of the source image showing the text contained in the corresponding text segment.

7. The computer-implemented method of claim 1, further comprising:

responsive to the selection of the displayed text segment by the user, storing the displayed text segment in association with the document.

8. A computer-readable storage medium encoded with executable computer program code for displaying a publication, the computer program code comprising program code for:

receiving a document including text segments generated responsive to an Optical Character Recognition (OCR) process performed on a source image of the publication;

displaying one or more of the text segments on a display of a client device used by a user;

responsive to a selection of a displayed text segment by the user, retrieving an image segment of the source image corresponding to the selected text segment; and

displaying the image segment in place of the selected text segment on the display of the client device.

9. The computer-readable storage medium of claim 8, further comprising program code for:

responsive to a selection of the image segment by the user, displaying the text segment in place of the selected image segment on the display of the client device.

10. The computer-readable storage medium of claim 8, wherein the document further includes a garbage score indicating a quality of an associated text segment, and wherein the program code for displaying one or more of the text segments comprises program code for:

comparing the garbage score of the associated text segment with a threshold value;

responsive to the garbage score being below the threshold value, displaying the associated text segment; and

responsive to the garbage score exceeding the threshold value, displaying an image segment corresponding to the associated text segment in place of the associated text segment on the display of the client device.

11. The computer-readable storage medium of claim 8, wherein the document includes positional information relating the included text segments to image segments of the source image, and wherein the program code for retrieving the image segment comprises program code for:

identifying positional information in the document for the displayed text segment; and

transmitting a request for the image segment, the request including the identified positional information.

12. A computer-implemented method for providing a publication to a user, comprising:

receiving a document including segments of text generated responsive to an Optical Character Recognition (OCR) process performed on a source image of the publication, the document also including positional information specifying segments of the source image corresponding to the text segments;

providing the document to a client device; and

responsive to receiving a request from the client device including positional information for an image segment corresponding to a text segment included in the document, providing the corresponding image segment to the client device.

13. The computer-implemented method of claim 12, wherein receiving the document further comprises:

generating a garbage score indicating a quality of an associated text segment; and

generating the document to include the associated text segment and the garbage score.

14. The computer-implemented method of claim 13, wherein generating the garbage score for an associated text segment further comprises:

calculating a set of language-conditional character probabilities for each character in the associated text segment based on a language model; and

generating the garbage score for the associated text segment based on the set of language-conditional character probabilities for the characters in the text segment.

15. The computer-implemented method of claim 13, further comprising:

including a control module in the document, the control module adapted to execute on the client device to control display of the document on the client device and to generate the request for an image segment corresponding to the text segment responsive to an interaction with a user of the client device.

16. The computer-implemented method of claim 12, wherein the positional information describes a region of the source image showing the text contained in the corresponding text segment.

17. A computer-readable storage medium encoded with executable computer program code for providing a publication to a user, the computer program code comprising program code for:

receiving a document including segments of text generated responsive to an Optical Character Recognition (OCR) process performed on a source image of the publication, the document also including positional information specifying segments of the source image corresponding to the text segments;

providing the document to a client device; and

responsive to receiving a request from the client device including positional information for an image segment corresponding to a text segment included in the document, providing the corresponding image segment to the client device.

18. The computer-readable storage medium of claim 17, wherein the program code for receiving the document further comprises program code for:

generating a garbage score indicating a quality of an associated text segment; and

generating the document to include the associated text segment and the garbage score.

19. The computer-readable storage medium of claim 18, wherein the program code for generating the garbage score for an associated text segment further comprises program code for:

calculating a set of language-conditional character probabilities for each character in the associated text segment based on a language model; and

generating the garbage score for the associated text segment based on the set of language-conditional character probabilities for the characters in the text segment.

20. The computer-readable storage medium of claim 18, further comprising program code for:

including a control module in the document, the control module adapted to execute on the client device to control display of the document on the client device and to generate the request for an image segment corresponding to the text segment responsive to an interaction with a user of the client device.