Extracting documents from a natural scene image

• US 8,897,565 B1
• Filed: 06/29/2012
• Issued: 11/25/2014
• Est. Priority Date: 06/29/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Palm, Leon, Adam, Hartwig
• Primary Examiner(s): Desire, Gregory M
• Application Number:
  US13/538,145
• Time in Patent Office: 879 Days
• Field of Search: 382/176, 382/182, 382/283, 382/181, 382/311
• US Class Current: 382/182
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/1475 Inclination or skew detection or correction of characters or of image to be recognised

Abstract

The present technology proposes techniques for extracting forms and other types of documents from images taken with a mobile client device. By calculating and making adjustments along a document'"'"'s detected borders, an input image can be transformed such that the document within the image may be properly aligned and background clutter completely removed. The resulting text fields of the extracted document are thus upright, aligned and locatable at predictable points.

Claims

1. A computer-implemented method comprising:

receiving an input image;

automatically identifying, by one or more computing devices, a non-rectangular, quadrilateral-shaped region within the input image;

mapping, by the one or more computing devices, the non-rectangular, quadrilateral-shaped region of the image to a rectangular-shaped output image; and

providing the output image for processing by an optical character recognition engine.

2. The method of claim 1, wherein automatically identifying a non-rectangular, quadrilateral-shaped region within the input image comprises:

detecting, by the one or more computing devices, one or more edges of the non-rectangular, quadrilateral-shaped region in the input image; and

identifying, by the one or more computing devices, the non-rectangular, quadrilateral-shaped region based on the detected one or more edges.

3. The method of claim 2, wherein detecting one or more edges of the non-rectangular, quadrilateral-shaped region in the input image comprises:

detecting, by the one or more computing devices, one or more changes in brightness of one or more portions of the input image; and

determining, by the one or more computing devices, the one or more detected changes in brightness are due to a presence of one or more edges of the non-rectangular, quadrilateral-shaped region.

4. The method of claim 1, wherein automatically identifying a non-rectangular, quadrilateral-shaped region within the input image comprises:

segmenting the input image into a set of regions that correspond to a set of candidate non-rectangular, quadrilateral-shaped regions that includes the non-rectangular, quadrilateral-shaped region; and

selecting the non-rectangular, quadrilateral-shaped region from the set of candidate non-rectangular, quadrilateral-shaped regions.

5. The method of claim 4, wherein selecting the non-rectangular, quadrilateral-shaped region from the set of candidate non-rectangular, quadrilateral-shaped regions comprises:

filtering out candidate non-rectangular, quadrilateral-shaped regions from the set of candidate non-rectangular, quadrilateral-shaped regions that are larger than an area threshold or more oblong than an oblong threshold.

6. The method of claim 4, wherein selecting the non-rectangular, quadrilateral-shaped region from the set of candidate non-rectangular, quadrilateral-shaped regions comprises:

filtering out candidate non-rectangular, quadrilateral-shaped regions from the set of candidate non-rectangular, quadrilateral-shaped regions that are duplicative.

7. The method of claim 1, wherein mapping the non-rectangular, quadrilateral-shaped region of the image to a rectangular-shaped output image comprises:

identifying corners in the non-rectangular, quadrilateral-shaped region; and

mapping the identified corners to corresponding corners of a rectangular-shaped region.

8. The method of claim 1, wherein automatically identifying a non-rectangular, quadrilateral-shaped region within the input image comprises:

detecting lines in the non-rectangular, quadrilateral-shaped region;

calculating a model quadrilateral for the non-rectangular, quadrilateral-shaped region based on the lines;

determining that the model quadrilateral matches the non-rectangular, quadrilateral-shaped region by a threshold amount; and

identifying the non-rectangular, quadrilateral-shaped region based on determining that the model quadrilateral matches the non-rectangular, quadrilateral-shaped region by the threshold amount.

9. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

receiving an input image;

automatically detecting one or more edges of a non-rectangular, quadrilateral-shaped region in the input image;

identifying the non-rectangular, quadrilateral-shaped region based on the detected one or more edges;

mapping the non-rectangular, quadrilateral-shaped region of the image to a rectangular-shaped output image; and

providing the output image for processing by an optical character recognition engine.

10. The system of claim 9, wherein detecting one or more edges of the non-rectangular, quadrilateral-shaped region in the input image comprises:

detecting one or more changes in brightness of one or more portions of the input image; and

determining the one or more detected changes in brightness are due to a presence of one or more edges of the non-rectangular, quadrilateral-shaped region.

11. The system of claim 9, wherein identifying the non-rectangular, quadrilateral-shaped region based on the detected one or more edges comprises:

segmenting the input image into a set of regions that correspond to a set of candidate non-rectangular, quadrilateral-shaped regions that includes the non-rectangular, quadrilateral-shaped region; and

selecting the non-rectangular, quadrilateral-shaped region from the set of candidate non-rectangular, quadrilateral-shaped regions.

12. The system of claim 11, wherein selecting the non-rectangular, quadrilateral-shaped region from the set of candidate non-rectangular, quadrilateral-shaped regions comprises:

filtering out candidate non-rectangular, quadrilateral-shaped regions from the set of candidate non-rectangular, quadrilateral-shaped regions that are larger than an area threshold or more oblong than an oblong threshold.

13. The system of claim 11, wherein selecting the non-rectangular, quadrilateral-shaped region from the set of candidate non-rectangular, quadrilateral-shaped regions comprises:

filtering out candidate non-rectangular, quadrilateral-shaped regions from the set of candidate non-rectangular, quadrilateral-shaped regions that are duplicative.

14. The system of claim 9, wherein mapping the non-rectangular, quadrilateral-shaped region of the image to a rectangular-shaped output image comprises:

identifying corners in the non-rectangular, quadrilateral-shaped region; and

mapping the identified corners to corresponding corners of a rectangular-shaped region.

15. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

receiving an input image;

detecting one or more changes in brightness of one or more portions of the input image;

detecting one or more edges of a non-rectangular, quadrilateral-shaped region in the input image based on the detected one or more changes in brightness of one or more portions of the input image;

identifying the non-rectangular, quadrilateral-shaped region based on the detected one or more edges;

identifying a non-rectangular, quadrilateral-shaped region within the input image;

mapping the identified non-rectangular, quadrilateral-shaped region of the image to a rectangular-shaped output image; and

providing the output image for processing by an optical character recognition engine.

16. The medium of claim 15, wherein identifying the non-rectangular, quadrilateral-shaped region based on the detected one or more edges comprises:

segmenting the input image into a set of regions that correspond to a set of candidate non-rectangular, quadrilateral-shaped regions that includes the non-rectangular, quadrilateral-shaped region; and

selecting the non-rectangular, quadrilateral-shaped region from the set of candidate non-rectangular, quadrilateral-shaped regions.

17. The medium of claim 16, wherein selecting the non-rectangular, quadrilateral-shaped region from the set of candidate non-rectangular, quadrilateral-shaped regions comprises:

filtering out candidate non-rectangular, quadrilateral-shaped regions from the set of candidate non-rectangular, quadrilateral-shaped regions that are larger than an area threshold or more oblong than an oblong threshold.

18. The medium of claim 16, wherein selecting the non-rectangular, quadrilateral-shaped region from the set of candidate non-rectangular, quadrilateral-shaped regions comprises:

filtering out candidate non-rectangular, quadrilateral-shaped regions from the set of candidate quadrilateral-shaped regions that are duplicative.