Out-of bounds detection of a document in a live camera feed

• US 10,171,695 B1
• Filed: 06/14/2017
• Issued: 01/01/2019
• Est. Priority Date: 06/14/2017
• Status: Active Grant

• Current Assignee: Intuit, Inc.
• Original Assignee: Intuit, Inc.
• Inventors: Yellapragada, Vijay, Chiang, Peijun, Lee, Daniel, Hall, Jason, Soliwal, Shailesh
• Primary Examiner(s): Reinier, Barbara D
• Application Number:
  US15/623,113
• Publication Number:
  US 20180367689A1
• Time in Patent Office: 566 Days
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/146 Aligning or centring of the image pick-up or image-field
• CPC Class Codes: G06V 30/40 Document-oriented image-based pattern recognition
• CPC Class Codes: H04N 1/00702 Position
• CPC Class Codes: H04N 1/00708 Size or dimensions
• CPC Class Codes: H04N 1/00713 Length
• CPC Class Codes: H04N 1/00718 Skew
• CPC Class Codes: H04N 1/00721 Orientation
• CPC Class Codes: H04N 1/00726 Other properties of the sheet, e.g. curvature or reflectivity
• CPC Class Codes: H04N 1/00737 using the scanning elements as detectors
• CPC Class Codes: H04N 1/00748 Detecting edges, e.g. of a stationary sheet
• CPC Class Codes: H04N 1/00771 Indicating or reporting, e.g. issuing an alarm
• CPC Class Codes: H04N 1/19594 using a television camera or a still video camera
• CPC Class Codes: H04N 1/387 Composing, repositioning or otherwise geometrically modifying originals
• CPC Class Codes: H04N 1/3873 defined only by a limited number of coordinate points or parameters, e.g. corners, centre; for trimming
• CPC Class Codes: H04N 2101/00 Still video cameras
• CPC Class Codes: H04N 2201/0081 Image reader H04N2201/0091 - H04N2201/0094 take precedence

Abstract

Aspects of the present disclosure provide methods and apparatuses for processing a digital image of a document, for example, to determine whether the document is a long document. An exemplary method generally includes obtaining a plurality of digital images of the document, segmenting at least a first digital image of the plurality of images into pixels associated with a foreground of the first digital image and pixels associated with a background of the first digital image, detecting a plurality of contours in the segmented first digital image, deciding, for each detected contour of the plurality of contours, whether that contour is an open contour or a closed contour, and determining that one or more sides of the document is out-of-bounds based, at least in part, on the decisions.

Claims

1. A computer-implemented method for processing digital images of a document, comprising:

segmenting a first digital image of a document into pixels associated with a foreground of the first digital image and pixels associated with a background of the first digital image to produce a segmented first digital image;

detecting a plurality of contours in the segmented first digital image;

deciding, for each respective contour of the plurality of contours, whether the respective contour is an open contour or a closed contour;

creating a bounding rectangle for each respective contour of the plurality of contours;

computing an area of the bounding rectangle for each respective contour of the plurality of contours to identity a largest bounding rectangle;

determining that a side of a plurality of sides of the document is out-of-bounds based on whether the largest bounding rectangle bounds a respective contour of the plurality of contours that is an open contour;

informing a user that the document is out-of-bounds;

informing the user of which particular side of the plurality of sides of the document is out-of-bounds; and

directing the user to capture at least one additional image of the document.

2. The method of claim 1, wherein determining that the side of the plurality of sides of the document is out-of-bounds comprises determining whether a side of the largest bounding rectangle touches one or more edges of the segmented first digital image.

3. The method of claim 2, wherein determining that the side of the plurality of sides of the document is out-of-bounds further comprises determining one or more corners of the largest bounding rectangle that are out-of-bounds.

4. The method of claim 1, wherein directing the user to capture the at least one additional image of the document comprises alerting the user to capture a single image that fully encompasses the document.

5. The method of claim 1, wherein directing the user to capture the at least one additional image of the document comprises:

when a font size on the document is greater than or equal to an upper font size threshold, directing the user to capture a single image of the document at a further distance than the first digital image.

6. The method of claim 1, wherein directing the user to capture the at least one additional image of the document comprises:

when a font size of text on the document is less than or equal to a lower font size threshold, directing the user to capture multiple images of the document, each image of the multiple images focusing on a different portion of the document, wherein a combination of the multiple images of the document entirely encompasses the document.

7. The method of claim 6, further comprising stitching together the multiple images of the document.

8. The method of claim 1, further comprising stopping processing of the first digital image based on the determining that the side of the plurality of sides of the document is out-of-bounds.

9. An apparatus for processing digital images of a document, comprising:

a processor; and

a memory having instructions which, when executed by the processor, performs an operation for processing a digital image, the operation comprising;

segmenting a first digital image of a document into pixels associated with a foreground of the first digital image and pixels associated with a background of the first digital image to produce a segmented first digital image;

detecting a plurality of contours in the segmented first digital image;

deciding, for each respective contour of the plurality of contours, whether the respective contour is an open contour or a closed contour;

creating a bounding rectangle for each respective contour of the plurality of contours;

computing an area of the bounding rectangle for each respective contour of the plurality of contours to identify a largest bounding rectangle;

determining that a side of a plurality of sides of the document is out-of-bounds based on whether the largest bounding rectangle bounds a contour of the plurality of contours that is an open contour;

informing a user that the document is out-of-bounds;

informing the user of which particular side of the plurality of sides of the document is out-of-bounds; and

directing the user to capture at least one additional image of the document.

10. The apparatus of claim 9, wherein determining that the side of the plurality of sides of the document is out-of-bounds comprises determining whether a side of the largest bounding rectangle touches one or more edges of the segmented first digital image.

11. The apparatus of claim 10 wherein determining that the side of the plurality of sides of the document is out-of-bounds further comprises determining one or more corners of the largest bounding rectangle that are out-of-bounds.

12. The apparatus of claim 9, wherein directing the user to capture the at least one additional image of the document comprises alerting the user to capture a single image that fully encompasses the document.

13. The apparatus of claim 9, wherein directing the user to capture the at least one additional image of the document comprises:

when a font size on the document is greater than or equal to an upper font size threshold, directing the user to capture a single image of the document at a further distance than the first digital image.

14. The apparatus of claim 9, wherein directing the user to capture the at least one additional image of the document comprises:

when a font size of text on the document is less than or equal to a lower font size threshold, directing the user to capture multiple images of the document, each image of the multiple images focusing on a different portion of the document, wherein a combination of the multiple images of the document entirely encompasses the document.

15. The apparatus of claim 14, wherein the operation further comprises stitching together the multiple images of the document.

16. The apparatus of claim 9, further comprising stopping processing of the first digital image based on the determining that the side of the plurality of sides of the document is out-of-bounds.

17. A non-transitory computer-readable medium comprising instructions which, when executed on one or more processors, performs an operation for processing a digital image of a document, comprising:

segmenting a first digital image of a document into pixels associated with a foreground of the first digital image and pixels associated with a background of the first digital image to produce a segmented first digital image;

detecting a plurality of contours in the segmented first digital image;

deciding, for each respective contour of the plurality of contours, whether that the respective contour is an open contour or a closed contour;

creating a bounding rectangle for each respective contour of the plurality of contours;

computing an area of the bounding rectangle for each respective contour of the plurality of contours to identify a largest bounding rectangle;

determining that a side of a plurality of sides of the document is out-of-bounds based on whether the largest bounding rectangle bounds a contour of the plurality of contours that is an open contour;

informing a user that the document is out-of-bounds;

informing the user of which particular side of the plurality of sides of the document is out-of-bounds; and

directing the user to capture at least one additional image of the document.

18. The non-transitory computer-readable medium of claim 17, wherein determining that the side of the plurality of sides of the document is out-of-bounds comprises determining whether a side of the largest bounding rectangle touches one or more edges of the segmented first digital image.

19. The non-transitory computer-readable medium of claim 18 wherein determining that the side of the plurality of sides of the document is out-of-bounds further comprises determining one or more corners of the largest bounding rectangle that are out-of-bounds.

20. The non-transitory computer-readable medium of claim 17, wherein directing the user to capture the at least one additional image of the document comprises alerting the user to capture a single image that fully encompasses the document.