Systems and methods for mobile image capture and processing

• US 8,855,375 B2
• Filed: 01/11/2013
• Issued: 10/07/2014
• Est. Priority Date: 01/12/2012
• Status: Active Grant

• Current Assignee: Kofax Incorporated
• Original Assignee: Kofax Incorporated
• Inventors: Macciola, Anthony, Shustorovich, Alexander, Thrasher, Christopher W.
• Primary Examiner(s): AZARIAN, SEYED H
• Application Number:
  US13/740,123
• Publication Number:
  US 20130182002A1
• Time in Patent Office: 634 Days
• Field of Search: 382/100, 382/103, 382/106, 382113-116, 382135-140, 382/155, 382/168, 382173-194, 382/199, 382209-220, 382/232, 382/254, 382/266, 382/274, 382/276, 382286-294, 382/305, 382/312, 382/321, 382/167, 382/112, 348/222.1, 358/448
• US Class Current: 382/112
• CPC Class Codes: G06F 17/40 Data acquisition and logging for input to computer G06F3/00
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06Q 40/08 Insurance
• CPC Class Codes: G06Q 50/18 Legal services
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 11/20 Drawing from basic elements, e.g. lines or circles
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20164 Salient point detection; Corner detection
• CPC Class Codes: G06T 2207/30004 Biomedical image processing
• CPC Class Codes: G06T 2207/30176 Document
• CPC Class Codes: G06T 3/02 Affine transformations for image registration G06T3/147; for image mosaicing G06T3/4038
• CPC Class Codes: G06T 3/10 Selection of transformation methods according to the characteristics of the input images
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/00 Image analysis
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/143 : involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06T 7/194 : involving foreground-background segmentation
• CPC Class Codes: G06V 10/247 : by affine transforms, e.g. correction due to perspective effects; Quadrilaterals, e.g. trapezoids
• CPC Class Codes: G06V 10/25 : Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 30/413 : Classification of content, e.g. text, photographs or tables
• CPC Class Codes: G06V 30/414 : Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: H04N 1/00116 : of multimedia information
• CPC Class Codes: H04N 1/00251 : with an apparatus for taking photographic images, e.g. a camera
• CPC Class Codes: H04N 1/00403 : Voice input means, e.g. voice commands
• CPC Class Codes: H04N 1/387 : Composing, repositioning or otherwise geometrically modifying originals
• CPC Class Codes: H04N 1/3872 : Repositioning or masking
• CPC Class Codes: H04N 1/3878 : Skew detection or correction
• CPC Class Codes: H04N 1/40 : Picture signal circuits H04N1/387 takes precedence
• CPC Class Codes: H04N 2201/0081 : Image reader H04N2201/0091 - H04N2201/0094 take precedence
• CPC Class Codes: H04N 2201/0096 : Portable devices

Abstract

In various embodiments, methods, systems, and computer program products for processing digital images captured by a mobile device are disclosed. Myriad features enable and/or facilitate processing of such digital images using a mobile device that would otherwise be technically impossible or impractical, and furthermore address unique challenges presented by images captured using a camera rather than a traditional flat-bed scanner, paper-feed scanner, or multifunction peripheral.

Claims

1. A method for processing a digital image comprising a digital representation of a document, the method comprising:

capturing one or more of image data containing the digital representation of the document and audio data relating to the digital representation of the document;

using a processor, defining a plurality of candidate edge points, wherein defining each candidate edge point comprises;

defining one or more large analysis windows within a digital image;

defining a plurality of small analysis windows within the digital image;

estimating one or more distributions of statistics for each large analysis window;

calculating one or more statistics for each small analysis window;

determining whether a statistically significant difference exists between one or more of the statistics calculated for each of the small analysis windows and a corresponding distribution of statistics estimated for the one or more large analysis windows;

designating a point in each small analysis window for which the statistically significant difference exists as a candidate edge point upon determining the statistically significant difference exists; and

defining four sides of a tetragon based on the plurality of candidate edge points; and

outputting the digital representation of the document and the tetragon to a display of a mobile device.

2. The method as recited in claim 1, wherein capturing the one or more of the image data containing the digital representation of the document and the audio data relating to the digital representation of the document is performed using one or more capture components coupled to the mobile device.

3. The method as recited in claim 1, wherein one or more of the image data and the audio data is captured from a computer-readable storage medium coupled to the mobile device.

4. The method as recited in claim 1, further comprising:

defining a new large analysis window upon determining the statistically significant difference exists for one of the small analysis windows;

reestimating the distributions of statistics for the new large analysis window; and

determining whether the statistically significant difference exists between the one or more statistics calculated for the small analysis window and the reestimated distributions of statistics.

5. The method as recited in claim 1, wherein the statistics calculated for each small analysis windows comprise one or more of:

a minimum brightness value;

a maximum brightness value; and

a brightness value range; and

wherein the distributions of statistics estimated for each large analysis window comprise one or more of;

a distribution of color channel value minima;

a distribution of color channel value maxima; and

a distribution of color channel value ranges.

6. The method as recited in claim 1, wherein the defining the small analysis windows comprises defining each small analysis window along a path, and wherein the path proceeds from an outer region of the digital image toward a center of the digital image.

7. The method as recited in claim 6, wherein the path proceeds along one or more of rows and columns of the digital image.

8. The method as recited in claim 1, wherein at least one large analysis window comprises a corner region of the image.

9. The method as recited in claim 1, further comprising bypassing one or more variations in a texture of a background of the digital image.

10. The method as recited in claim 1, wherein each large analysis window is characterized by a substantially rectangular shape, and

wherein each small analysis window is characterized by a substantially rectangular shape.

11. The method as recited in claim 1, wherein each small analysis window is characterized by a single center pixel, and

wherein the center pixel is designated as a candidate edge point upon determining the statistically significant difference exists.

12. The method as recited in claim 1, wherein defining each of the tetragon sides comprises performing at least one least-mean-squares (LMS) approximation.

13. The method as recited in claim 12, further comprising:

removing one or more outlier candidate edge points from the plurality of candidate edge points.

14. The method as recited in claim 13, further comprising performing at least one additional LMS approximation excluding the one or more outlier candidate edge points.

15. The method as recited in claim 13,

wherein each side of the tetragon is characterized by an equation chosen from a class of functions, and

wherein performing the at least one LMS approximation comprises determining one or more coefficients for each equation.

16. The method as recited in claim 1, wherein defining the sides of the tetragon comprises determining whether each side of the digital representation of the document falls within a class of functions.

17. The method as recited in claim 1, further comprising defining one or more corners of the tetragon, the defining comprising:

calculating one or more intersections between two adjacent sides; and

designating an appropriate intersection from the one or more calculated intersections.

18. The method as recited in claim 1, wherein a corner of the tetragon comprises an intersection of two curved adjacent sides of the tetragon.

19. The method as recited in claim 1, wherein a corner of the tetragon comprises an intersection of two substantially straight lines.

20. The method as recited in claim 1, wherein a corner of the tetragon comprises an intersection of one substantially straight line and one substantially curved line.

21. The method as recited in claim 1, wherein a corner of the tetragon comprises an intersection of one substantially straight line and one substantially parabolic curve.

22. The method as recited in claim 1, wherein a corner of the tetragon comprises an intersection of two substantially parabolic curves.

23. The method as recited in claim 1, further comprising defining one or more tetragon corners, the defining comprising solving one or more of:

a first degree polynomial equation;

a second degree polynomial equation;

a third degree polynomial equation; and

a fourth degree polynomial equation.

24. The method as recited in claim 1, wherein an area of the tetragon comprises at least a threshold percentage of a total area of the digital image.

25. The method as recited in claim 1, wherein a first line connects a calculated top left corner of the tetragon to a calculated bottom right corner of the tetragon,

wherein a second line connects a calculated top right corner of the tetragon and a calculated bottom left corner of the tetragon, and

wherein the first line and the second line intersect inside the tetragon.

26. The method as recited in claim 1, further comprising:

determining whether the tetragon satisfies one or more quality control metrics; and

rejecting the tetragon upon determining the tetragon does not satisfy one or more of the quality control metrics,

wherein the quality control metrics comprise one or more of;

a LMS support metric,

a minimum tetragon area metric,

a tetragon corner location metric; and

a tetragon diagonal intersection location metric.

27. The method as recited in claim 1, further comprising:

receiving an indication that the defining the four sides of the tetragon based on the plurality of candidate edge points failed to define a valid tetragon; and

redefining the plurality of candidate edge points,

wherein the redefining comprises sampling a greater number of points within the digital image than a number of points sampled in the defining.

28. The method as recited in claim 27, further comprising:

designating an entirety of the digital image as the digital representation of the document.

29. The method as recited in claim 1, wherein the processor is part of a mobile device, the mobile device having an integrated camera.

30. A system, comprising:

a processor configured to execute logic;

logic for capturing one or more of image data containing a digital representation of a document and audio data relating to the digital representation of the document;

logic for defining a plurality of candidate edge points, wherein logic for defining each candidate edge point comprises;

logic for defining one or more large analysis windows within a digital image;

logic for defining a plurality of small analysis windows within the digital image;

logic for estimating one or more distributions of statistics for each large analysis window;

logic for calculating one or more statistics for each small analysis window;

logic for determining whether a statistically significant difference exists between one or more of the statistics calculated for one of the small analysis windows and a corresponding distribution of statistics estimated for one of the large analysis windows;

logic for designating a pixel in the small analysis window as a candidate edge point upon determining the statistically significant difference exists; and

logic for defining four sides of a tetragon based on the plurality of candidate edge points; and

logic for outputting the digital representation of the document and the tetragon to a display of a mobile device.

31. A computer program product comprising a computer readable storage medium having computer readable program code stored thereon, the computer readable program code comprising:

computer readable program code configured to capture one or more of image data containing a digital representation of a document and audio data relating to the digital representation of the document;

computer readable program code configured to define a plurality of candidate edge points, wherein computer readable program code configured to define each candidate edge point comprises;

computer readable program code configured to define one or more large analysis windows within a digital image;

computer readable program code configured to define a plurality of small analysis windows within the digital image;

computer readable program code configured to estimate one or more distributions of statistics for each large analysis window;

computer readable program code configured to calculate one or more statistics for each small analysis window;

computer readable program code configured to determine whether a statistically significant difference exists between one or more of the statistics calculated for one of the small analysis windows and a corresponding distribution of statistics estimated for one of the large analysis windows;

computer readable program code configured to designate a pixel in the small analysis window as a candidate edge point upon determine the statistically significant difference exists; and

computer readable program code configured to define four sides of a tetragon based on the plurality of candidate edge points; and

computer readable program code configured to output the digital representation of the document and the tetragon to a display of a mobile device.