Systems and methods for mobile image capture and processing

• US 9,158,967 B2
• Filed: 01/11/2013
• Issued: 10/13/2015
• Est. Priority Date: 01/12/2012
• Status: Active Grant

• Current Assignee: Kofax Incorporated
• Original Assignee: Kofax Incorporated
• Inventors: Shustorovich, Alexander, Macciola, Anthony, Thrasher, Christopher W.
• Primary Examiner(s): COUSO, JOSE L
• Application Number:
  US13/740,134
• Publication Number:
  US 20130182951A1
• Time in Patent Office: 1,005 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 17/40 Data acquisition and logging for input to computer G06F3/00
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06Q 40/08 Insurance
• CPC Class Codes: G06Q 50/18 Legal services
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 11/20 Drawing from basic elements, e.g. lines or circles
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20164 Salient point detection; Corner detection
• CPC Class Codes: G06T 2207/30004 Biomedical image processing
• CPC Class Codes: G06T 2207/30176 Document
• CPC Class Codes: G06T 3/02 Affine transformations for image registration G06T3/147; for image mosaicing G06T3/4038
• CPC Class Codes: G06T 3/10 Selection of transformation methods according to the characteristics of the input images
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/00 Image analysis
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/143 : involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06T 7/194 : involving foreground-background segmentation
• CPC Class Codes: G06V 10/247 : by affine transforms, e.g. correction due to perspective effects; Quadrilaterals, e.g. trapezoids
• CPC Class Codes: G06V 10/25 : Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 30/413 : Classification of content, e.g. text, photographs or tables
• CPC Class Codes: G06V 30/414 : Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: H04N 1/00116 : of multimedia information
• CPC Class Codes: H04N 1/00251 : with an apparatus for taking photographic images, e.g. a camera
• CPC Class Codes: H04N 1/00403 : Voice input means, e.g. voice commands
• CPC Class Codes: H04N 1/387 : Composing, repositioning or otherwise geometrically modifying originals
• CPC Class Codes: H04N 1/3872 : Repositioning or masking
• CPC Class Codes: H04N 1/3878 : Skew detection or correction
• CPC Class Codes: H04N 1/40 : Picture signal circuits H04N1/387 takes precedence
• CPC Class Codes: H04N 2201/0081 : Image reader H04N2201/0091 - H04N2201/0094 take precedence
• CPC Class Codes: H04N 2201/0096 : Portable devices

Abstract

In various embodiments, methods, systems, and computer program products for processing digital images captured by a mobile device are disclosed. Myriad features enable and/or facilitate processing of such digital images using a mobile device that would otherwise be technically impossible or impractical, and furthermore address unique challenges presented by images captured using a camera rather than a traditional flat-bed scanner, paper-feed scanner or multifunction peripheral.

Claims

1. A method for correcting unevenness of illumination in one or more regions of a digital image, the method comprising:

using a processor, deriving a two-dimensional illumination model from the digital image, the deriving comprising;

dividing, using a processor, a tetragon comprising a detected digital representation of a document within the digital image into a plurality of sections, each section comprising a plurality of pixels;

determining a distribution of color channel values of each section, each distribution of color channel value corresponding to one of a plurality of color channels;

determining, for each distribution of color channel values, one or more of;

an average color of a primary background of the corresponding section; and

an average color of a secondary background of the corresponding section; and

for each section, designating either the primary background color or the secondary background color as a local representation of a main background of the digital representation of the document, each local representation being characterized by either the average color of the primary background of the corresponding section or the average color of the secondary background of the corresponding section; and

applying the two-dimensional illumination model to each pixel in the digital image.

2. The method as recited in claim 1, wherein the two-dimensional illumination model is a second-degree polynomial characterized by an equation v=ax²+bxy+cy²+dx+ey+f,

wherein v is an average color channel value for one of the plurality of color channels,

wherein a, b, c, d, e, and f are each unknown parameters of the two-dimensional illumination model,

wherein each unknown parameter a, b, c, d, e, and f is approximated using a least-mean-squares approximation,

wherein x is a x-coordinate of a mid-point pixel in the section, and

wherein y is a y-coordinate of the mid-point pixel in the section.

3. The method as recited in claim 1, further comprising:

determining, for each section, a plurality of color clusters;

determining a plurality of numerous color clusters, each numerous color cluster corresponding to high frequency of representation in the section;

determining a total area of the section;

determining a plurality of partial section areas, each partial section area corresponding to an area represented by one of the plurality of numerous color clusters;

dividing each partial section area by the total area to obtain a cluster percentage area for each numerous color cluster; and

classifying each numerous color cluster as either a background cluster or a non-background cluster based on the cluster percentage area,

wherein the classifying identifies either;

no background in the section,

a single most numerous background in the section, or

two most numerous backgrounds in the section.

4. The method as recited in claim 3, the applying comprising:

for a plurality of background clusters;

calculating an average color channel value of each background cluster,

calculating a hue ratio of each background cluster, and

calculating a median hue ratio for the plurality of background clusters; and

comparing the hue ratio of each background cluster to the median hue ratio of the plurality of clusters;

selecting a most likely background as the local representation based on the comparison;

calculating a plurality of average main background color channel values over a plurality of local representations;

calculating a difference between one or more predicted background channel values and the plurality of average main background color channel values;

multiplying the difference by a fraction to generate a fractional difference; and

adding the fractional difference to one or more color channel values for each pixel in the digital image.

5. The method as recited in claim 4, wherein the fraction is a value in a range from 0 to 1.

6. The method as recited in claim 3, wherein the classifying comprises classifying each pixel belonging to a cluster containing a number of pixels greater than a background threshold as a background pixel.

7. The method as recited in claim 6, wherein the background threshold is in a range from 0 to 100%.

8. The method as recited in claim 1, wherein the processor is part of a mobile device, the mobile device having an integrated camera.

11. The method as recited in claim 1, wherein the digital image comprises a digital representation of a document.

12. The method as recited in claim 1, further comprising:

determining an illumination problem exists within the digital image, and, in response to determining the illumination problem exists, performing the deriving and the applying.

13. The method as recited in claim 3, wherein determining the plurality of color clusters utilizes a clustering technique selected from:

a Markov-chain Monte Carlo technique, a nearest neighbor joining technique, a distribution-based clustering technique, an expectation-maximization technique, a density-based clustering technique, a density-based spatial clustering of applications with noise (DBSCAN) technique, and an ordering points to identify clustering structure (OPTICS) technique.

14. The method as recited in claim 4, wherein the fraction is approximately ¾

.

15. The method as recited in claim 7, wherein the threshold is approximately 15%.

16. The method as recited in claim 1, wherein applying the two-dimensional illumination model normalizes illumination values of a plurality of pixels of the digital image, generating an illumination-normalized digital image.

17. The method as recited in claim 16, further comprising submitting the illumination-normalized digital image to a business workflow.

18. The method as recited in claim 17, wherein the business workflow comprises a financial transaction.

19. The method as recited in claim 17, wherein the workflow comprises a loan application.

9. A system, comprising:

a processor configured to execute logic;

logic for deriving a two-dimensional illumination model from the digital image by;

dividing, using the processor, a tetragon comprising a detected digital representation of a document within the digital image into a plurality of sections, each section comprising a plurality of pixels;

determining a distribution of color channel values of each section, each distribution of color channel value corresponding to one of a plurality of color channels;

determining, for each distribution of color channel values, one or more of;

an average color of a primary background of the corresponding section; and

an average color of a secondary background of the corresponding section; and

for each section, designating either the primary background color or the secondary background color as a local representation of a main background of the digital representation of the document, each local representation being characterized by either the average color of the primary background of the corresponding section or the average color of the secondary background of the corresponding section; and

logic for applying the two-dimensional illumination model to each pixel in the digital image.

10. A computer program product comprising a computer readable storage medium having computer readable program code stored thereon, the computer readable program code comprising:

computer readable program code configured to derive a two-dimensional illumination model from the digital image by;

dividing, using a processor, a tetragon comprising a detected digital representation of a document within the digital image into a plurality of sections, each section comprising a plurality of pixels;

determining a distribution of color channel values of each section, each distribution of color channel value corresponding to one of a plurality of color channels;

determining, for each distribution of color channel values, one or more of;

an average color of a primary background of the corresponding section; and

an average color of a secondary background of the corresponding section; and

for each section, designating either the primary background color or the secondary background color as a local representation of a main background of the digital representation of the document, each local representation being characterized by either the average color of the primary background of the corresponding section or the average color of the secondary background of the corresponding section; and

computer readable program code configured to apply the two-dimensional illumination model to each pixel in the digital image.

20. The computer program product as recited in claim 10, wherein applying the two-dimensional illumination model normalizes illumination values of a plurality of pixels of the digital image, generating an illumination-normalized digital image, and further comprising computer readable program code configured to submit the illumination-normalized digital image to a business workflow, wherein the business workflow comprises a financial transaction.