Systems and methods for mobile image capture and processing

• US 8,989,515 B2
• Filed: 01/11/2013
• Issued: 03/24/2015
• Est. Priority Date: 01/12/2012
• Status: Active Grant

• Current Assignee: Kofax Incorporated
• Original Assignee: Kofax Incorporated
• Inventors: Shustorovich, Alexander, Macciola, Anthony, Thrasher, Christopher W.
• Primary Examiner(s): COUSO, JOSE L
• Application Number:
  US13/740,131
• Publication Number:
  US 20130182970A1
• Time in Patent Office: 802 Days
• US Class Current: 382/274
• CPC Class Codes: G06F 17/40 Data acquisition and logging for input to computer G06F3/00
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06Q 40/08 Insurance
• CPC Class Codes: G06Q 50/18 Legal services
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 11/20 Drawing from basic elements, e.g. lines or circles
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20164 Salient point detection; Corner detection
• CPC Class Codes: G06T 2207/30004 Biomedical image processing
• CPC Class Codes: G06T 2207/30176 Document
• CPC Class Codes: G06T 3/02 Affine transformations for image registration G06T3/147; for image mosaicing G06T3/4038
• CPC Class Codes: G06T 3/10 Selection of transformation methods according to the characteristics of the input images
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/00 Image analysis
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/143 : involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06T 7/194 : involving foreground-background segmentation
• CPC Class Codes: G06V 10/247 : by affine transforms, e.g. correction due to perspective effects; Quadrilaterals, e.g. trapezoids
• CPC Class Codes: G06V 10/25 : Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 30/413 : Classification of content, e.g. text, photographs or tables
• CPC Class Codes: G06V 30/414 : Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: H04N 1/00116 : of multimedia information
• CPC Class Codes: H04N 1/00251 : with an apparatus for taking photographic images, e.g. a camera
• CPC Class Codes: H04N 1/00403 : Voice input means, e.g. voice commands
• CPC Class Codes: H04N 1/387 : Composing, repositioning or otherwise geometrically modifying originals
• CPC Class Codes: H04N 1/3872 : Repositioning or masking
• CPC Class Codes: H04N 1/3878 : Skew detection or correction
• CPC Class Codes: H04N 1/40 : Picture signal circuits H04N1/387 takes precedence
• CPC Class Codes: H04N 2201/0081 : Image reader H04N2201/0091 - H04N2201/0094 take precedence
• CPC Class Codes: H04N 2201/0096 : Portable devices

Abstract

In various embodiments, methods, systems, and computer program products for processing digital images captured by a mobile device are disclosed. Myriad features enable and/or facilitate processing of such digital images using a mobile device that would otherwise be technically impossible or impractical, and furthermore address unique challenges presented by images captured using a camera rather than a traditional flat-bed scanner, paper-feed scanner or multifunction peripheral.

Claims

1. A method for determining whether illumination problems exist in a digital representation of a document, the method comprising:

dividing, using a processor, a tetragon comprising a detected digital representation of a document within the digital image into a plurality of sections, each section comprising a plurality of pixels;

determining a distribution of brightness values of each section;

determining a brightness value range of each section;

determining a variability of brightness values of each section;

determining whether each section is oversaturated;

determining whether each section is undersaturated;

determining a number of oversaturated sections;

determining a number of undersaturated sections;

determining that the digital image is oversaturated upon determining that a ratio of the number of oversaturated sections to a total number of sections exceeds an oversaturation threshold;

determining that the digital image is undersaturated upon determining that a ratio of the number of undersaturated sections to the total number of sections exceeds an undersaturation threshold; and

determining that the illumination problem exists in the digital image upon determining that the digital image is either undersaturated or oversaturated.

2. The method as recited in claim 1, further comprising:

determining a section oversaturation ratio for each section, each section oversaturation ratio comprising a number of pixels exhibiting a maximum brightness value in the section divided by a total number of pixels in the section.

3. The method as recited in claim 2, further comprising:

determining, for each section, whether the oversaturation level of the section is greater than a predetermined threshold; and

characterizing the section as oversaturated upon determining that the saturation level of the section is greater than a predetermined threshold.

4. The method as recited in claim 1, further comprising:

determining a median variability of the distribution of brightness values of each section;

determining whether each median variability is greater than a predetermined variability threshold; and

determining, for each section, that the section is undersaturated upon determining that the median variability of the section is greater than the predetermined variability threshold.

5. The method as recited in claim 1, wherein the brightness value range of each section comprises a difference between:

a maximum characteristic brightness value in the distribution of brightness values of the section; and

a minimum characteristic brightness value in the distribution of brightness values of the section.

6. The method as recited in claim 1, wherein each brightness value is an integer value in a range from 0 to 255.

7. The method as recited in claim 1, wherein determining the variability comprises:

determining a brightness value of a target pixel in the plurality of pixels;

calculating a difference between the brightness value of the target pixel and a brightness value for one or more neighboring pixels, each neighboring pixel being one or more pixels away from the target pixel;

repeating the determining and the calculating for each pixel in the plurality of pixels to obtain each target pixel variability; and

generating a distribution of target pixel variability values,

wherein each target pixel variability value is an integer in a range from 0 to 255.

8. The method as recited in claim 7, wherein the one or more neighboring pixels are located within five pixels of the target pixel along a vertical direction.

9. The method as recited in claim 7, wherein the one or more neighboring pixels are located within five pixels of the target pixel along a horizontal direction.

10. The method as recited in claim 7, further comprising:

removing one or more target pixel variability values from the distribution of target pixel variability values to generate a corrected distribution; and

defining a characteristic background variability based on the corrected distribution.

11. The method as recited in claim 1, further comprising, for each section:

defining a section height by dividing the height of the document into a predefined number of horizontal sections; and

defining a section width by dividing the width of the document into a predetermined number of vertical sections.

12. The method as recited in claim 1, wherein each section is characterized by a section height and width,

wherein the digital image is characterized by an image width w and an image height h,

wherein h>

=w,

wherein the section size is characterized by a section width w_s and a section height h_s

wherein w_s =w/m,

wherein h_s =h/n,

wherein m and n are defined so that w_s is approximately equal to h_s.

14. The method as recited in claim 1, wherein the processor is part of a mobile device, the mobile device having an integrated camera.

13. A method for determining whether illumination problems exist in a digital representation of a document, comprising:

dividing, using a processor, a tetragon comprising a detected digital representation of a document within a digital image into a plurality of sections, each section comprising a plurality of pixels;

determining a distribution of brightness values of each section;

determining a brightness value range of each section;

determining a variability of brightness values of each section;

determining a section oversaturation ratio for each section, each section oversaturation ratio comprising a number of pixels exhibiting a maximum brightness value in the section divided by a total number of pixels in the section;

determining, for each section, whether the oversaturation ratio of the section is greater than a predetermined oversaturation threshold;

determining, for each section, that the section is oversaturated upon determining that the saturation level of the section is greater than a predetermined oversaturation threshold;

determining, for each section, a median variability of the distribution of brightness values of the section;

determining whether each median variability is greater than a predetermined median variability threshold; and

determining, for each section, that the section is undersaturated upon determining that the median variability of the section is greater than the predetermined median variability threshold; and

determining that the illumination problem exists in the digital image upon determining that the digital image is either undersaturated or oversaturated.

15. A system, comprising:

a processor configured to execute logic;

logic for dividing a tetragon comprising a detected digital representation of a document within the digital image into a plurality of sections, each section comprising a plurality of pixels;

logic for determining a distribution of brightness values of each section;

logic for determining a brightness value range of each section;

logic for determining a variability of brightness values of each section;

logic for determining whether each section is oversaturated;

logic for determining whether each section is undersaturated;

logic for determining a number of oversaturated sections;

logic for determining a number of undersaturated sections;

logic for determining that the digital image is oversaturated upon determining that a ratio of the number of oversaturated sections to a total number of sections exceeds an oversaturation threshold;

logic for determining that the digital image is undersaturated upon determining that a ratio of the number of undersaturated sections to the total number of sections exceeds an undersaturation threshold; and

logic for determining that the illumination problem exists in the digital image upon determining that the digital image is either undersaturated or oversaturated.

16. A computer program product comprising a computer readable storage medium having computer readable program code stored thereon, the computer readable program code comprising:

computer readable program code configured to divide a tetragon comprise a detected digital representation of a document within the digital image into a plurality of sections, each section comprise a plurality of pixels;

computer readable program code configured to determine a distribution of brightness values of each section;

computer readable program code configured to determine a brightness value range of each section;

computer readable program code configured to determine a variability of brightness values of each section;

computer readable program code configured to determine whether each section is oversaturated;

computer readable program code configured to determine whether each section is undersaturated;

computer readable program code configured to determine a number of oversaturated sections;

computer readable program code configured to determine a number of undersaturated sections;

computer readable program code configured to determine that the digital image is oversaturated upon determine that a ratio of the number of oversaturated sections to a total number of sections exceeds an oversaturation threshold;

computer readable program code configured to determine that the digital image is undersaturated upon determine that a ratio of the number of undersaturated sections to the total number of sections exceeds an undersaturation threshold; and

computer readable program code configured to determine that the illumination problem exists in the digital image upon determine that the digital image is either undersaturated or oversaturated.