SYSTEMS AND METHODS FOR MOBILE IMAGE CAPTURE AND PROCESSING

• US 20130185618A1
• Filed: 01/11/2013
• Published: 07/18/2013
• Est. Priority Date: 01/12/2012
• Status: Active Grant

• Current Assignee: Kofax Incorporated
• Original Assignee: Kofax Incorporated
• Inventors: Macciola, Anthony, Amtrup, Jan Willers, Shustorovich, Alexander, Thrasher, Christopher W.
• Granted Patent: US 9,514,357 B2
• Time in Patent Office: Days
• US Class Current: 715/201
• CPC Class Codes: G06F 17/40 Data acquisition and logging for input to computer G06F3/00
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06Q 40/08 Insurance
• CPC Class Codes: G06Q 50/18 Legal services
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 11/20 Drawing from basic elements, e.g. lines or circles
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20164 Salient point detection; Corner detection
• CPC Class Codes: G06T 2207/30004 Biomedical image processing
• CPC Class Codes: G06T 2207/30176 Document
• CPC Class Codes: G06T 3/02 Affine transformations for image registration G06T3/147; for image mosaicing G06T3/4038
• CPC Class Codes: G06T 3/10 Selection of transformation methods according to the characteristics of the input images
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/00 Image analysis
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/143 : involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06T 7/194 : involving foreground-background segmentation
• CPC Class Codes: G06V 10/247 : by affine transforms, e.g. correction due to perspective effects; Quadrilaterals, e.g. trapezoids
• CPC Class Codes: G06V 10/25 : Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 30/413 : Classification of content, e.g. text, photographs or tables
• CPC Class Codes: G06V 30/414 : Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: H04N 1/00116 : of multimedia information
• CPC Class Codes: H04N 1/00251 : with an apparatus for taking photographic images, e.g. a camera
• CPC Class Codes: H04N 1/00403 : Voice input means, e.g. voice commands
• CPC Class Codes: H04N 1/387 : Composing, repositioning or otherwise geometrically modifying originals
• CPC Class Codes: H04N 1/3872 : Repositioning or masking
• CPC Class Codes: H04N 1/3878 : Skew detection or correction
• CPC Class Codes: H04N 1/40 : Picture signal circuits H04N1/387 takes precedence
• CPC Class Codes: H04N 2201/0081 : Image reader H04N2201/0091 - H04N2201/0094 take precedence
• CPC Class Codes: H04N 2201/0096 : Portable devices

Abstract

In various embodiments, methods, systems, and computer program products for processing digital images captured by a mobile device are disclosed. Myriad features enable and/or facilitate processing of such digital images using a mobile device that would otherwise be technically impossible or impractical, and furthermore address unique challenges presented by images captured using a camera rather than a traditional flat-bed scanner, paper-feed scanner or multifunction peripheral.

Claims

1. A method, comprising:

receiving a request to capture data comprising one or more of;

a digital image, and

a digital video; and

capturing the data using a capture component of a mobile device in response to receiving the capture request;

outputting the captured data to one or more of a display of the mobile device, a processor of the mobile device, and a storage medium of the mobile device;

receiving a request to analyze the captured data;

analyzing the captured data using the mobile device in response to receiving the request to analyze; and

outputting a result of the analyzing to one or more of the display of the mobile device, the processor of the mobile device, and the storage medium of the mobile device.

2. The method as recited in claim 1, further comprising:

receiving a request to modify one or more capture data parameters; and

modifying the one or more capture data parameters in response to the request, wherein the one or more capture data parameters comprise one or more of;

a flash setting;

a capture stability assistance setting;

a capture alignment assistance setting;

a zoom level;

a capture color mode; and

a capture data destination.

3. The method as recited in claim 1, wherein the captured data corresponds to a digital representation of a document, the analyzing comprising:

comparing one or more characteristics of the digital representation of the document to one or more quality control metrics;

determining whether each characteristic is acceptable based on the comparison;

for each characteristic;

outputting an indication that the characteristic is acceptable upon determining the characteristic is acceptable, and

outputting an indication that the characteristic is not acceptable upon determining the characteristic is not acceptable; and

outputting an indication that the digital representation of the document is acceptable upon determining that each characteristic is acceptable.

4. The method as recited in claim 3, the one or more quality control metrics comprising one or more of:

a page detection metric;

an illumination metric; and

a blur metric.

5. The method as recited in claim 3, further comprising:

displaying, via a display of the mobile device, the indication that the characteristic is not acceptable;

receiving instructions to recapture data in response to the displaying;

recapturing the data in response to receiving the instructions; and

outputting the recaptured data.

6. The method as recited in claim 3, further comprising:

displaying, via a display of the mobile device, the indication that the characteristic is not acceptable;

receiving instructions to enhance the captured data in response to the displaying;

enhancing the captured data in response to receiving the instructions without recapturing the data; and

outputting the enhanced data.

7. The method as recited in claim 1, wherein the captured data corresponds to a digital representation of a document, the method further comprising:

outputting the digital representation of the document to a display of the mobile device; and

receiving user input corresponding to instructions to modify the digital representation of the document.

8. The method as recited in claim 1, wherein the captured data corresponds to a digital representation of a document, wherein the analyzing further comprises:

receiving a request to compare the digital representation of the document to one or more known document types;

estimating a resolution of the digital representation of the document in response to receiving the compare request;

correlating the estimated resolution to one or more known resolutions, each known resolution corresponding to one of the known document types; and

outputting an estimated document type based on the correlation.

9. The method as recited in claim 1, wherein the captured data corresponds to a digital representation of a document, wherein the analyzing further comprises:

processing the digital representation of the document to determine a content of the document;

determining a document type based on the content; and

determining a resolution of the digital representation of the document based on the document type.

10. The method as recited in claim 1, further comprising:

receiving metadata corresponding to the captured data; and

associating the metadata with the captured data,

wherein the metadata correspond to one or more of;

one or more alphanumeric characters,

one or more symbols,

one or more signatures; and

one or more pointers.

11. The method as recited in claim 1, the capturing comprising reading the data from a storage medium of the mobile device.

12. The method as recited in claim 1, wherein the captured data corresponds to a digital representation of a document and further comprising:

correcting distortions in one or more portions of the digital representation of the document.

13. The method as recited in claim 12, wherein correcting the one or more artifacts comprises transforming a tetragon corresponding to the digital representation of the document into a rectangle,

wherein the tetragon is characterized by a plurality of equations,

wherein each equation corresponds to a side of the tetragon, and

wherein each equation is selected from a chosen class of functions.

14. The method as recited in claim 1, wherein the captured data corresponds to a digital representation of a document and further comprising:

detecting the digital representation of the document.

15. The method as recited in claim 14, wherein detecting the digital representation of the document in the captured data comprises:

using a processor, defining a plurality of candidate edge points corresponding to the digital representation of the document;

defining four sides of a tetragon based on the plurality of candidate edge points; and

outputting the digital representation of the document and the tetragon to the display of the mobile device.

16. The method as recited in claim 15, wherein defining the plurality of candidate edge points comprises:

defining one or more large analysis windows within a digital image;

defining a plurality of small analysis windows within the digital image;

estimating one or more distributions of statistics for each large analysis window;

calculating one or more statistics for each small analysis window;

determining whether a statistically significant difference exists between one or more of the statistics calculated for one of the small analysis windows and a corresponding distribution of statistics estimated for one of the large analysis windows; and

designating a point in the small analysis window as a candidate edge point upon determining the statistically significant difference exists.

17. The method as recited in claim 1, wherein the captured data corresponds to a digital representation of a document and further comprising determining whether one or more illumination problems exist in the digital representation of the document.

18. The method as recited in claim 17, wherein determining whether the one or more illumination problems exist comprises:

dividing, using a processor, a tetragon comprising a detected digital representation of the document into a plurality of sections, each section comprising a plurality of pixels;

determining a distribution of brightness values of each section;

determining a brightness value range of each section;

determining a variability of brightness values of each section;

determining whether each section is oversaturated;

determining whether each section is undersaturated;

determining a number of oversaturated sections;

determining a number of undersaturated sections;

determining that the captured data is oversaturated upon determining that a ratio of the number of oversaturated sections to a total number of sections exceeds an oversaturation threshold;

determining that the captured data is undersaturated upon determining that a ratio of the number of undersaturated sections to the total number of sections exceeds an undersaturation threshold; and

determining that one or more illumination problems exists in the captured data upon determining that the captured data is either undersaturated or oversaturated.

19. The method as recited in claim 1, wherein the captured data corresponds to a digital representation of a document and further comprising correcting unevenness of illumination in one or more regions of the digital representation of the document.

20. The method as recited in claim 19, wherein the correcting comprises:

dividing, using a processor, a tetragon comprising the digital representation of the document into a plurality of sections, each section comprising a plurality of pixels;

determining a distribution of color channel values of each section, each distribution of color channel value corresponding to one of a plurality of color channels;

determining, for each distribution of color channel values, one or more of;

an average color of a primary background of the corresponding section; and

an average color of a secondary background of the corresponding section;

for each section, designating either the primary background color or the secondary background color as a local representation of a main background of the digital representation of the document, each local representation being characterized by either the average color of the primary background of the corresponding section or the average color of the secondary background of the corresponding section; and

applying the two-dimensional illumination model to the captured data.

21. The method as recited in claim 1, wherein the captured data corresponds to a digital representation of a document and further comprising estimating a resolution of the digital representation of the document.

22. The method as recited in claim 21, the estimating comprising:

detecting a plurality of connected components of a plurality of non-background elements in the captured data;

determining a plurality of likely characters based on the plurality of connected components;

determining one or more average character dimensions based on the plurality of likely characters; and

estimating the resolution of the captured data based on the one or more average character dimensions.

23. The method as recited in claim 1, wherein the captured data corresponds to a digital representation of a document and further comprising determining whether the captured data comprises one or more blurred regions.

24. The method as recited in claim 23, wherein determining whether the captured data comprises one or more blurred regions comprises:

dividing, using a processor, a tetragon comprising the digital representation of the document into a plurality of sections, each section comprising a plurality of pixels;

for each section;

determining whether the section contains one or more sharp pixel-to-pixel transitions in a first direction;

counting a total number of first direction sharp pixel-to-pixel transitions for the section (S_S1);

determining whether the section contains one or more blurred pixel-to-pixel transitions in the first direction;

counting a total number of first-direction blurred pixel-to-pixel transitions for the section (S_B1);

determining whether the section contains one or more sharp pixel-to-pixel transitions in a second direction;

counting a total number of second direction sharp pixel-to-pixel transitions for the section (S_S2);

determining whether the section contains one or more blurred pixel-to-pixel transitions in the second direction;

counting a total number of second-direction blurred pixel-to-pixel transitions for the section (S_B2);

determining the section is blank upon determining;

S_S1 is less than a predetermined sharp transition threshold,

S_B1 is less than a predetermined blurred transition threshold,

S_S2 is less than a predetermined sharp transition threshold, and

S_B2 is less than a predetermined blurred transition threshold; and

determining, for all non-blank sections, a first direction blur ratio r₁=S_S1/S_B1;

determining, for all non-blank sections, a second direction blur ratio r₂=S_S2/S_B2;

determining that a non-blank section is blurred in the first direction upon determining that r₁ is less than a predefined section blur ratio threshold; and

determining that a non-blank section is blurred in the second direction upon determining that r₂ is less than the predefined section blur ratio threshold; and

determining that a non-blank section is blurred upon determining one or more of;

the section is blurred in the first direction, and

the section is blurred in the section direction; and

determining a total number of blurred sections;

calculating an image blur ratio R comprising;

the total number blurred sections to a total number of sections; and

determining the captured data comprises a blurred region upon determining the image blur ratio is greater than a predetermined image blur threshold.

25. A system, comprising:

a processor configured to execute logic;

logic for receiving a request to capture data comprising one or more of;

a digital image, and

a digital video; and

logic for capturing the data using a capture component of a mobile device in response to receiving the capture request;

outputting the captured data to one or more of a display of the mobile device, a processor of the mobile device, and a storage medium of the mobile device;

logic for receiving a request to analyze the captured data;

logic for analyzing the captured data using the mobile device in response to receiving the request to analyze; and

logic for outputting a result of the analyzing to one or more of the display of the mobile device, the processor of the mobile device, and the storage medium of the mobile device.

26. A computer program product comprising a computer readable storage medium having computer readable program code stored thereon, the computer readable program code comprising:

computer readable program code configured to receive a request to capture data comprise one or more of;

a digital image, and

a digital video; and

computer readable program code configured to capture the data use a capture component of a mobile device in response to receive the capture request;

output the captured data to one or more of a display of the mobile device, a processor of the mobile device, and a storage medium of the mobile device;

computer readable program code configured to receive a request to analyze the captured data;

computer readable program code configured to analyze the captured data use the mobile device in response to receive the request to analyze; and

computer readable program code configured to output a result of the analyze to one or more of the display of the mobile device, the processor of the mobile device, and the storage medium of the mobile device.