SYSTEMS AND METHODS FOR MOBILE IMAGE CAPTURE AND PROCESSING

• US 20130182128A1
• Filed: 01/11/2013
• Published: 07/18/2013
• Est. Priority Date: 01/12/2012
• Status: Active Grant

• Current Assignee: Tungsten Automation Corp.
• Original Assignee: Kofax Incorporated
• Inventors: Macciola, Anthony, Shustorovich, Alexander, Thrasher, Christopher W., Amtrup, Jan Willers
• Granted Patent: US 9,342,742 B2
• Time in Patent Office: Days
• US Class Current: 348/207.1
• CPC Class Codes: G06F 17/40 Data acquisition and logging for input to computer G06F3/00
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06Q 40/08 Insurance
• CPC Class Codes: G06Q 50/18 Legal services
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 11/20 Drawing from basic elements, e.g. lines or circles
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20164 Salient point detection; Corner detection
• CPC Class Codes: G06T 2207/30004 Biomedical image processing
• CPC Class Codes: G06T 2207/30176 Document
• CPC Class Codes: G06T 3/02 Affine transformations for image registration G06T3/147; for image mosaicing G06T3/4038
• CPC Class Codes: G06T 3/10 Selection of transformation methods according to the characteristics of the input images
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/00 Image analysis
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/143 : involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06T 7/194 : involving foreground-background segmentation
• CPC Class Codes: G06V 10/247 : by affine transforms, e.g. correction due to perspective effects; Quadrilaterals, e.g. trapezoids
• CPC Class Codes: G06V 10/25 : Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 30/413 : Classification of content, e.g. text, photographs or tables
• CPC Class Codes: G06V 30/414 : Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: H04N 1/00116 : of multimedia information
• CPC Class Codes: H04N 1/00251 : with an apparatus for taking photographic images, e.g. a camera
• CPC Class Codes: H04N 1/00403 : Voice input means, e.g. voice commands
• CPC Class Codes: H04N 1/387 : Composing, repositioning or otherwise geometrically modifying originals
• CPC Class Codes: H04N 1/3872 : Repositioning or masking
• CPC Class Codes: H04N 1/3878 : Skew detection or correction
• CPC Class Codes: H04N 1/40 : Picture signal circuits H04N1/387 takes precedence
• CPC Class Codes: H04N 2201/0081 : Image reader H04N2201/0091 - H04N2201/0094 take precedence
• CPC Class Codes: H04N 2201/0096 : Portable devices

Abstract

In various embodiments, methods, systems, and computer program products for processing digital images captured by a mobile device are disclosed. Myriad features enable and/or facilitate processing of such digital images using a mobile device that would otherwise be technically impossible or impractical, and furthermore address unique challenges presented by images captured using a camera rather than a traditional flat-bed scanner, paper-feed scanner or multifunction peripheral.

Claims

1. A method, comprising:

outputting a case management interface to a display of a mobile device;

receiving one or more instructions corresponding to one or more case actions via the displayed case management interface, each case action being associated with at least one case; and

performing one or more of the case actions in response to receiving the instructions.

2. The method as recited in claim 1, further comprising:

outputting a data capture interface to the display of the mobile device for capturing an image comprising a digital representation of a document;

receiving a request from a user to capture the image via the data capture interface;

capturing the image in response to receiving the request, the capturing being performed using a capture component of the mobile device; and

associating the captured image with one or more cases.

3. The method as recited in claim 2, further comprising:

detecting the digital representation of the document.

4. The method as recited in claim 3, wherein detecting the digital representation of the document in the image comprises:

using a processor, defining a plurality of candidate edge points corresponding to the digital representation of the document;

defining four sides of a tetragon based on the plurality of candidate edge points; and

outputting the digital representation of the document and the tetragon to the display of the mobile device.

5. The method as recited in claim 4, wherein defining the plurality of candidate edge points comprises:

defining one or more large analysis windows within a digital image;

defining a plurality of small analysis windows within the digital image;

estimating one or more distributions of statistics for each large analysis window;

calculating one or more statistics for each small analysis window;

determining whether a statistically significant difference exists between one or more of the statistics calculated for one of the small analysis windows and a corresponding distribution of statistics estimated for one of the large analysis windows; and

designating a point in the small analysis window as a candidate edge point upon determining the statistically significant difference exists.

6. The method as recited in claim 2, further comprising:

correcting one or more distortions of one or more portions of the digital representation of the document.

8. The method as recited in claim 2, further comprising determining whether one or more illumination problems exist in the digital representation of the document.

9. The method as recited in claim 8, wherein determining whether the one or more illumination problems exist comprises:

dividing, using a processor, a tetragon comprising a detected digital representation of a document within the digital image into a plurality of sections, each section comprising a plurality of pixels;

determining a distribution of brightness values of each section;

determining a brightness value range of each section;

determining a variability of brightness values of each section;

determining whether each section is oversaturated;

determining whether each section is undersaturated;

determining a number of oversaturated sections;

determining a number of undersaturated sections;

determining that the digital image is oversaturated upon determining that a ratio of the number of oversaturated sections to a total number of sections exceeds an oversaturation threshold;

determining that the digital image is undersaturated upon determining that a ratio of the number of undersaturated sections to the total number of sections exceeds an undersaturation threshold; and

determining that the illumination problem exists in the digital image upon determining that the digital image is either undersaturated or oversaturated.

10. The method as recited in claim 2, further comprising correcting unevenness of illumination in one or more regions of the digital representation of the document.

11. The method as recited in claim 10, wherein the correcting comprises:

deriving a two-dimensional illumination model from the digital image; and

applying the two-dimensional illumination model to each pixel in the digital image.

12. The method as recited in claim 2, further comprising estimating a resolution of the digital representation of the document.

13. The method as recited in claim 12, the estimating comprising:

detecting a plurality of connected components of a plurality of non-background elements in the digital image;

determining a plurality of likely characters based on the plurality of connected components;

determining one or more average character dimensions based on the plurality of likely text characters; and

estimating the resolution of the digital image based on the one or more average character dimensions.

14. The method as recited in claim 2, further comprising determining whether the image comprises one or more blurred regions.

15. The method as recited in claim 14, wherein determining whether the image comprises one or more blurred regions comprises:

dividing, using a processor, a tetragon comprising a digital representation of a document in a digital image into a plurality of sections, each section comprising a plurality of pixels;

for each section;

determining whether the section contains one or more sharp pixel-to-pixel transitions in a first direction;

counting a total number of first direction sharp pixel-to-pixel transitions for the section (S_S1);

determining whether the section contains one or more blurred pixel-to-pixel transitions in the first direction;

counting a total number of first-direction blurred pixel-to-pixel transitions for the section (S_B1);

determining whether the section contains one or more sharp pixel-to-pixel transitions in a second direction;

counting a total number of second direction sharp pixel-to-pixel transitions for the section (S_S2);

determining whether the section contains one or more blurred pixel-to-pixel transitions in the second direction;

counting a total number of second-direction blurred pixel-to-pixel transitions for the section (S_B2);

determining the section is blank upon determining;

S_S1 is less than a predetermined sharp transition threshold,

S_B1 is less than a predetermined blurred transition threshold,

S_S2 is less than a predetermined sharp transition threshold, and

S_B2 is less than a predetermined blurred transition threshold; and

determining, for all non-blank sections, a first direction blur ratio r₁=S_S1/S_B1;

determining, for all non-blank sections, a second direction blur ratio r₂=S_S2/S_B2;

determining that a non-blank section is blurred in the first direction upon determining that r₁ is less than a predefined section blur ratio threshold; and

determining that a non-blank section is blurred in the second direction upon determining that r₂ is less than the predefined section blur ratio threshold; and

determining that a non-blank section is blurred upon determining one or more of;

the section is blurred in the first direction, and

the section is blurred in the section direction; and

determining a total number of blurred sections;

calculating an image blur ratio R comprising;

the total number blurred sections to a total number of sections; and

determining the digital image is blurred upon determining the image blur ratio is greater than a predetermined image blur threshold.

16. The method as recited in claim 2, further comprising:

outputting an authentication interface to a display of the mobile device;

receiving authentication data via the authentication interface;

receiving an authentication request;

determining whether the authentication data is valid in response to receiving the authentication request;

granting access to one or more secure case actions upon determining the authentication data is valid; and

denying access to the secure case actions upon determining the authentication data is not valid.

17. The method as recited in claim 1, the case actions comprising:

creating a new case;

opening an existing case;

deleting one or more of the existing cases;

designating a case type;

capturing case information;

capturing data corresponding to a digital representation of a document the data comprising either image data or video data;

outputting the captured data to the display of the mobile device;

associating the captured data with one or more of the new case and the existing case;

dissociating the captured data from one or more of the new case and the existing case;

processing the captured data;

outputting the processed data to the display of the mobile device;

receiving user input via the display of the mobile device; and

submitting one or more of the new case and the existing case to a remote server.

18. The method as recited in claim 17, wherein the capturing comprises:

receiving user input via the display of the mobile device; and

capturing data using a capture component of the mobile device in response to the user input.

19. The method as recited in claim 17, wherein the capturing comprises:

receiving user input via the display of the mobile device; and

reading the data from a computer-readable storage medium of the mobile device in response to the user input.

20. The method as recited in claim 17, wherein the user input corresponds to metadata related to the one or more of the new case and the existing case;

and

further comprising associating the metadata with one or more of the new case and the existing case, the metadata comprising one or more of;

one or more alphanumeric characters,

one or more symbols,

one or more signatures; and

one or more pointers.

21. The method as recited in claim 17, wherein designating the case type comprises:

displaying a plurality of potential case types on the display of the mobile device;

receiving user input via the display of the mobile device, the user input indicating one of the plurality of the potential case types is a desired case type; and

designating the case type as the desired case type.

22. The method as recited in claim 21, wherein the potential case types comprise:

an insurance claim;

a loan application;

a proof of delivery;

an undefined case type;

a new account opening;

an educational program application;

a medical record;

an expense management;

an accident report; and

a user-defined case type.

23. The method as recited in claim 17, wherein the case information comprises one or more of a digital signature, an electronic signature, and a signature created by hand.

24. The method as recited in claim 17, wherein the processing comprises:

detecting the digital representation of the document in the captured data;

determining whether the digital representation of the document is characterized by a rectangular shape;

determining whether the digital representation of the document is characterized by a skew angle≠

0°

; and

transforming the digital representation of the document into a substantially rectangular digital representation of the document characterized by a skew angle=0°

upon determining the digital representation of the document is not characterized by a rectangular shape and the skew angle≠

0°

.

25. The method as recited in claim 24, the processing further comprising:

estimating a resolution of the digital representation of the document; and

matching the digital representation of the document to a known document type based on the estimated resolution.

26. The method as recited in claim 24, the processing further comprising:

determining whether the substantially rectangular, deskewed digital representation of the document is characterized by one or more problematic illumination regions;

outputting a preview interface to the display of the mobile device upon determining the substantially rectangular, deskewed digital representation of the document is characterized by one or more problematic illumination regions;

receiving user input relating to the one or more problematic illumination regions via the display of the mobile device; and

correcting one or more unevenly illuminated regions in response to the user input relating to the one or more unevenly illuminated regions.

27. The method as recited in claim 24, the processing further comprising:

determining whether the substantially rectangular, deskewed digital representation of the document is characterized by one or more blurred regions;

outputting a preview interface to the display of the mobile device upon determining the substantially rectangular, deskewed digital representation of the document is characterized by one or more blurred regions;

receiving user input relating to the one or more blurred regions via the display of the mobile device; and

accepting, rejecting, or reprocessing the substantially rectangular, deskewed digital representation characterized by one or more blurred regions in response to the user input relating to the one or more blurred regions.

28. The method as recited in claim 1, the case actions comprising printing one or more documents related to one or more cases, the printing comprising:

submitting a print request from the mobile device to a remote resource, the remote resource and the mobile device not being in communication via a local network; and

printing the one or more documents at a remote location using the remote resource in response to the print request.

29. The method as recited in claim 1, wherein performing the one or more case actions utilizes a processor of the mobile device, the mobile device having an integrated camera.

7. The method as recited in claim 7, wherein correcting the one or more distortions comprises transforming a tetragon corresponding to the digital representation of the document into a rectangle,

wherein the tetragon is characterized by a plurality of equations,

wherein each equation corresponds to a side of the tetragon, and

wherein each equation is selected from a chosen class of functions.

30. A system, comprising:

a processor configured to execute logic;

logic for outputting a case management interface to a display of a mobile device;

logic for receiving one or more instructions corresponding to one or more case actions via the displayed case management interface, each case action being associated with at least one case; and

logic for performing one or more of the case actions in response to receiving the instructions.

31. A computer program product comprising a computer readable storage medium having computer readable program code stored thereon, the computer readable program code comprising:

computer readable program code configured to output a case management interface to a display of a mobile device;

computer readable program code configured to receive one or more instructions corresponding to one or more case actions via the displayed case management interface, each case action being associated with at least one case; and

computer readable program code configured to perform one or more of the case actions in response to receiving the instructions.