Systems and methods for classifying objects in digital images captured using mobile devices

• US 9,754,164 B2
• Filed: 08/04/2015
• Issued: 09/05/2017
• Est. Priority Date: 03/13/2013
• Status: Active Grant

• Current Assignee: Tungsten Automation Corp.
• Original Assignee: Kofax Incorporated
• Inventors: Macciola, Anthony, Amtrup, Jan W., Ma, Jiyong, Shustorovich, Alexander, Thrasher, Christopher W., Thompson, Stephen Michael
• Primary Examiner(s): Dunphy, David F
• Application Number:
  US14/818,196
• Publication Number:
  US 20150339526A1
• Time in Patent Office: 763 Days
• CPC Class Codes: G06F 2218/08 Feature extraction
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06T 2207/20104 Interactive definition of region of interest [ROI]
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 30/40 Document-oriented image-based pattern recognition
• CPC Class Codes: G06V 30/418 Document matching, e.g. of document images

Abstract

Systems and methods for mobile image data capture and processing are disclosed. The techniques encompass receipt or capture of digital image data, detecting an object such as a document depicted in a digital image corresponding to the digital image data, processing the digital image to improve image quality, classifying the object from the processed image data, and extracting useful information from the object. Processing may improve image quality by correcting artifacts such as distortion, skew, blur, shadows, etc. common to digital images captured using mobile devices. Classification is based on identifying unique features (and/or combinations thereof) within the image data and determining whether the identified features indicate the object belongs to a class of known objects having similar characteristics, or is unique to all known classes. Extraction is based in whole or in part on object classification. All operations may be performed using mobile technology exclusively.

Claims

1. A method, comprising:

receiving or capturing digital image data;

detecting an object depicted in a digital image represented by the digital image data, wherein detecting the object comprises defining a plurality of candidate edge points based on statistically significant differences between each respective candidate edge point and a background of the digital image;

processing the digital image to improve an image quality of at least the object;

determining an object classification based on the processed digital image;

building or determining an extraction model based at least in part on the object classification; and

extracting, using the extraction model, information from one or more regions of interest within the digital image;

wherein each region of interest depicts one or more object features selected from;

text characters, symbols, photographs, and image characteristics; and

wherein the information comprises one or more of;

an object feature color;

an object feature size;

an object feature dimensions;

an object feature shape;

an object feature texture;

an object feature brightness;

an object feature intensity in one or more color channels;

a position of one or more object features depicted in the digital image; and

a positional relationship between one or more object features depicted in the digital image.

2. The method as recited in claim 1, wherein the detecting is based on the one or more object features.

3. The method as recited in claim 1, further comprising associating metadata labels to one or more of the object features, wherein the metadata labels identify one or more of:

an object feature data type;

an object feature data format;

an object feature text information content; and

an object feature location information.

4. The method as recited in claim 1, wherein the processing comprises one or more of:

rectangularizing the object based at least in part on the one or more object features;

cropping the digital image based at least in part on the one or more object features; and

binarizing the digital image based at least in part on the one or more object features.

5. The method as recited in claim 1, wherein the detecting comprises a window-based analysis of a plurality of sections of the digital image.

6. The method as recited in claim 5, wherein defining each candidate edge point comprises:

defining one or more large analysis windows within the digital image;

defining a plurality of small analysis windows within the digital image;

estimating one or more distributions of statistics for each large analysis window;

calculating one or more statistics for each small analysis window;

determining whether a statistically significant difference exists between one or more of the statistics calculated for each of the small analysis windows and a corresponding distribution of statistics estimated for the one or more large analysis windows; and

designating a point in each small analysis window for which the statistically significant difference exists as a candidate edge point upon determining the statistically significant difference exists.

7. The method as recited in claim 5, wherein the window-based analysis employs a plurality of large analysis windows several pluralities of small analysis windows, each plurality of small analysis windows corresponding to one of the large analysis windows.

8. The method as recited in claim 5, wherein the window-based analysis comprises a step-over approach analyzing the digital image according to a predefined pattern.

9. The method as recited in claim 8, wherein the predefined pattern is selected from:

spiral;

serpentine;

shingled;

checkerboard; and

striped.

10. The method as recited in claim 1, wherein the statistically significant differences are evaluated according to one or more metrics selected from:

a p-value, a z-test, and a chi-squared correlation.

11. The method as recited in claim 1, wherein the processing comprises correcting one or more of distortion and skew within the digital image, and wherein the correcting comprises:

generating a rectangularized digital image from the digital image;

determining a pair of rectangle-based coordinates for each point within the rectangularized digital image; and

matching each pair of rectangle-based coordinates to an equivalent pair of tetragon-based intrinsic coordinates within the digital image.

12. The method as recited in claim 1, wherein the processing comprises:

detecting uneven illumination within the digital image, and

normalizing illumination within the digital image in response to detecting the uneven illumination.

13. The method as recited in claim 12, wherein detecting the uneven illumination comprises:

dividing the digital image into a plurality of sections;

determining whether each section is oversaturated;

determining whether each section is undersaturated;

determining a number of oversaturated sections;

determining a number of undersaturated sections;

determining that the digital image is oversaturated upon determining that a ratio of the number of oversaturated sections to a total number of sections exceeds an oversaturation threshold;

determining that the digital image is undersaturated upon determining that a ratio of the number of undersaturated sections to the total number of sections exceeds an undersaturation threshold; and

determining that an illumination problem exists in the digital image upon determining that the digital image is either undersaturated or oversaturated; and

wherein normalizing the illumination comprises;

deriving a two-dimensional illumination model from the digital image; and

applying the two-dimensional illumination model to each pixel in the digital image.

14. The method as recited in claim 1, comprising determining one or more object features of the object based on a feature vector corresponding to the object classification.

15. The method as recited in claim 1, wherein the processing comprises detecting blur within the digital image;

and wherein detecting the blur comprises;

dividing the digital image into a plurality of sections, each section comprising a plurality of pixels, and, for each section;

determining whether the section contains one or more sharp pixel-to-pixel transitions in a first direction;

counting a total number of first direction sharp pixel-to-pixel transitions for the section (S_S1);

determining whether the section contains one or more blurred pixel-to-pixel transitions in the first direction;

counting a total number of first-direction blurred pixel-to-pixel transitions for the section (S_B1);

determining whether the section contains one or more sharp pixel-to-pixel transitions in a second direction;

counting a total number of second-direction sharp pixel-to-pixel transitions for the section (S_S2);

determining whether the section contains one or more blurred pixel-to-pixel transitions in the second direction;

counting a total number of second-direction blurred pixel-to-pixel transitions for the section (S_B2);

determining the section is blank upon determining;

S_S1 is less than a predetermined sharp transition threshold,

S_B1 is less than a predetermined blurred transition threshold,

S_S2 is less than a predetermined sharp transition threshold, and

S_B2 is less than a predetermined blurred transition threshold; and

determining, for all non-blank sections, a first direction blur ratio r₁=S_S1/S_B1;

determining, for all non-blank sections, a second direction blur ratio r₂=S_S2/S_B2;

determining that a non-blank section is blurred in the first direction upon determining that r₁ is less than a predefined section blur ratio threshold; and

determining that a non-blank section is blurred in the second direction upon determining that r₂ is less than the predefined section blur ratio threshold; and

determining that a non-blank section is bluffed upon determining one or more of;

the section is blurred in the first direction, and

the section is blurred in the section direction; and

determining a total number of blurred sections;

calculating an image blur ratio R comprising the total number blurred sections to a total number of sections; and

determining the digital image is blurred upon determining the image blur ratio is greater than a predetermined image blur threshold.

16. The method as recited in claim 1, wherein the processing comprises:

estimating a resolution of the digital image; and

wherein estimating the resolution of the digital image comprises;

detecting a plurality of connected components of a plurality of non-background elements in the digital image;

determining a plurality of likely characters based on the plurality of connected components;

determining one or more average character dimensions based on the plurality of likely characters; and

estimating the resolution of the digital image based on the one or more average character dimensions.

17. The method as recited in claim 1, wherein the information further comprises either or both of:

an object feature text information; and

an object feature text format.

18. A mobile device configured to execute logic, the mobile device comprising logic configured to cause the mobile device, upon execution thereof, to:

receive or capture digital image data;

detect a document depicted in a digital image represented by the digital image data, wherein detecting the document comprises defining a plurality of candidate edge points based on statistically significant differences between each respective candidate edge point and a background of the digital image;

process the digital image to improve an image quality of at least the document;

determine a document classification based on the processed digital image;

build or determine an extraction model based at least in part on the document classification; and

extract, using the extraction model, information from one or more regions of interest within the document;

wherein each region of interest depicts one or more object features selected from;

text characters, symbols, photographs, and image characteristics; and

wherein the information comprises one or more of;

an object feature color;

an object feature size;

an object feature dimensions;

an object feature shape;

an object feature texture;

an object feature brightness;

an object feature intensity in one or more color channels;

a position of one or more object features depicted in the digital image; and

a positional relationship between one or more object features depicted in the digital image.

19. A computer program product comprising a non-transitory computer readable storage medium having embodied thereon executable instructions configured to cause a mobile device, upon execution of the instructions, to:

receive or capture digital image data;

detect a document depicted in a digital image represented by the digital image data, wherein detecting the document comprises defining a plurality of candidate edge points based on statistically significant differences between each respective candidate edge point and a background of the digital image;

process the digital image to improve an image quality of at least the document;

determine a document classification based on the processed digital image;

build or determine an extraction model based at least in part on the document classification; and

extract, using the extraction model, information from one or more regions of interest within the document;

wherein each region of interest depicts one or more object features selected from;

text characters, symbols, photographs, and image characteristics; and

wherein the information comprises one or more of;

an object feature color;

an object feature size;

an object feature dimensions;

an object feature shape;

an object feature texture;

an object feature brightness;

an object feature intensity in one or more color channels;

a position of one or more object features depicted in the digital image; and

a positional relationship between one or more object features depicted in the digital image.