Systems and methods for classifying objects in digital images captured using mobile devices

• US 9,996,741 B2
• Filed: 05/17/2016
• Issued: 06/12/2018
• Est. Priority Date: 03/13/2013
• Status: Active Grant

• Current Assignee: Tungsten Automation Corp.
• Original Assignee: Kofax Incorporated
• Inventors: Amtrup, Jan W., Macciola, Anthony, Thompson, Steve, Ma, Jiyong, Shustorovich, Alexander, Thrasher, Christopher W.
• Primary Examiner(s): SHAH, UTPAL D
• Application Number:
  US15/157,325
• Publication Number:
  US 20160259973A1
• Time in Patent Office: 756 Days
• Field of Search: None
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06V 10/247 by affine transforms, e.g. correction due to perspective effects; Quadrilaterals, e.g. trapezoids
• CPC Class Codes: G06V 20/62 Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/224 of printed characters having additional code marks or containing code marks
• CPC Class Codes: G06V 30/40 Document-oriented image-based pattern recognition
• CPC Class Codes: G06V 30/412 Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
• CPC Class Codes: G06V 30/413 Classification of content, e.g. text, photographs or tables
• CPC Class Codes: G06V 30/416 Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors

Abstract

In one embodiment, a method includes receiving a digital image captured by a mobile device; and using a processor of the mobile device: generating a first representation of the digital image, the first representation being characterized by a reduced resolution; generating a first feature vector based on the first representation; comparing the first feature vector to a plurality of reference feature matrices; classifying an object depicted in the digital image as a member of a particular object class based at least in part on the comparing; and determining one or more object features of the object based at least in part on the particular object class. Corresponding systems and computer program products are also disclosed.

Claims

1. A computer-implemented method, comprising:

receiving a digital image captured by a mobile device; and

using a processor of the mobile device;

generating a first representation of the digital image, the first representation being characterized by a reduced resolution;

generating a first feature vector based on the first representation;

comparing the first feature vector to a plurality of reference feature matrices;

classifying an object depicted in the digital image as a member of a particular object class based at least in part on the comparing;

determining one or more object features of the object based at least in part on the particular object class; and

detecting the object within the digital image based on the one or more object features, wherein the detecting comprises directly detecting an object color profile within the digital image rather than detecting a transition from a first color profile to a second color profile.

2. The method as recited in claim 1, comprising:

receiving an object class identification code; and

retrieving the one or more object features based on the object class identification code.

3. The method as recited in claim 1, wherein the one or more object features comprise a location of a subregion of the digital image, the subregion depicting one or more reference objects.

4. The method as recited in claim 3, wherein the location corresponds to a particular object orientation state.

5. The method as recited in claim 1, wherein the one or more object features are selected from a group consisting of:

an object color profile; and

an object subregion color profile.

6. The method as recited in claim 1, wherein the one or more object features comprise a location of text depicted within the object.

7. The method as recited in claim 1, wherein the one or more object features are selected from a group consisting of:

one or more dimensions of the object;

an object shape;

an object color; and

one or more reference features of the object class.

8. The method as recited in claim 1, comprising rectangularizing the object based at least in part on the one or more object features.

9. The method as recited in claim 8, wherein the one or more object features comprise one or more known characteristics describing a true configuration of the object.

10. The method as recited in claim 9, wherein the rectangularizing comprises transforming the object from a native appearance within the digital image to the true configuration based at least in part on the known characteristics, rather than estimating the true configuration based on the native appearance of the object within the digital image.

11. The method as recited in claim 1, comprising predicting an identity of text depicted on the object using optical character recognition (OCR), wherein the predicting is based at least in part on the particular object class.

12. The method as recited in claim 11, comprising modifying at least one identity predicted using OCR, the modifying being based at least in part on an expected format of text depicted on the object;

and

wherein the expected format is determined based on the particular object class.

13. The method as recited in claim 1, comprising cropping the digital image based at least in part on the one or more object features.

14. A computer program product, comprising a non-transitory computer readable medium having stored thereon program instructions readable/executable by a processor of a mobile device to cause the processor to:

generate a first representation of a digital image, the first representation being characterized by a reduced resolution;

generate a first feature vector based on the first representation;

compare the first feature vector to a plurality of reference feature matrices;

classify an object depicted in the digital image as a member of a particular object class based at least in part on the comparing; and

determine one or more object features of the object based at least in part on the particular object class; and

detect the object within the digital image based on the one or more object features, wherein the detecting comprises directly detecting an object color profile within the digital image rather than detecting a transition from a first color profile to a second color profile.

15. The computer program product as recited in claim 14, wherein the one or more object features are selected from a group consisting of:

one or more dimensions of the object;

an object shape;

an object color; and

one or more reference features of the object class.

16. A computer-implemented method, comprising:

receiving a digital image captured by a mobile device; and

using a processor of the mobile device;

generating a first representation of a digital image, the first representation being characterized by a reduced resolution;

generating a first feature vector based on the first representation;

comparing the first feature vector to a plurality of reference feature matrices;

classifying an object depicted in the digital image as a member of a particular object class based at least in part on the comparing; and

determining one or more object features of the object based at least in part on the particular object class;

predicting an identity of text depicted on the object using optical character recognition (OCR), wherein the predicting is based at least in part on the particular object class; and

modifying at least one identity predicted using OCR, the modifying being based at least in part on an expected format of text depicted on the object, wherein the expected format is determined based on the particular object class.

17. A computer program product, comprising a non-transitory computer readable medium having stored thereon program instructions readable/executable by a processor of a mobile device to cause the processor to:

generate a first representation of a digital image, the first representation being characterized by a reduced resolution;

generate a first feature vector based on the first representation;

compare the first feature vector to a plurality of reference feature matrices;

classify an object depicted in the digital image as a member of a particular object class based at least in part on the comparing; and

determine one or more object features of the object based at least in part on the particular object class;

predict an identity of text depicted on the object using optical character recognition (OCR), wherein the predicting is based at least in part on the particular object class; and

modify at least one identity predicted using OCR, the modifying being based at least in part on an expected format of text depicted on the object, wherein the expected format is determined based on the particular object class.