Content-based detection and three dimensional geometric reconstruction of objects in image and video data

• US 9,779,296 B1
• Filed: 08/11/2016
• Issued: 10/03/2017
• Est. Priority Date: 04/01/2016
• Status: Active Grant

• Current Assignee: Tungsten Automation Corp.
• Original Assignee: Kofax Incorporated
• Inventors: Ma, Jiyong, Thompson, Stephen Michael, Amtrup, Jan W.
• Primary Examiner(s): Desire, Gregory M
• Application Number:
  US15/234,969
• Publication Number:
  US 20170286764A1
• Time in Patent Office: 418 Days
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/30176 Document
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: G06V 30/418 Document matching, e.g. of document images

Abstract

Systems, computer program products, and techniques for detecting objects depicted in digital image data are disclosed, according to various exemplary embodiments. The inventive concepts uniquely utilize internal features to accomplish object detection, thereby avoiding reliance on detecting object edges and/or transitions between the object and other portions of the digital image data, e.g. background textures or other objects. The inventive concepts thus provide an improvement over conventional object detection since objects may be detected even when edges are obscured or not depicted in the digital image data. In one aspect, a computer-implemented method of detecting an object depicted in a digital image includes: detecting a plurality of identifying features of the object, wherein the plurality of identifying features are located internally with respect to the object; and projecting a location of one or more edges of the object based at least in part on the plurality of identifying features.

Claims

1. A computer-implemented method of detecting an object depicted in a digital image, the method comprising:

detecting, using a hardware processor, a plurality of identifying features of the object, wherein the plurality of identifying features are located internally with respect to the object; and

projecting, using the hardware processor, a location of one or more edges of the object based at least in part on the plurality of identifying features; and

outputting the projected location of the one or more edges of the object to at least one of;

a display of a computer, and a non-transitory computer readable medium.

2. The computer-implemented method as recited in claim 1, wherein detecting the plurality of identifying features comprises analyzing a plurality of feature vectors each corresponding to pixels within a patch of the digital image to determine whether the patch includes a sharp transition in intensity.

3. The computer-implemented method as recited in claim 1, wherein detecting the plurality of identifying features comprises automatic feature zone discovery;

and

wherein automatic feature zone discovery comprises;

matching a plurality of pixels in the digital image to a plurality of corresponding pixels in a plurality of reference images to form a set of matching pairs, each matching pair including one pixel from the digital image and one pixel from one of the plurality of reference images; and

determining a subset of the matching pairs exhibiting a frequency within the set of matching pairs that is greater than a predetermined frequency threshold.

4. The computer-implemented method as recited in claim 1, comprising transforming the digital image to display the projected location of the one or more edges of the object.

5. The computer-implemented method as recited in claim 4, wherein at least portions of at least one of the one or more edges displayed in the transformed digital image are missing from the digital image.

6. The computer-implemented method as recited in claim 4, wherein the digital image is characterized by a complex background comprising a plurality of sharp intensity transitions not corresponding to edges of the object.

7. The computer-implemented method as recited in claim 1, wherein the plurality of identifying features comprise boilerplate content.

8. The computer-implemented method as recited in claim 1, comprising identifying a plurality of distinctive pixels within the plurality of identifying features of the object, wherein the distinctive pixels are located at positions within the digital image characterized by a sharp transition in intensity.

9. The computer-implemented method as recited in claim 1, comprising matching the digital image depicting the object to one of a plurality of reference images each depicting a known object type, wherein the matching comprises determining whether the object includes distinctive pixels that correspond to distinctive pixels present in one or more of the plurality of reference images.

10. The computer-implemented method as recited in claim 1, comprising:

matching the digital image depicting the object to one of a plurality of reference images each depicting a known object type; and

designating as an outlier a candidate match between a distinctive pixel in the digital image and one or more candidate corresponding distinctive pixels present in one of the plurality of reference images;

wherein the outlier is designated in response to determining a distance ratio is greater than a predetermined distance ratio threshold, wherein the distance ratio is a ratio describing;

a first distance between the distinctive pixel in the digital image and a first of the one or more candidate corresponding distinctive pixels; and

a second distance between the distinctive pixel in the digital image and a second of the one or more candidate corresponding distinctive pixels.

11. The computer-implemented method as recited in claim 1, comprising:

matching the digital image depicting the object to one of a plurality of reference images each depicting a known object type; and

designating as an outlier a candidate match between a distinctive pixel in the digital image and a candidate corresponding distinctive pixel present in one of the plurality of reference images in response to determining the candidate match is not unique.

12. The computer-implemented method as recited in claim 1, wherein at least a portion of one or more edges of the object for which the location is projected is missing in the digital image.

13. The computer-implemented method as recited in claim 1, wherein projecting the location of the one or more edges of the object is based on a mapping of key points within some or all of the plurality of identifying features to key points of a reference image depicting an object belonging to a same class as the object depicted in the digital image.

14. The computer-implemented method as recited in claim 1, comprising cropping the digital image based at least in part on the projected location of the one or more edges of the object;

wherein the cropped digital image depicts a portion of a background of the digital image surrounding the object; and

wherein the method comprises detecting one or more transitions between the background and the object within the cropped digital image.

15. The computer-implemented method as recited in claim 1, comprising:

cropping the digital image based at least in part on the projected location of the one or more edges of the object; and

classifying the object depicted within the cropped digital image.

16. The computer-implemented method as recited in claim 1, comprising:

generating a plurality of scaled images based on the digital image, each scaled image being characterized by a different resolution;

extracting one or more feature vectors from each scaled image; and

matching one or more of the scaled images to one of a plurality of reference images, each reference image depicting a known object type and being characterized by a known resolution.

17. The computer-implemented method as recited in claim 1, comprising:

attempting to detect the object within the digital image using a plurality of predetermined object detection models each corresponding to a known object type; and

determining a classification of the object based on a result of attempting to detect the object within the digital image using the plurality of predetermined object detection models; and

wherein the classification of the object is determined to be the known object type corresponding to one of the object detection models for which the attempt to detect the object within the digital image was successful.

18. A computer program product for detecting an object depicted in a digital image, comprising a non-transitory computer readable medium having stored thereon computer readable program instructions configured to cause a processor, upon execution thereof, to:

generate, using the processor, a plurality of scaled images based on the digital image, each scaled image being characterized by a different resolution;

extract, using the processor, one or more feature vectors from each scaled image;

match, using the processor, one or more of the scaled images to one of a plurality of reference images based on the one or more feature vectors, each reference image depicting a known object type and being characterized by a known resolution;

detect, using the processor, a plurality of identifying features of the object within the scaled image matched to the one of the plurality of reference images, wherein the plurality of identifying features are located internally with respect to the object; and

project, using the processor, a location of one or more edges of the object based at least in part on the plurality of identifying features.

19. The computer program product as recited in claim 18, comprising computer readable program instructions configured to cause the processor, upon execution thereof, to:

match the digital image depicting the object to one of a plurality of reference images each depicting a known object type.

20. A system for detecting an object depicted in a digital image, comprising a processor and logic embodied with and/or executable by the processor, the logic being configured to cause the processor, upon execution thereof, to:

detect a plurality of identifying features of the object, wherein the plurality of identifying features are located internally with respect to the object; and

project a location of one or more edges of the object based at least in part on the plurality of identifying features, wherein at least portions of at least one of the one or more edges of the object displayed in the digital image are missing from the digital image.