Systems and methods for detecting and classifying objects in video captured using mobile devices

• US 8,885,229 B1
• Filed: 05/02/2014
• Issued: 11/11/2014
• Est. Priority Date: 05/03/2013
• Status: Active Grant

• Current Assignee: Tungsten Automation Corp.
• Original Assignee: Kofax Incorporated
• Inventors: Amtrup, Jan W., Ma, Jiyong, Macciola, Anthony
• Primary Examiner(s): Baker, Charlotte M
• Assistant Examiner(s): GRISHAM, RURY L
• Application Number:
  US14/268,876
• Publication Number:
  US 20140327940A1
• Time in Patent Office: 193 Days
• Field of Search: 358/473, 358/1.15, 358/1.18, 382/190, 382/137, 382/182, 382/293, 382/274, 382/124
• US Class Current: 358/473
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06V 2201/09 Recognition of logos
• CPC Class Codes: G06V 30/224 of printed characters having additional code marks or containing code marks
• CPC Class Codes: H04N 1/00251 with an apparatus for taking photographic images, e.g. a camera
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/00801 according to characteristics of the original
• CPC Class Codes: H04N 1/19594 using a television camera or a still video camera
• CPC Class Codes: H04N 2201/0032 where the still picture apparatus acts as the slave
• CPC Class Codes: H04N 2201/0075 by a user operated remote control device, e.g. receiving instructions from a user via a computer terminal or mobile telephone handset
• CPC Class Codes: H04N 2201/0084 Digital still camera
• CPC Class Codes: H04N 2201/0458 Additional arrangements for improving or optimising scanning resolution or quality
• CPC Class Codes: H04N 23/635 Region indicators; Field of view indicators
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 5/265 Mixing

Abstract

A method includes invoking an image capture interface via a mobile device; and analyzing video data captured via the capture interface. The analysis includes determining whether an object exhibiting one or more defining characteristics is depicted within the viewfinder; and if so, whether that object satisfies one or more predetermined quality control criteria. The method further includes displaying an indication of success or failure to satisfy the predetermined control criteria on the mobile device display. Where the object depicted within the viewfinder satisfies the one or more predetermined quality control criteria, the method also includes: displaying an indication that the object depicted in the viewfinder exhibits the one or more defining characteristics; automatically capturing an image of the object; and/or automatically storing to memory one or more of the frames in which the object is depicted in the viewfinder. Systems and computer program products are also disclosed.

Claims

1. A method, comprising:

invoking an image capture interface via a mobile device, the capture interface comprising a viewfinder represented on a display of the mobile device;

analyzing a plurality of frames of video data captured via the capture interface, wherein the analyzing comprises determining;

whether an object exhibiting one or more defining characteristics is depicted within the viewfinder; and

whether the object depicted within the viewfinder satisfies one or more predetermined quality control criteria; and

in response to determining a frame fails one or more of the predetermined quality control criteria, displaying an indication of the failure on the mobile device display; and

in response to determining the object depicted within the viewfinder satisfies the one or more predetermined quality control criteria, one or more of;

displaying an indication that the object depicted in the viewfinder exhibits the one or more defining characteristics;

automatically capturing an image of the object, wherein the image is characterized by a resolution higher than a resolution of the video data; and

automatically storing to a memory one or more of the frames in which the object satisfying the predetermined quality control criteria is depicted in the viewfinder.

2. The method as recited in claim 1, wherein the object comprises a document.

3. The method as recited in claim 1, further comprising processing at least one of the automatically captured image and the automatically stored frame(s) at least in part using a processor of the mobile device, wherein the processing comprises one or more of:

tracking the object depicted within the viewfinder in real-time

classifying the object depicted within the viewfinder; and

extracting data from the object depicted in the viewfinder.

4. The method as recited in claim 3, wherein the object comprises multiple documents or multiple pages of a single document.

5. The method as recited in claim 4, therein the processing is performed with respect to at least two of the multiple documents or multiple pages.

6. The method as recited in claim 4, wherein the processing comprises simultaneously tracking a plurality of the multiple documents or multiple pages across multiple of the frames.

7. The method as recited in claim 4, wherein the processing comprises simultaneously classifying a plurality of the multiple documents or multiple pages.

8. The method as recited in claim 4, wherein the processing comprises simultaneously extracting at least some data from a plurality of the multiple documents or multiple pages.

9. The method as recited in claim 3, wherein the classifying comprises:

determining one or more defining characteristics of the object depicted within the viewfinder; and

comparing the one or more determined defining characteristics to defining characteristics of each of a plurality of object classifications.

10. The method as recited in claim 9, further comprising either:

determining an object classification based at least in part on the comparison;

or

determining the object does not correspond to any of the plurality of object classifications based at least in part on the comparison.

11. The method as recited in claim 10, further comprising, in response to determining the object does not correspond to any of the plurality of object classifications:

requesting user input relating to the object;

receiving the user input;

defining a new object classification based at least in part on the user input; and

assigning the object to the new object classification.

12. The method as recited in claim 1, wherein the indication comprises a bounding border surrounding a periphery of the object depicted within the viewfinder, and

wherein the method further comprises displaying the bounding border around a periphery of the object.

13. The method as recited in claim 12, wherein the tracking comprises one or more of:

repositioning or redefining the bounding border to surround the periphery of the object in each of the frames where the object is depicted within the viewfinder; and

repositioning or redisplaying the indication that the object depicted in the viewfinder exhibits the one or more defining characteristics.

14. The method as recited in claim 3, wherein the tracking further comprises receiving real-time feedback from the mobile device, the real-time feedback being based at least in part on one or more measurements performed using one or more mobile device components selected from:

a camera, an accelerometer, a gyroscope, and a clock.

15. The method as recited in claim 14, wherein the real-time feedback comprises stability feedback including an angle of orientation of the mobile device being within a predetermined orientation range;

and a motion vector of the mobile device having a magnitude less than a predetermined threshold.

16. The method as recited in claim 15, wherein the motion vector of the mobile device is determined based on real-time feedback received from the camera, and wherein the motion vector of the mobile device is not determined based on feedback from the accelerometer.

17. The method as recited in claim 1, further comprising synthesizing at least a portion of two or more frames of the video data;

and

generating a composite image based on the synthesizing.

18. The method as recited in claim 17, wherein the composite image is characterized by a height and a width,

wherein the composite image height is greater than or equal to a height of any single frame of the video data, and

wherein the composite image width is greater than or equal to a width of any single frame of the video data.

19. The method as recited in claim 17, wherein each of the synthesized frames of the video data depicts a portion of the object, and wherein the composite image depicts an entirety of the object.

20. The method as recited in claim 17, wherein the synthesizing comprises:

detecting a first feature of the object depicted in the viewfinder;

automatically initiating a capture operation in response to detecting the first feature of the object;

capturing one or more of high-resolution image data and low-resolution video data via the automatically initiated capture operation;

detecting a second feature of the object depicted in the viewfinder;

capturing one or more of high-resolution image data and low-resolution video data via the automatically initiated capture operation; and

automatically terminating the capture operation in response to detecting the second feature of the object.

21. The method as recited in claim 17, wherein the synthesizing comprises:

performing at least one homography transformation on two or more of the frames of the video data, and

aligning at least portions of the two or more frames of the video data based at least in part on the homography transformations.

22. The method as recited in claim 21, wherein the object is a document characterized by at least one dimension being too large to encompass the entire document within the viewfinder.

23. The method as recited in claim 19, wherein at least a portion of the composite image is characterized by a higher resolution than a resolution of any of the two or more frames of the video data.

24. The method as recited in claim 23, further comprising detecting the object in the composite image based at least in part on the portion of the composite image characterized by the higher resolution.

25. The method as recited in claim 1, further comprising:

determining whether the one or more frames satisfying the one or more predefined control criteria correspond to a high-resolution image stored on the mobile device; and

processing the stored high-resolution image.

26. The method as recited in claim 1, further comprising associating metadata with the stored video data.

27. The method as recited in claim 1, wherein the indication of the failure displayed on the mobile device specifically identifies the one or more quality control criteria not satisfied by the frame(s).

28. The method as recited in claim 1, wherein the predetermined quality control criteria comprise one or more of:

a minimum illumination level;

a maximum illumination level;

a minimum illumination evenness;

a minimum resolution;

a minimum sharpness;

a minimum projection angle;

a maximum projection angle;

a threshold visibility criterion;

a presence of glare; and

an object classification.

29. A system, comprising:

a processor; and

logic in and/or executable by the processor to cause the processor to;

invoke an image capture interface via a mobile device, the capture interface comprising a viewfinder represented on a display of the mobile device;

analyze a plurality of frames of video data captured via the capture interface, wherein the analysis comprises;

determining whether an object exhibiting one or more defining characteristics is depicted within the viewfinder; and

determining whether the object depicted within the viewfinder satisfies one or more predetermined quality control criteria; and

in response to determining a frame fails one or more of the predetermined quality control criteria, display an indication of the failure on the mobile device display; and

in response to determining the object depicted within the viewfinder satisfies the one or more predetermined quality control criteria, one or more of;

display an indication that the object depicted in the viewfinder exhibits the one or more defining characteristics;

automatically capture an image of the object, wherein the image is characterized by a resolution higher than a resolution of the video data; and

automatically store to a memory one or more of the frames in which the object satisfying the predetermined quality control criteria is depicted in the viewfinder.

30. A computer program product comprising:

a non-transitory computer readable storage medium having program code embodied therewith, the program code readable/executable by a processor to cause the processor to;

invoke an image capture interface via a mobile device, the capture interface comprising a viewfinder represented on a display of the mobile device;

analyze a plurality of frames of video data captured via the capture interface, wherein the analysis comprises;

determining whether an object exhibiting one or more defining characteristics is depicted within the viewfinder; and

determining whether the object depicted within the viewfinder satisfies one or more predetermined quality control criteria; and

in response to determining a frame fails one or more of the predetermined quality control criteria, display an indication of the failure on the mobile device display; and

in response to determining the object depicted within the viewfinder satisfies the one or more predetermined quality control criteria, one or more of;

display an indication that the object depicted in the viewfinder exhibits the one or more defining characteristics;

automatically capture an image of the object, wherein the image is characterized by a resolution higher than a resolution of the video data; and

automatically store to a memory one or more of the frames in which the object satisfying the predetermined quality control criteria is depicted in the viewfinder.