Systems and methods for generating composite images of long documents using mobile video data

• US 9,747,504 B2
• Filed: 06/23/2016
• Issued: 08/29/2017
• Est. Priority Date: 11/15/2013
• Status: Active Grant

• Current Assignee: Kofax Incorporated
• Original Assignee: Kofax Incorporated
• Inventors: Ma, Jiyong, Macciola, Anthony, Amtrup, Jan W.
• Primary Examiner(s): COLEMAN, STEPHEN P
• Application Number:
  US15/191,442
• Publication Number:
  US 20160307045A1
• Time in Patent Office: 432 Days
• Field of Search: None
• CPC Class Codes: G06Q 20/047 using payment protocols involving electronic receipts
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 3/14 Transformations for image registration, e.g. adjusting or mapping for alignment of images
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 30/413 Classification of content, e.g. text, photographs or tables
• CPC Class Codes: H04N 1/3876 Recombination of partial images to recreate the original image
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 5/265 Mixing

Abstract

Techniques for capturing long document images and generating composite images therefrom include: detecting a document depicted in image data; tracking a position of the detected document within the image data; selecting a plurality of images, wherein the selection is based at least in part on the tracked position of the detected document; and generating a composite image based on at least one of the selected plurality of images. The tracking and selection are optionally but preferably based in whole or in part on motion vectors estimated at least partially based on analyzing image data such as test and reference frames within the captured video data/images. Corresponding systems and computer program products are also disclosed.

Claims

1. A computer program product comprising a non-transitory computer readable medium having stored thereon instructions executable by a processor of a mobile device, the instructions being configured to cause the processor, upon execution thereof, to generate a composite image of a long document with sufficient resolution for downstream processing by:

detecting a long document depicted in image data;

tracking a position of the detected long document within the image data;

selecting a plurality of images, wherein the selection is based at least in part on the tracked position of the detected long document; and

generating a composite image of the long document based on at least two of the selected plurality of images, wherein the composite image of the long document is characterized by a resolution greater than a resolution of any of the selected plurality of images, wherein the resolution of the composite image is at least about 200 dots per inch (DPI) or at least about 200 pixels per inch (PPI).

2. The computer program product as recited in claim 1, further comprising instructions configured to cause the processor to identify at least one edge of the document depicted in the image data.

3. The computer program product as recited in claim 1, wherein each of the selected plurality of images depicts a portion of the document, and

wherein the composite image depicts an entirety of the document.

4. The computer program product as recited in claim 1, wherein the tracking comprises generating, using the processor, alignment hypotheses between at least some of the plurality of frames of image data, wherein the alignment hypotheses are generated based on matching sampled features between frames of the image data.

5. The computer program product as recited in claim 1, further comprising instructions configured to cause the processor to:

estimate one or more motion vectors corresponding to motion of an image capture component used to capture the image data.

6. The computer program product as recited in claim 5, wherein the selection is further based at least in part on the one or more estimated motion vector.

7. The computer program product as recited in claim 5, wherein the tracking is based exclusively on the estimated motion vector(s).

8. The computer program product as recited in claim 5, further comprising instructions configured to cause the processor to:

determine at least one motion displacement based on some or all of the estimated motion vector(s);

either terminate or pause a capture operation in response to determining one of the motion displacement(s) is characterized by a value exceeding a predefined motion displacement threshold; and

either

initiate a new capture operation in response to terminating the capture operation;

or

resume the capture operation in response to pausing the capture operation.

9. The computer program product as recited in claim 1, further comprising instructions configured to cause the processor to:

identify, based on the composite image, one or more portions of the document depicting textual information;

classify each identified portion of the document based on the textual information depicted therein;

determine whether each classified portion is relevant to a financial transaction or irrelevant to the financial transaction, the determination being based on the portion classification; and

remove each portion determined to be irrelevant to the financial transaction from the composite image.

10. The computer program product as recited in claim 1, wherein generating the composite image comprises:

estimating a homograph transform matrix or an affine transform matrix, wherein the estimation is based on text block matching between the selected plurality of images; and

transforming one of the plurality of images to a coordinate system of another of the plurality of images using the homograph transform matrix or the affine transform matrix.

11. The computer program product as recited in claim 1, the instructions configured to cause the processor to select the plurality of images further comprising instructions configured to cause the processor to define at least one frame pair,

wherein each frame pair consists of a reference frame and a test frame, and

wherein each reference frame and each test frame are selected from the image data.

12. The computer program product as recited in claim 11, the instructions configured to cause the processor to generate the composite image further comprising instructions configured to cause the processor to:

detect a skew angle in one or more of the reference frame and the test frame of at least one of the frame pairs, the skew angle corresponding to the document and having a magnitude of >

0.0 degrees; and

correct the skew angle in at least one of the reference frame and the test frame,

wherein the document depicted in the composite image is characterized by a skew angle of approximately 0.0 degrees.

13. The computer program product as recited in claim 11, the instructions configured to cause the processor to select the plurality of images further comprising instructions configured to cause the processor to:

determine an amount of overlap between the reference frame and the test frame of at least one frame pair; and

select an image corresponding to at least one frame pair for which the amount of overlap between the reference frame and the test frame is greater than a predetermined overlap threshold.

14. The computer program product as recited in claim 13, wherein the amount of overlap corresponds to the document;

and

wherein the predetermined overlap threshold is a distance of at least 40% of a length of the reference frame.

15. The computer program product as recited in claim 11, the instructions configured to cause the processor to generate the composite image further comprising instructions configured to cause the processor to:

detect textual information in each of the reference frame and the test frame of at least one frame pair, the textual information being depicted in the document.

16. The computer program product as recited in claim 15, the instructions configured to cause the processor to detect textual information in each of the reference frame and the test frame of at least one frame pair further comprising instructions configured to cause the processor to:

define, in the reference frame, at least one rectangular portion of the document depicting some or all of the textual information;

define, in the test frame, at least one corresponding rectangular portion of the document depicting some or all of the textual information; and

align the document depicted in the test frame with the document depicted in the reference frame.

17. The computer program product as recited in claim 16, wherein the textual information comprises at least one feature selected from a group consisting of:

an identity of one or more characters represented in the rectangular portion;

an identity of one or more characters represented in the corresponding rectangular portion;

a sequence of characters represented in the rectangular portion;

a sequence of characters represented in the corresponding rectangular portion;

a position of one or more characters represented in the rectangular portion;

a position of one or more characters represented in the corresponding rectangular portion;

an absolute size of one or more characters represented in the rectangular portion;

an absolute size of one or more characters represented in the corresponding rectangular portion

a size of one or more characters represented in the rectangular portion relative to a size of one or more characters represented in the corresponding rectangular portion;

a size of one or more characters represented in the corresponding rectangular portion relative to a size of one or more characters represented in the rectangular portion;

a color of one or more characters represented in the rectangular portion;

a color of one or more characters represented in the corresponding rectangular portion;

a shape of one or more characters represented in the rectangular portion; and

a shape of one or more characters represented in the corresponding rectangular portion.

18. The computer program product as recited in claim 16, the instructions configured to cause the processor to align the document depicted in the test frame with the document depicted in the reference frame further comprising instructions configured to cause the processor to perform optical character recognition (OCR) on at least the rectangular portion and the corresponding rectangular portion.

19. A mobile device having logic embodied therewith, the logic being configured to cause the mobile device, upon execution thereof, to generate an image of a long document sufficient for downstream processing by:

detecting, using a processor of the mobile device, a long document depicted in image data;

tracking, using the processor of the mobile device, a position of the detected long document within the image data;

selecting, using the processor of the mobile device, a plurality of images, wherein the selection is based at least in part on the tracked position of the detected long document; and

generating, using the processor of the mobile device, a composite image of the long document based on at least two of the selected plurality of images wherein the composite image of the long document is characterized by a resolution greater than a resolution of any of the selected plurality of images, wherein the resolution of the composite image is at least about 200 dots per inch (DPI) or at least about 200 pixels per inch (PPI).

20. A computer-implemented method for generating a composite image of a long document suitable for downstream processing, the method comprising:

tracking, using a processor of a mobile device, a long document within a plurality of frames of image data;

selecting, using the processor, a subset of the plurality of frames of the image data based on the tracking;

generating, using the processor, alignment hypotheses between at least some of the selected subset of frames of image data, wherein the alignment hypotheses are generated based on matching sampled features of one or more reference frames of the image data with sampled features of one or more test frames of the image data;

storing at least some of the selected frames of the image data to a memory of the mobile device; and

generating, using the processor, a composite image of the long document by stitching together at least two of the selected subset of frames;

wherein the at least two of the selected subset of frames are characterized by an overlap greater than a predefined overlap threshold; and

wherein the composite image is characterized by a resolution of at least about 200 dots per inch (DPI) or a resolution of at least about 200 pixels per inch (PPI).