Systems and methods for generating composite images of long documents using mobile video data

• US 10,108,860 B2
• Filed: 12/23/2016
• Issued: 10/23/2018
• Est. Priority Date: 11/15/2013
• Status: Active Grant

• Current Assignee: Kofax Incorporated
• Original Assignee: Kofax Incorporated
• Inventors: Ma, Jiyong, Macciola, Anthony, Amtrup, Jan W.
• Primary Examiner(s): Coleman, Stephen P
• Application Number:
  US15/390,321
• Publication Number:
  US 20170109588A1
• Time in Patent Office: 669 Days
• Field of Search: None
• CPC Class Codes: G06Q 20/047 using payment protocols involving electronic receipts
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 3/14 Transformations for image registration, e.g. adjusting or mapping for alignment of images
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 30/413 Classification of content, e.g. text, photographs or tables
• CPC Class Codes: H04N 1/3876 Recombination of partial images to recreate the original image
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 5/265 Mixing

Abstract

According to one embodiment, a system includes a processor and logic in and/or executable by the processor to cause the processor to: initiate a capture operation using an image capture component of the mobile device, the capture operation comprising; capturing video data; and estimating a plurality of motion vectors corresponding to motion of the image capture component during the capture operation; detect a document depicted in the video data; track a position of the detected document throughout the video data; select a plurality of images using the image capture component of the mobile device, wherein the selection is based at least in part on: the tracked position of the detected document; and the estimated motion vectors; and generate a composite image based on at least some of the selected plurality of images.

Claims

1. A system comprising a processor and logic in and/or executable by the processor to cause the processor to:

initiate a capture operation using an image capture component of a mobile device, the capture operation comprising;

capturing video data; and

estimating a plurality of motion vectors corresponding to motion of the image capture component during the capture operation;

detect a document depicted in the video data;

track a position of the detected document throughout the video data;

select a plurality of images using the image capture component of the mobile device, wherein the selection is based at least in part on;

the tracked position of the detected document; and

the estimated motion vectors; and

generate a composite image based on at least some of the selected plurality of images;

wherein the document is a long document; and

wherein the composite image is characterized by a resolution sufficient for downstream processing.

2. The system as recited in claim 1,

wherein the composite image is a composite of two or more of the selected plurality of images.

3. The system as recited in claim 1, wherein the tracking is based exclusively on the estimated plurality of motion vectors.

4. The system as recited in claim 1, the logic configured to cause the processor to detect the document further comprising logic configured to cause the mobile device to identify at least one edge of the document depicted in the captured video data.

5. The system as recited in claim 1, wherein each of the selected plurality of images depicts a portion of the document, and

wherein the composite image depicts an entirety of the document.

6. The system as recited in claim 1, wherein the composite image is characterized by at least one of:

an image resolution greater than an image resolution of any of the selected plurality of images; and

an image size greater than an image size of any of the selected plurality of images.

7. The system as recited in claim 1, further comprising logic configured to cause the processor to downsample the video data, and

wherein the logic configured to cause the processor to detect the document, track the position of the document, and select the plurality of images is configured to perform the detecting, the tracking, and the selecting using the downsampled video data.

8. The system as recited in claim 1, further comprising logic configured to cause the processor to:

determine at least one motion displacement based on some or all of the estimated plurality of motion vectors, each motion displacement corresponding to motion of the image capture component during the capture operation; and

either terminate or pause the capture operation in response to determining one of the motion displacement(s) is characterized by a value exceeding a predefined motion displacement threshold; and

either initiate a new capture operation in response to terminating the capture operation;

or

resume the capture operation in response to pausing the capture operation.

9. The system as recited in claim 8, wherein the predefined motion displacement threshold has a value in a range from about 5 pixels to about 25 pixels.

10. The system as recited in claim 1, wherein each selected image depicts a portion of the document, and

wherein the composite image depicts only portion(s) of the document that correspond to a financial transaction memorialized by the document.

11. The system as recited in claim 10, further comprising logic configured to cause the processor to:

identify, based on the composite image, one or more portions of the document depicting textual information;

classify each identified portion of the document based on the textual information depicted therein;

determine whether each classified portion is relevant to the financial transaction or irrelevant to the financial transaction, the determining being based on the portion classification; and

remove each portion determined to be irrelevant to the financial transaction from the composite image.

12. The system as recited in claim 11, further comprising logic configured to cause the processor to:

align the portions determined to be relevant to the financial transaction; and

generate a second composite image, wherein the second composite image is characterized by;

approximately a same image size as an image size of the composite image;

approximately a same image resolution as an image resolution of the composite image;

excluding textual information irrelevant to the financial transaction; and

including textual information relevant to the financial transaction, wherein a plurality of characters comprising the textual information relevant to the financial transaction are aligned.

13. The system as recited in claim 1, the logic configured to cause the processor to select the plurality of images further comprising logic configured to cause the processor to define a plurality of frame pairs,

wherein each frame pair consists of a reference frame and a test frame, and

wherein each reference frame and each test frame is selected from the video data.

14. The system as recited in claim 13, the logic configured to cause the processor to generate the composite image further comprising logic configured to cause the processor to:

detect a skew angle in one or more of the reference frame and the test frame of at least one of the frame pairs, the skew angle corresponding to the document and having a magnitude of >

0.0 degrees; and

correct the skew angle in at least one of the reference frame and the test frame,

wherein the document depicted in the composite image is characterized by a skew angle of approximately 0.0 degrees.

15. The system as recited in claim 13, the logic configured to cause the processor to select the plurality of images further comprising logic configured to cause the processor to:

determine an amount of overlap between the reference frame and the test frame of each frame pair; and

select an image corresponding to each frame pair for which the amount of overlap between the reference frame and the test frame is greater than a predetermined overlap threshold.

16. The system as recited in claim 15, wherein the amount of overlap corresponds to the document,

wherein the amount of overlap does not correspond to a background depicted in the reference frame, and

wherein the amount of overlap does not correspond to a background depicted in the test frame.

17. The system as recited in claim 15, wherein the predetermined overlap threshold corresponds to a distance of at least 40% of a length of the reference frame.

18. The system as recited in claim 13, the logic configured to cause the processor to generate the composite image further comprising logic configured to cause the processor to:

detect textual information in each of the reference frame and the test frame of at least one frame pair, the textual information being depicted in the document.

19. The system as recited in claim 18, the logic configured to cause the processor to detect textual information in each of the reference frame and the test frame of at least one frame pair further comprising logic configured to cause the processor to:

define, in the reference frame, at least one rectangular portion of the document depicting some or all of the textual information;

define, in the test frame, at least one corresponding rectangular portion of the document depicting some or all of the textual information; and

align the document depicted in the test frame with the document depicted in the reference frame, the alignment being based on;

the textual information depicted in at least one of the rectangular portion(s); and

the textual information depicted in at least one of the corresponding rectangular portion(s).

20. The system as recited in claim 19, wherein the textual information comprises at least one of:

an identity of one or more characters represented in the rectangular portion;

an identity of one or more characters represented in the corresponding rectangular portion;

a sequence of characters represented in the rectangular portion;

a sequence of characters represented in the corresponding rectangular portion;

a position of one or more characters represented in the rectangular portion;

a position of one or more characters represented in the corresponding rectangular portion;

an absolute size of one or more characters represented in the rectangular portion;

an absolute size of one or more characters represented in the corresponding rectangular portion

a size of one or more characters represented in the rectangular portion relative to a size of one or more characters represented in the corresponding rectangular portion;

a size of one or more characters represented in the corresponding rectangular portion relative to a size of one or more characters represented in the rectangular portion;

a color of one or more characters represented in the rectangular portion;

a color of one or more characters represented in the corresponding rectangular portion;

a shape of one or more characters represented in the rectangular portion; and

a shape of one or more characters represented in the corresponding rectangular portion.

21. The system as recited in claim 19, the logic configured to cause the processor to align the document depicted in the test frame with the document depicted in the reference frame further comprising logic configured to cause the processor to perform optical character recognition (OCR) on at least the rectangular portion and the corresponding rectangular portion.