Devices and methods for enhanced image capture of documents

• US 10,178,246 B1
• Filed: 10/06/2017
• Issued: 01/08/2019
• Est. Priority Date: 10/06/2017
• Status: Active Grant

• Current Assignee: Toronto-Dominion Bank
• Original Assignee: Toronto-Dominion Bank
• Inventors: Horvath, Peter, Kaiser, Jr., Eric Paul, Gervais, Steven, Lee, John Jong-Suk, Jagga, Arun Victor
• Primary Examiner(s): Liew, Alex
• Application Number:
  US15/726,429
• Time in Patent Office: 459 Days
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/412 Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: H04N 1/00129 with a display device, e.g. CRT or LCD monitor with a television apparatus H04N1/00283; arrangements for control of display devices G09G
• CPC Class Codes: H04N 1/19594 using a television camera or a still video camera
• CPC Class Codes: H04N 1/32106 separate from the image data, e.g. in a different computer file
• CPC Class Codes: H04N 2201/0084 Digital still camera
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A computer-implemented method for processing images of a document is disclosed. The method includes: displaying, on a display of an electronic device, a live view image of at least part of a document captured by a camera of the electronic device, the at least part of the document having imprinted thereon a plurality of data fields and corresponding data entry items, wherein displaying the live view image includes: receiving image data for a first image of the at least part of the document captured using the camera; extracting one or more of the data entry items from the document based on processing the first image; and displaying the extracted data entry items on the live view image of the at least part of the document, the extracted data entry items being displayed in annotation overlays on the live view image in association with representations of their respective corresponding data fields in the live view image. The method further includes displaying, on the display, a first user interface element concurrently with the live view image of the at least part of the document, the first user interface element being selectable to update a database record using the extracted data entry items.

Claims

1. An electronic device comprising:

a display;

at least one camera;

a memory storing an image capture application module; and

a processor coupled to the display, the at least one camera, and the memory, the processor being configured to;

display, on the display, a live view image of at least part of a document captured by the at least one camera, the at least part of the document having imprinted thereon a plurality of data fields and corresponding data entry items, wherein displaying the live view image includes;

receiving image data for a first image of the at least part of the document captured using the at least one camera;

extracting one or more of the data entry items from the document by processing the first image; and

displaying the extracted data entry items on the live view image of the at least part of the document, the extracted data entry items being displayed in annotation overlays on the live view image in association with representations of their respective corresponding data fields in the live view image, and

display, on the display, a first user interface element concurrently with the live view image of the at least part of the document, the first user interface element being selectable to update a database record using the extracted data entry items.

2. The electronic device of claim 1, wherein the processor is further configured to:

display, on the display, a second user interface element concurrently with the live view image of the at least part of the document, the first user interface element being selectable to confirm that information contained in the annotations is correct.

3. The electronic device of claim 1, wherein the processor is further configured to:

receive a selection of the first user interface element; and

in response to receiving the selection of the first user interface element, transmit, to a remote server, at least a subset of the extracted data entry items in association with corresponding data fields.

4. The electronic device of claim 1, wherein extracting the one or more of the data entry items from the document comprises:

identifying regions of the first image corresponding to one or more of the plurality of data fields;

for each of the identified regions;

detecting an image representation of a data entry item in the identified region; and

converting the image representation of the data entry item to text format.

5. The electronic device of claim 4, wherein converting the image representation of the data entry item to text format comprises processing the image representation of the data entry item using optical character recognition.

6. The electronic device of claim 1, wherein one or more of the annotation overlays on the live view image comprise editable text fields.

7. The electronic device of claim 1, wherein the annotation overlays associated with the extracted data entry items are positioned in predetermined regions of the live view image associated with the respective corresponding data fields.

8. The electronic device of claim 7, wherein a predetermined region of the live view image associated with a data field is defined by boundaries surrounding a representation of the data field on the live view image.

9. The electronic device of claim 1, wherein an annotation overlay associated with an extracted data entry item is positioned in spaced relation to a representation of the extracted data entry item on the live view image.

10. The electronic device of claim 1, wherein the processor is configured to automatically detect a type associated with the document and wherein the displaying the live view image of the at least part of the document is performed in response to detecting that the document is of a first type.

11. A computer-implemented method, comprising:

displaying, on a display of an electronic device, a live view image of at least part of a document captured by a camera of the electronic device, the at least part of the document having imprinted thereon a plurality of data fields and corresponding data entry items, wherein displaying the live view image includes;

receiving image data for a first image of the at least part of the document captured using the camera;

extracting one or more of the data entry items from the document by processing the first image; and

displaying the extracted data entry items on the live view image of the at least part of the document, the extracted data entry items being displayed in annotation overlays on the live view image in association with representations of their respective corresponding data fields in the live view image, and

displaying, on the display, a first user interface element concurrently with the live view image of the at least part of the document, the first user interface element being selectable to update a database record using the extracted data entry items.

12. The method of claim 11, further comprising:

displaying, on the display, a second user interface element concurrently with the live view image of the at least part of the document, the first user interface element being selectable to confirm that information contained in the annotations is correct.

13. The method of claim 11, further comprising:

receiving a selection of the first user interface element; and

in response to receiving the selection of the first user interface element, transmit, to a remote server, at least a subset of the extracted data entry items in association with their corresponding data fields.

14. The method of claim 11, wherein extracting the one or more of the data entry items from the document comprises:

identifying regions of the first image corresponding to one or more of the plurality of data fields;

for each of the identified regions;

detecting an image representation of a data entry item in the identified region; and

converting the image representation of the data entry item to text format.

15. The method of claim 14, wherein converting the image representation of the data entry item to text format comprises processing the image representation of the data entry item using optical character recognition.

16. The method of claim 11, wherein one or more of the annotation overlays on the live view image comprise editable text fields.

17. The method of claim 11, wherein the annotation overlays associated with the extracted data entry items are positioned in predetermined regions of the live view image associated with the respective corresponding data fields.

18. The method of claim 17, wherein a predetermined region of the live view image associated with a data field is defined by boundaries surrounding a representation of the data field on the live view image.

19. The method of claim 11, wherein an annotation overlay associated with an extracted data entry item is positioned in spaced relation to a representation of the extracted data entry item on the live view image.

20. A non-transitory computer-readable storage medium comprising processor-executable instructions which, when executed, configure a processor to:

display, on a display of an electronic device, a live view image of at least part of a document captured by at least one camera of the electronic device, the document having imprinted thereon a plurality of data fields and data entry items associated with one or more of the data fields, wherein displaying the live view image includes;

receiving image data for a first image of the at least part of the document captured using the at least one camera;

extracting one or more of the data entry items from the document by processing the first image; and

displaying the extracted data entry items on the live view image of the at least part of the document, the extracted data entry items being displayed in annotation overlays on the live view image in association with representations of their respective corresponding data fields in the live view image, and

display, on the display, a first user interface element concurrently with the live view image of the at least part of the document, the first user interface element being selectable to update a database record using the extracted data entry items.