Devices and methods for enhanced image capture of documents

• US 10,440,196 B2
• Filed: 11/30/2018
• Issued: 10/08/2019
• Est. Priority Date: 10/06/2017
• Status: Active Grant

• Current Assignee: Toronto-Dominion Bank
• Original Assignee: Toronto-Dominion Bank
• Inventors: Horvath, Peter, Kaiser, Jr., Eric Paul, Gervais, Steven, Lee, John Jong-Suk, Jagga, Arun Victor
• Primary Examiner(s): Liew, Alex Kok S
• Application Number:
  US16/205,804
• Publication Number:
  US 20190109951A1
• Time in Patent Office: 312 Days
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/412 Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: H04N 1/00129 with a display device, e.g. CRT or LCD monitor with a television apparatus H04N1/00283; arrangements for control of display devices G09G
• CPC Class Codes: H04N 1/19594 using a television camera or a still video camera
• CPC Class Codes: H04N 1/32106 separate from the image data, e.g. in a different computer file
• CPC Class Codes: H04N 2201/0084 Digital still camera
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A computer-implemented method for processing images of a document is disclosed. The method includes: capturing an image of at least part of a document by a camera of an electronic device, the document having one or more data fields; determining, by the electronic device, one or more data entry items from analyzing the one or more data fields in the image; and displaying, on a display of the electronic device, a live view image of the document with annotation overlays on the live view image, each annotation overlay containing a respective one of the one or more data entry items and being displayed in association with its respective data field in the live view image.

Claims

1. An electronic device comprising:

a display;

a camera;

a memory storing an image capture application module; and

a processor coupled to the display, the camera and the memory, the processor being configured to;

receive an image of at least part of a document captured by the camera, the document having one or more data fields;

determine, from the image, one or more data entry items representing content of the one or more data fields; and

display, on the display, a live view image of the document with annotation overlays on the live view image, each annotation overlay containing a respective one of the one or more data entry items and being displayed in association with its respective data field in the live view image.

2. The electronic device of claim 1, wherein the processor is further configured to display, on the display, a first user interface element for initiating a first action using the one or more data entry items, the first user interface element being displayed concurrently with the live view image of the document.

3. The electronic device of claim 2, wherein the processor is further configured to:

receive a selection of the first user interface element; and

in response to receiving the selection of the first user interface element, transmit, to a remote server, at least a subset of the one or more data entry items in association with respective corresponding data fields.

4. The electronic device of claim 1, wherein determining the one or more data entry items comprises:

identifying regions of the image corresponding to the one or more data fields;

for each of the identified regions;

detecting an image representation of a data entry item in the identified region; and

converting the image representation of the data entry item to text format.

5. The electronic device of claim 4, wherein converting the image representation of the data entry item to text format comprises processing the image representation of the data entry item using optical character recognition.

6. The electronic device of claim 1, wherein one or more of the annotation overlays on the live view image comprise editable text fields.

7. The electronic device of claim 1, wherein the annotation overlays associated with the one or more data entry items are positioned in predetermined regions of the live view image associated with the respective corresponding data fields.

8. The electronic device of claim 7, wherein a predetermined region of the live view image associated with a data field is defined by boundaries surrounding a representation of the data field on the live view image.

9. The electronic device of claim 1, wherein an annotation overlay associated with a first one of the one or more data entry item is positioned in spaced relation to a representation of the first one data entry item on the live view image.

10. The electronic device of claim 1, wherein the processor is configured to automatically detect a type associated with the document and wherein the displaying the live view image of the document is performed in response to detecting that the document is of a first type.

11. A computer-implemented method, comprising:

capturing an image of at least part of a document by a camera of an electronic device, the document having one or more data fields;

determining, by the electronic device, from the image one or more data entry items representing content of the one or more data fields; and

displaying, on a display of the electronic device, a live view image of the document with annotation overlays on the live view image, each annotation overlay containing a respective one of the one or more data entry items and being displayed in association with its respective data field in the live view image.

12. The method of claim 11, further comprising displaying, on the display, a first user interface element for initiating a first action using the one or more data entry items, the first user interface element being displayed concurrently with the live view image of the document.

13. The method of claim 12, further comprising:

receiving a selection of the first user interface element; and

in response to receiving the selection of the first user interface element, transmitting, to a remote server, at least a subset of the one or more data entry items in association with respective corresponding data fields.

14. The method of claim 11, wherein determining the one or more data entry items comprises:

identifying regions of the image corresponding to the one or more data fields;

for each of the identified regions;

detecting an image representation of a data entry item in the identified region; and

converting the image representation of the data entry item to text format.

15. The method of claim 14, wherein converting the image representation of the data entry item to text format comprises processing the image representation of the data entry item using optical character recognition.

16. The method of claim 11, wherein one or more of the annotation overlays on the live view image comprise editable text fields.

17. The method of claim 11, wherein the annotation overlays associated with the one or more data entry items are positioned in predetermined regions of the live view image associated with the respective corresponding data fields.

18. The method of claim 17, wherein a predetermined region of the live view image associated with a data field is defined by boundaries surrounding a representation of the data field on the live view image.

19. The method of claim 11, wherein an annotation overlay associated with a first one of the one or more data entry item is positioned in spaced relation to a representation of the first one data entry item on the live view image.

20. A non-transitory computer-readable storage medium comprising processor-executable instructions which, when executed, configure a processor to:

receive an image of at least part of a document captured by a camera of an electronic device, the document having one or more data fields;

determine, from the image, one or more data entry items representing content of the one or more data fields in the image; and

display, on a display of the electronic device, a live view image of the document with annotation overlays on the live view image, each annotation overlay containing a respective one of the one or more data entry items and being displayed in association with its respective data field in the live view image.