Optical character recognition of series of images

• US 9,996,760 B2
• Filed: 05/31/2016
• Issued: 06/12/2018
• Est. Priority Date: 05/13/2016
• Status: Active Grant

• Current Assignee: ABBYY Development LLC
• Original Assignee: ABBYY Development LLC
• Inventors: Kalyuzhny, Aleksey
• Primary Examiner(s): Abdi, Amara
• Application Number:
  US15/168,525
• Publication Number:
  US 20170330048A1
• Time in Patent Office: 742 Days
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 3/14 Transformations for image registration, e.g. adjusting or mapping for alignment of images
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06V 20/62 Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/147 Determination of region of interest
• CPC Class Codes: G06V 30/412 Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: G06V 30/418 Document matching, e.g. of document images

Abstract

Systems and methods are described for receiving a current image that partially overlaps with a previous image of a series of images of an original document; performing optical character recognition (OCR) of the current image, producing an OCR text and a corresponding text layout; identifying textual artifacts in the current and previous images, each represented by a sequence of symbols having a frequency of occurrence within the OCR text below a threshold frequency; identifying corresponding base points associated with textual artifacts; identifying parameters of a coordinate transformation converting coordinates of the previous image into coordinates of the current image; associating part of the OCR text with a cluster of symbol sequences, wherein the symbol sequences are produced by processing previously received images; identifying an order of clusters of symbol sequences reflecting a layout of the original document; and producing a resulting OCR text representing a portion of the original document.

Claims

1. A method, comprising:

receiving, by a processing device, a current image of a series of images of an original document, wherein the current image at least partially overlaps with a previous image of the series of images;

performing optical character recognition (OCR) of the current image to produce an OCR text and a corresponding text layout;

identifying, using the OCR text and the corresponding text layout, a plurality of textual artifacts in each of the current image and the previous image, wherein each textual artifact is represented by a sequence of symbols that has a frequency of occurrence within the OCR text falling below a threshold frequency;

identifying, in each of the current image and the previous image, a corresponding plurality of base points, wherein each base point is associated with at least one textual artifact of the plurality of textual artifacts;

identifying, using coordinates of matching base points in the current image and the previous image, parameters of a coordinate transformation converting coordinates of the previous image into coordinates of the current image;

associating, using the coordinate transformation, at least part of the OCR text with a cluster of a plurality of clusters of symbol sequences, wherein the OCR text is produced by processing the current image and wherein the symbol sequences are produced by processing one or more previously received images of the series of images;

identifying an order of clusters of symbol sequences, the order reflecting a layout of the original document; and

producing, in view of the order of clusters, a resulting OCR text representing at least a portion of the original document.

2. The method of claim 1, wherein the current image and the previous image represent consecutive images of the series of images of the original document.

3. The method of claim 1, wherein the current image and the previous image differ in at least one of:

image scale, a shooting angle, image brightness, or presence of an external object that is covering at least part of the original document.

4. The method of claim 1, wherein identifying a base point further comprises determining a center of a minimum bounding rectangle of an associated textual artifact.

5. The method of claim 1, further comprising filtering the identified base points using invariant geometric features of base point groupings.

6. The method of claim 1, wherein the coordinate transformation is provided by a projective transformation.

7. The method of claim 1, further comprising:

identifying, for each cluster of symbol sequences, a median string representing the cluster.

8. The method of claim 7, wherein the median string has a minimal sum of values of a pre-defined metric with respect to the cluster of symbol sequences.

9. The method of claim 8, wherein the pre-defined metric represents an edit distance between the median string and a symbol sequence of the plurality of symbol sequences.

10. The method of claim 1, wherein identifying the median string comprises applying weight coefficients to each symbol sequence of the cluster of symbol sequences.

11. The method of claim 1, wherein identifying an order of clusters of symbol sequences comprises identifying a median of permutations of the clusters.

12. The method of claim 11, wherein the median of the permutations of the clusters has a minimal sum of Kendall tau distances to all other permutations.

13. The method of claim 1, wherein identifying the order of clusters further comprises:

producing a graph comprising a plurality of nodes, wherein each node represents a symbol sequence, the graph further comprising a plurality of edges, wherein an edge connects a first symbol sequence produced by OCR of at least a part of a first image of the series of images and a second symbol sequence produced by OCR of a corresponding part of a second image of the series of images.

14. The method of claim 1, wherein the OCR text is provided in a first natural language, the method further comprising:

translating the resulting OCR text to a second natural language.

15. A system, comprising:

a memory;

a processing device, coupled to the memory, the processing device configured to;

receive a current image of a series of images of an original document, wherein the current image at least partially overlaps with a previous image of the series of images;

perform optical character recognition (OCR) of the current image to produce an OCR text and a corresponding text layout;

identify, using the OCR text and the corresponding text layout, a plurality of textual artifacts in each of the current image and the previous image, wherein each textual artifact is represented by a sequence of symbols that has a frequency of occurrence within the OCR text falling below a threshold frequency;

identify, in each of the current image and the previous image, a corresponding plurality of base points, wherein each base point is associated with at least one textual artifact of the plurality of textual artifacts;

identify, using coordinates of matching base points in the current image and the previous image, parameters of a coordinate transformation converting coordinates of the previous image into coordinates of the current image;

associate, using the coordinate transformation, at least part of the OCR text with a cluster of a plurality of clusters of symbol sequences, wherein the OCR text is produced by processing the current image and wherein the symbol sequences are produced by processing one or more previously received images of the series of images;

identify an order of clusters of symbol sequences, the order reflecting a layout of the original document; and

produce, in view of the order of clusters, a resulting OCR text representing at least a portion of the original document.

16. The system of claim 15, wherein identifying a base point further comprises determining a center of a minimum bounding rectangle of an associated textual artifact.

17. The system of claim 15, wherein the median string has a minimal sum of values of a pre-defined metric with respect to the cluster of symbol sequences.

18. The system of claim 15, wherein identifying the order of clusters further comprises:

producing a graph comprising a plurality of nodes, wherein each node represents a symbol sequence, the graph further comprising a plurality of edges, wherein an edge connects a first symbol sequence produced by OCR of at least a part of a first image of the series of images and a second symbol sequence produced by OCR of a corresponding part of a second image of the series of images.

19. A computer-readable non-transitory storage medium comprising executable instructions that, when executed by a processing device, cause the processing device to:

receive a current image of a series of images of an original document, wherein the current image at least partially overlaps with a previous image of the series of images;

perform optical character recognition (OCR) of the current image to produce an OCR text and a corresponding text layout;

identify, using the OCR text and the corresponding text layout, a plurality of textual artifacts in each of the current image and the previous image, wherein each textual artifact is represented by a sequence of symbols that has a frequency of occurrence within the OCR text falling below a threshold frequency;

identify, in each of the current image and the previous image, a corresponding plurality of base points, wherein each base point is associated with at least one textual artifact of the plurality of textual artifacts;

identify, using coordinates of matching base points in the current image and the previous image, parameters of a coordinate transformation converting coordinates of the previous image into coordinates of the current image;

associate, using the coordinate transformation, at least part of the OCR text with a cluster of a plurality of clusters of symbol sequences, wherein the OCR text is produced by processing the current image and wherein the symbol sequences are produced by processing one or more previously received images of the series of images;

identify an order of clusters of symbol sequences, the order reflecting a layout of the original document; and

produce, in view of the order of clusters, a resulting OCR text representing at least a portion of the original document.

20. The computer-readable non-transitory storage medium of claim 19, wherein identifying a base point further comprises determining a center of a minimum bounding rectangle of an associated textual artifact.

21. The computer-readable non-transitory storage medium of claim 19, wherein the median string has a minimal sum of values of a pre-defined metric with respect to the cluster of symbol sequences.

22. The computer-readable non-transitory storage medium of claim 19, wherein identifying the order of clusters:

producing a graph comprising a plurality of nodes, wherein each node represents a symbol sequence, the graph further comprising a plurality of edges, wherein an edge connects a first symbol sequence produced by OCR of at least a part of a first image of the series of images and a second symbol sequence produced by OCR of a corresponding part of a second image of the series of images.