Method and system for preparing text images for optical-character recognition

• US 10,430,948 B2
• Filed: 08/16/2016
• Issued: 10/01/2019
• Est. Priority Date: 07/15/2016
• Status: Expired due to Fees

• Current Assignee: ABBYY Development LLC
• Original Assignee: ABBYY Production LLC (ABBYY Software)
• Inventors: Kacher, Olga Arnoldovna, Zagaynov, Ivan Germanovich, Rybkin, Vladimir
• Primary Examiner(s): Desire, Gregory M
• Application Number:
  US15/238,350
• Publication Number:
  US 20180018774A1
• Time in Patent Office: 1,141 Days
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20116 Active contour; Active surface; Snakes
• CPC Class Codes: G06T 3/067 Reshaping or unfolding 3D tree structures onto 2D planes
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 20/62 Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/18105 related to colour
• CPC Class Codes: G06V 30/224 of printed characters having additional code marks or containing code marks

Abstract

The current document is directed to methods and systems that straighten in the text lines of text-containing digital images. Initial processing of a text-containing image identifies the outline of a text-containing page. Next, aggregations of symbols, including words and word fragments, are identified within the outlined page image. The centroids and inclination angles of the symbol aggregations are determined, allowing each symbol aggregation to be circumscribed by a closest-fitting rectangle oriented in conformance with the inclination angle determined for the circumscribed symbol aggregation. A model is constructed for the text-line curvature within the text image based on the circumscribed symbol aggregations and is refined using additional information extracted from the text image. The model, essentially an inclination-angle map, allows for assigning local displacements to pixels within the page image which are then used to straighten the text lines in the text image.

Claims

1. An image-processing subsystem included within a device, appliance, or system that acquires an image containing text with curved text lines and inputs the acquired, text-containing image to the image-processing subsystem to generate a corresponding corrected image in which the text lines are straightened and have a rectilinear organization, the image-processing subsystem comprising:

a separate hardware layer or a hardware layer shared with the device, appliance, or system that includes one or more processors, one or more memories; and

computer instructions stored in one or more physical data-storage devices that, when executed by one or more of the one or more processors, control the image-processing subsystem to receive the input text-containing image,

identify a page sub-image within the text-containing image,

correct a vertical perspective of the page sub-image,

generate a text-line-curvature model for the page sub-image that associates inclination angles with pixels in the page sub-image,

generate local displacements, using the text-line-curvature model, for pixels in the page sub-image from the text-line-curvature model, and

transfer pixels from the page sub-image to a corrected page-sub-image using pixel displacements generated from the local displacements to construct a corrected page sub-image in which the text lines are straightened and in which the text characters and symbols have a rectilinear arrangement.

2. The image-processing subsystem of claim 1 wherein the image-processing subsystem identifies a page sub-image within the text-containing image by:

identifying intensity edges within the text-containing image; and

using the identified intensity edges along with known geometrical constraints associated with text-containing pages to identify the sub-images corresponding to one or more text-containing pages in the text-containing image.

3. The image-processing subsystem of claim 1 wherein the image-processing subsystem corrects the vertical perspective of the page sub-image by:

rotating the page sub-image so that the spine edge of the page adjacent to the spine of the book has a vertical orientation;

when a non-spine-adjacent edge is contained in the page sub-image,

adjusting the image so that the non-spine-adjacent edge also has a vertical orientation; and

when the non-spine-adjacent edge is not contained in the page sub-image, correcting the vertical perspective of the page sub-image by another method.

4. The image-processing subsystem of claim 1 wherein the image-processing subsystem generates a text-line-curvature model for the page sub-image that associates inclination angles with pixels in the page sub-image by:

blocking fragments within the page sub-image, each fragment associated with an inclination angle;

using the inclination angles associated with the fragments to determine coefficients for a polynomial representation of an initial text-line-curvature model; and

refining the initial text-line-curvature model to produce a final text-line-curvature model.

5. The image-processing subsystem of claim 4 wherein the image-processing subsystem blocks fragments within the page sub-image by:

identifying fragments in the page sub-image that comprise aggregations of characters and/or symbols that correspond to words, word fragments, and/or fragments of multiple words; and

for each fragment,

identifying the centroid of the fragment, and

employing a minimization method that minimizes the distance between pixels of each fragment and a line through the centroid to determine and associate an inclination angle with the fragment that is an angle between a principle axis of the fragment and a horizontal direction.

6. The image-processing subsystem of claim 5 wherein the image-processing subsystem identifies fragments in the page sub-image by:

selecting seed character pixels within the page image;

alternately

expanding each seed character pixel into a blob of contiguous character pixels, and

searching for new seed character pixels within a maximum distance of the blob of contiguous character pixels.

7. The image-processing subsystem of claim 5 wherein, when the text image includes darker characters on a brighter background, a character pixel has an intensity lower than a threshold value;

and

wherein, when the text image includes lighter characters on a darker background, a character pixel has an intensity greater than a threshold value.

8. The image-processing subsystem of claim 4 wherein, after blocking fragments within the page sub-image, the image-processing subsystem filters the blocked fragments by inclination angle and dimension to select a best set of blocked fragments for determining coefficients for the polynomial representation of the initial text-line-curvature model.

9. The image-processing subsystem of claim 4 wherein the image-processing subsystem uses the inclination angles associated with the fragments to determine coefficients for the polynomial representation of the initial text-line-curvature model by:

initializing the values of a set of coefficients for the initial polynomial representation of the text-line-curvature model; and

using a data-fitting method to determine the coefficients for the initial polynomial representation of the text-line-curvature model from the inclination angles associated with the fragments.

10. The image-processing subsystem of claim 9 wherein data-fitting method is selected from among:

Gaussian elimination applied to a system of linear equations generated from derivatives, with respect to model coefficients, of expressions for squared inclination angles;

the Levenberg-Marquart method;

the Gauss-Newton method;

the Nonlinear Conjugate Gradient Method; and

additional linear and non-linear regression-based curve-fitting methods.

11. The image-processing subsystem of claim 9 wherein the image-processing subsystem:

filters the blocked fragments for correspondence of the inclination angles associated with each blocked fragment with the inclination angles determined for the pixels of the blocked fragment using the initial polynomial representation of the text-line-curvature model; and

when filtering the blocked fragments removes more than a threshold number of blocked fragments,

regenerates coefficients for the initial polynomial representation of the text-line-curvature model by application of a data-fitting method.

12. The image-processing subsystem of claim 4 wherein the image-processing subsystem refines the initial text-line-curvature model to produce a final text-line-curvature model by:

applying closing, opening, and vertical smoothing operations to the page sub-image to generate filled contours corresponding to the text lines in the page sub-image;

constructing curved text-line segments within the filled contours about the text lines using inclination angles generated by the polynomial representation of the text-line-curvature model; and

using points and associated angles of inclination along the constructed curved text-line segments as data for application of a data-fitting method to generate refined coefficients for the polynomial representation of the text-line-curvature model.

13. The image-processing subsystem of claim 12 wherein the image-processing subsystem constructs curved text-line segments within the filled contours about the text lines using inclination angles generated by the polynomial representation of the text-line-curvature model by:

for each blocked fragment in the page sub-image,

selecting, as a starting point, a pixel corresponding to the centroid of the blocked fragment within the filled contour corresponding to the text line; and

constructing a curved text-line segment in two directions from the starting point by iteratively adding segment portions to the current ends of the curved text-line segment until no further segment portions can be added.

14. The image-processing subsystem of claim 13 wherein the image-processing subsystem attempts to construct, from each starting point, a first segment portion in the direction of an inclination angle obtained from the polynomial representation of the text-line-curvature model for the starting point and a second segment portion in a direction equal to the direction of the inclination angle plus 180°

, the starting point representing two current curved text-line-segment endpoints.

15. The image-processing subsystem of claim 13 wherein the image-processing subsystem adds a segment portion to a current curved text-line-segment endpoint by:

determining an inclination angle using the polynomial representation of the text-line-curvature model for a pixel corresponding to the current curved text-line-segment endpoint;

constructing an initial vector from current curved text-line-segment endpoint a direction equal to the determined inclination angle;

considering a vertical neighborhood of pixels coincident with a head of the initial vector;

when a pixel in the vertical neighborhood has an intensity value more representative of an intensity expected for text pixels than a subset of the remaining pixels in the vertical neighborhood,

reconfiguring the initial vector, when necessary, to produce a final vector having a head coincident with the pixel having an intensity value more representative of the expected intensity, and

extending the curved text-line-segment to the pixel coincident with the head of the final vector; and

when no pixel in the vertical neighborhood has an intensity value more representative of an intensity expected for text pixels than a subset of the remaining pixels in the vertical neighborhood, terminating the curved text-line-segment at the current curved text-line-segment endpoint.

16. The image-processing subsystem of claim 1 wherein the text-line-curvature model returns a tangent of an inclination angle when called with arguments representing the location of a pixel within the page sub-image.

17. The image-processing subsystem of claim 1 wherein the image-processing subsystem generates local displacements, using the text-line-curvature model, for pixels in the page sub-image from the text-line-curvature model by:

generating an inclination-angle map that includes an inclination angle for each pixel of the page sub-image produced by input of a position of the pixel within the page sub-image to the final text-line-curvature model;

modifying the inclination angles in a vertical region of the angle map by multiplication by a coefficient that adjusts the inclination angles for increased curvature and compression in corresponding vertical region of the page sub-image; and

generating an offset map that includes a local displacement for each offset-map cell in the page sub-image.

18. The image-processing subsystem of claim 17 wherein the image-processing subsystem generates a local displacement for a cell in the offset map by:

determining a value for a vertical displacement as a compression coefficient multiplied by a cell width times the tangent of the inclination angle obtained from the final text-line-curvature model;

setting a value for a horizontal displacement to the compression coefficient; and

associating the vertical displacement and the horizontal displacement with the cell.

19. The image-processing subsystem of claim 18 wherein the compression coefficient for a cell is computed as a fraction equal to a cell width divided by a pixel width times one of the cosine of the difference between inclination angles for cells at the top and bottom of a vertical cell column spanning the page image and coincident with the cell and 0.4.

20. The image-processing subsystem of claim 1 wherein the image-processing subsystem transfers pixels from the page sub-image to a corrected page sub-image using pixel displacements generated from the local displacements by:

identifying pairs of corresponding anchor points in the page sub-image and the corrected page sub image;

transferring the intensity value of pixels corresponding to the anchor points in the page sub-image to pixels corresponding to pixels corresponding to the anchor points in the corrected page sub-image; and

for each non-anchor-point pixel in the page sub-image,

summing the local displacements coincident with a vector from the non-anchor-point pixel to a nearby anchor point to produce a pixel displacement, and

transferring the intensity value of the non-anchor-point pixel to a pixel displaced from a corresponding anchor point in the corrected page sub-image by the pixel displacement.

21. A method carried out by an image-processing subsystem included within a device, appliance, or system that acquires an image containing text with curved text lines and inputs the acquired, text-containing image to the image-processing subsystem to generate a corresponding corrected image in which the text lines are straightened and have a rectilinear organization, the method comprising:

receiving the input text-containing image and storing the input text-containing image in a memory;

identifying a page sub-image within the text-containing image;

correcting a vertical perspective of the page sub-image;

generating a text-line-curvature model for the page sub-image that associates inclination angles with pixels in the page sub-image;

generating local displacements, using the text-line-curvature model, for pixels in the page sub-image from the text-line-curvature model; and

transferring pixels from the page sub-image to a corrected page-sub-image using pixel displacements generated from the local displacements to construct, in a memory, the corrected page sub-image in which the text lines are straightened and in which the text characters and symbols have a rectilinear arrangement.

22. A non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processors of a hardware layer of an image-processing subsystem that includes one or more processors and one or more memories and that is included within a device, appliance, or system that acquires an image containing text with curved text lines to receive the acquired, text-containing image from the device, appliance, or system and generates a corresponding corrected image in which the text lines are straightened and have a rectilinear organization, causes the one or more processors to perform operations comprising:

receiving the input text-containing image and storing the input text-containing image in a memory;

identifying a page sub-image within the text-containing image;

correcting a vertical perspective of the page sub-image;

generating a text-line-curvature model for the page sub-image that associates inclination angles with pixels in the page sub-image;

generating local displacements, using the text-line-curvature model, for pixels in the page sub-image from the text-line-curvature model; and

transferring pixels from the page sub-image to a corrected page-sub-image using pixel displacements generated from the local displacements to construct, in a memory, the corrected page sub-image in which the text lines are straightened and in which the text characters and symbols have a rectilinear arrangement.