Rectification of characters and text as transform invariant low-rank textures

• US 8,774,558 B2
• Filed: 12/03/2011
• Issued: 07/08/2014
• Est. Priority Date: 11/29/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Xin, Zhang, Zhengdong, Liang, Xiao, Lin, Zhouchen, Ma, Yi
• Primary Examiner(s): Mehta, Bhavesh
• Assistant Examiner(s): MOYER, ANDREW M
• Application Number:
  US13/310,730
• Publication Number:
  US 20120134588A1
• Time in Patent Office: 948 Days
• US Class Current: 382/290
• CPC Class Codes: G06F 18/2136 based on sparsity criteria, e.g. with an overcomplete basis
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/1475 Inclination or skew detection or correction of characters or of image to be recognised

Abstract

A “Text Rectifier” provides various techniques for processing selected regions of an image containing text or characters by treating those images as matrices of low-rank textures and using a rank minimization technique that recovers and removes image deformations (e.g., affine and projective transforms as well as general classes of nonlinear transforms) while rectifying the text or characters in the image region. Once distortions have been removed and the text or characters rectified, the resulting text is made available for a variety of uses or further processing such as optical character recognition (OCR). In various embodiments, binarization and/or inversion techniques are applied to the selected image regions during the rank minimization process to both improve text rectification and to present the resulting images of text to an OCR engine in a form that enhances the accuracy of the OCR results.

Claims

1. A method for rectifying text in an image, comprising using a computing device to perform steps for:

receiving a single input image;

selecting a single region of the input image containing deformed text;

iteratively processing the selected region using an iterative convex optimization process that models the selected region as a low-rank matrix and repeatedly estimates and updates transforms that would cause the deformation of the text in the modeled region until a rank of the matrix is minimized; and

recovering a low-rank texture from the selected image region using the estimated transforms associated with the minimized rank to construct a low-rank texture version of the selected region from which the deformations have been removed from the text and wherein the text has been rectified to an upright position.

2. The method of claim 1 further comprising performing a binarization process on the selected image region prior to each iteration of the convex optimization process such that the selected region is modified to become white text on a black background.

3. The method of claim 2 wherein the binarization process further comprises:

detecting a foreground portion of the selected region corresponding to the deformed text;

setting the foreground to a white color; and

setting a corresponding background portion of the selected image region to a black color.

4. The method claim 1 further comprising providing the low-rank texture version of the selected region to an OCR engine for recognizing the text in the low-rank texture version of the selected region.

5. The method of claim 1 wherein the deformation of the text includes deformation resulting from the text being on a curved surface, and wherein the iterative convex optimization process is adapted to model 3D geometry and curvature of surfaces in the selected image region.

6. The method of claim 1 wherein the deformations include one or more of affine transforms, projective transforms, rotations, translations, and curvatures.

7. The method of claim 1 wherein the iterative convex optimization process further uses an estimated sparse error component to estimate and update the transforms.

8. A system for modeling an image containing text as a low-rank texture in order to rectify the text, comprising:

a general purpose computing device; and

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to;

receive a single input image;

select a single region of the input image containing deformed text;

perform binarization of the selected image region that makes the selected image region white text on a black background prior to performing each iteration of a convex optimization process;

iteratively process the selected region using the iterative convex optimization process that models the selected region as a low-rank matrix and repeatedly estimates and updates transforms that would cause the deformation of the text in the modeled region until a rank of the matrix is minimized; and

recover a low-rank texture from the selected image region using the estimated transforms associated with the minimized rank to construct a low-rank texture version of the selected region from which the deformations have been removed from the text and wherein the text has been rectified to an upright position.

9. The system of claim 8 wherein the binarization of the selected image region further comprises:

detecting a foreground portion of the selected region corresponding to the deformed text;

setting the foreground to a white color; and

setting a corresponding background portion of the selected image region to a black color.

10. The system of claim 8 further comprising providing the low-rank texture version of the selected region to an OCR engine for recognizing the text in the low-rank texture version of the selected region.

11. The system of claim 8 wherein the deformation of the text includes deformation resulting from the text being on a curved surface, and wherein the iterative convex optimization process is adapted to model 3D geometry and curvature of surfaces in the selected image region.

12. The system of claim 8 wherein the deformations include one or more of affine transforms, projective transforms, rotations, translations, and curvatures.

13. The system of claim 8 wherein the iterative convex optimization process further uses an estimated sparse error component to estimate and update the transforms.

14. A computer-readable storage device having computer executable instructions stored therein for extracting low-rank textural and geometric information and a sparse error component from an image, said instructions comprising:

receiving a single input image;

selecting a single region of the input image containing deformed text;

converting the selected image region to a binary image prior to performing each iteration of a convex optimization process;

iteratively processing the selected region using the iterative convex optimization process that models the selected region as a low-rank matrix and repeatedly estimates and updates transforms that would cause the deformation of the text in the modeled region until a rank of the matrix is minimized; and

recovering a low-rank texture from the selected image region using the estimated transforms associated with the minimized rank to construct a low-rank texture version of the selected region from which the deformations have been removed from the text and wherein the text has been rectified to an upright position.

15. The computer-readable storage device of claim 14 wherein the binary image provides white text on a black background.

16. The computer-readable storage device of claim 15 wherein converting the selected image region to a binary image further comprises:

detecting a foreground portion of the selected region corresponding to the deformed text;

setting the foreground to a white color; and

setting a corresponding background portion of the selected image region to a black color.

17. The computer-readable storage device of claim 14 further comprising providing the low-rank texture version of the selected region to an OCR engine for recognizing the text in the low-rank texture version of the selected region.

18. The computer-readable storage device of claim 14 wherein the deformation of the text includes deformation resulting from the text being on a curved surface, and wherein the iterative convex optimization process is adapted to model 3D geometry and curvature of surfaces in the selected image region.

19. The computer-readable storage device of claim 14 wherein the deformations include one or more of affine transforms, projective transforms, rotations, translations, and curvatures.

20. The computer-readable storage device of claim 14 wherein the iterative convex optimization process further uses an estimated sparse error component to estimate and update the transforms.