System and Method for Scanned Document Correction

• US 20170076169A1
• Filed: 11/22/2016
• Published: 03/16/2017
• Est. Priority Date: 10/17/2011
• Status: Active Grant

• Current Assignee: Sharp Kabushiki Kaisha (Hon Hai Precision Industry Co., Ltd.)
• Original Assignee: Sharp Laboratories of America Incorporated (Hon Hai Precision Industry Co., Ltd.)
• Inventors: Campbell, Richard John
• Granted Patent: US 10,289,924 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 30/153 using recognition of characters or words
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text

Abstract

A system and method are provided for the correction of a warped page image. The method first accepts a camera image of a page, creates a filtered edge map, and identifies text-likely regions. The filtered edge map and text-likely regions are projected into a polar coordinate system to determine page lines and warped image page curves. An adaptive two-dimensional (2D) ruled mesh piecewise planar approximation of a warped page surface is created. A three-dimensional (3D) model is created using the adaptive 2D ruled mesh and the estimate of the camera focal length estimate. Using the 3D model, a 2D target mesh is created for rectifying the image of the page. In one aspect, the adaptive 2D ruled mesh is projected onto a 3D warped page surface using the estimated camera focal length and an estimated surface normal of each planar strip from the adaptive 2D ruled mesh.

Claims

1. A method for the correction of a warped page image, the method comprising:

accepting a camera image of a page;

creating a filtered edge map and identifying text-likely regions;

projecting the filtered edge map and text-likely regions into a polar coordinate system to determine page lines and warped image page curves;

creating an adaptive two-dimensional (2D) ruled mesh piecewise planar approximation of a warped page surface;

estimating a camera focal length;

creating a three-dimensional (3D) model using the adaptive 2D ruled mesh and camera focal length estimate; and

,

using the 3D model, creating a 2D target mesh rectifying the image of the page.

2. The method of claim 1 further comprising:

subsequent to projecting the filtered edge map and text-likely regions into the polar coordinate system, performing a multi-scale edge localization refinement for each page curve.

3. The method of claim 2 wherein performing the multi-scale refinement of the page curves includes:

transforming top and bottom page curves into point sets having a first resolution;

scaling the point sets to a second resolution, lower than the first resolution;

creating a gradient image of the second resolution point sets;

identifying zero crossings points in the gradient image as a page edge; and

,

scaling the zero crossing points to the first resolution.

4. The method of claim 1 further comprising:

subsequent to projecting the filtered edge map and text-likely regions into the polar coordinate system, independently fitting a pair of warped image page curve sections to a corresponding pair of Bezier curves.

5. The method of claim 1 wherein creating the adaptive 2D ruled mesh includes creating a plurality of planar strips;

and,

estimating a camera focal length includes independently estimating a camera focal length for each planar strip.

6. The method of claim 5 wherein creating the adaptive 2D ruled mesh includes using a polygonal approximation of the Bezier curves and polygonal simplification, with a fit tolerance (fitTH) of 0.5 pixels, to assist in generating the adaptive 2D ruled mesh.

7. The method of claim 1 wherein estimating the camera focal length includes using the adaptive 2D ruled mesh of the warped page surface to estimate the camera focal length.

8. The method of claim 5 wherein creating the 3D model includes projecting the adaptive 2D ruled mesh onto a 3D warped page surface using the estimated camera focal length and an estimated surface normal of each planar strip from the adaptive 2D ruled mesh.

9. The method of claim 1 wherein creating the 2D target mesh includes determining a resolution of a 2D target mesh in response to scaling 3D mesh width and height values, where the scale factor is determined by measuring an overall 3D page width and height and an input region width and height.

10. The method of claim 1 wherein rectifying the image of the page includes using a perspective dewarp to interpolate image values from the adaptive 2D ruled mesh into the image defined by the 2D target rectilinear mesh.

11. The method of claim 1 further comprising:

subsequent to projecting the filtered edge map and text-likely regions into the polar coordinate system, histogramming an edge map into theta bins, computing minimum and maximum theta angles for text-likely regions, localizing a first page line as the closest significant edge bin less than minimum text-likely region theta angle and a second page line as the closest significant edge bin greater than maximum test-likely region theta angle.

12. The method of claim 11 further comprising:

identifying page gutters for images of adjacent pages, where the gutter is determined as the closest significant theta edge bin to the theta angle halfway between the minimum text-likely region theta angle and the maximum text-likely region angle.

13. The method of claim 11 further comprising:

filtering curves not between page boundary lines, and dividing remaining curves into a top set defined by curves that are within the region defined by boundary lines L1, L2, and LC1, and a bottom set defined by curves that are within the region defined by the boundary lines L1, L2, and LC2, where L1 is a left boundary, L2 is a right boundary, and LC2 is a bottom boundary;

selecting the top page curve as a top set curve with highest completeness and lowest curvature between the boundary lines L1 and L2; and

,

selecting the bottom page curve as bottom set curve with the highest completeness and lowest curvature between the boundary lines L1 and L2.

14. The method in claim 13 further comprising:

processing an image of adjacent pages, where each page is divided into upper and lower curve sets using boundary lines and gutter, and upper and lower curve lines are favored that join at the gutter.

15. A computer-readable medium with processor executable instructions for correcting a warped page image, the instructions comprising the steps of:

accepting a camera image of a page;

creating a filtered edge map and identifying text-likely regions;

projecting the filtered edge map and text-likely regions into a polar coordinate system to determine page lines and warped image page curves;

creating an adaptive two-dimensional (2D) ruled mesh piecewise planar approximation of a warped page surface;

estimating a camera focal length;

creating a three-dimensional (3D) model using the adaptive 2D ruled mesh and camera focal length estimate; and

,

using the 3D model, creating a 2D target mesh rectifying the image of the page.

16. A system for the correction of a warped page image, the system comprising:

a non-transitory memory;

a processor;

an interface to accept a camera image of a page, and to supply a rectified image of the page;

an image correction application stored in the memory and enabled as a sequence of processor executable steps for;

creating a filtered edge map and identifying text-likely regions;

projecting the filtered edge map and text-likely regions into a polar coordinate system to determine page lines and warped image page curves;

creating an adaptive two-dimensional (2D) ruled mesh piecewise planar approximation of a warped page surface;

estimating a camera focal length;

creating a three-dimensional (3D) model using the adaptive 2D ruled mesh and camera focal length estimate; and

,

using the 3D model, creating a 2D target mesh rectifying the image of the page.

17. The system of claim 16 wherein the image correction application, subsequent to projecting the filtered edge map and text-likely regions into the polar coordinate system, performs a multi-scale edge localization refinement for each page curve.

18. The system of claim 17 wherein the image correction application performs the multi-scale refinement of the page curves by:

transforming top and bottom page curves into point sets having a first resolution;

scaling the point sets to a second resolution, lower than the first resolution;

creating a gradient image of the second resolution point sets;

identifying zero crossings points in the gradient image as a page edge; and

,

scaling the zero crossing points to the first resolution.

19. The system of claim 16 wherein the image correction application, subsequent to projecting the filtered edge map and text-likely regions into the polar coordinate system, independently fits a pair of warped image page curve sections to a corresponding pair of Bezier curves.

20. The system of claim 16 wherein the image correction application creates the adaptive 2D ruled mesh with a plurality of planar strips, and independently estimates a camera focal length for each planar strip.

21. The system of claim 20 wherein the image correction application creates the adaptive 2D ruled mesh using a polygonal approximation of the Bezier curves and polygonal simplification, with a fit tolerance (fitTH) of 0.5 pixels, to assist in generating the adaptive 2D ruled mesh.

22. The system of claim 16 wherein the image correction application estimates the camera focal length includes using the adaptive 2D ruled mesh of the warped page surface.

23. The system of claim 20 wherein the image correction application creates the 3D model by projecting the adaptive 2D ruled mesh onto a 3D warped page surface using the estimated camera focal length and an estimated surface normal of each planar strip from the adaptive 2D ruled mesh.

24. The system of claim 20 wherein the image correction application creates the 2D target mesh by determining a resolution of a 2D target mesh in response to scaling 3D mesh width and height values, where the scale factor is determined by measuring an overall 3D page width and height and an input region width and height.

25. The system of claim 16 wherein the image correction application rectifies the image of the page using a perspective dewarp to interpolate image values from the adaptive 2D ruled mesh into the image defined by the 2D target rectilinear mesh.

26. The system of claim 16 wherein the image correction application, subsequent to projecting the filtered edge map and text-likely regions into the polar coordinate system, histograms an edge map into theta bins, computes minimum and maximum theta angles for text-likely regions, and localizes a first page line as the closest significant edge bin less than minimum text-likely region theta angle and a second page line as the closest significant edge bin greater than maximum test-likely region theta angle.

27. The system of claim 26 wherein the image correction application identifies page gutters for images of adjacent pages, where the gutter is determined as the closest significant theta edge bin to the theta angle halfway between the minimum text-likely region theta angle and the maximum text-likely region angle.

28. The system of claim 26 wherein the image correction application filters curves not between page boundary lines, and divides remaining curves into a top set defined by curves that are within the region defined by boundary lines L1, L2, and LC1, and a bottom set defined by curves that are within the region defined by the boundary lines L1, L2, and LC2, where L1 is a left boundary, L2 is a right boundary, and LC2 is a bottom boundary;

and,

wherein the image correction application selects the top page curve as a top set curve with highest completeness and lowest curvature between the boundary lines L1 and L2, and selects the bottom page curve as bottom set curve with the highest completeness and lowest curvature between the boundary lines L1 and L2.

29. The method in claim 28 wherein the image correction application processes an image of adjacent pages, where each page is divided into upper and lower curve sets using boundary lines and gutter, and upper and lower curve lines are favored that join at the gutter.