System and method for scanned document correction

• US 10,289,924 B2
• Filed: 11/22/2016
• Issued: 05/14/2019
• Est. Priority Date: 10/17/2011
• Status: Active Grant

• Current Assignee: Sharp Kabushiki Kaisha (Hon Hai Precision Industry Co., Ltd.)
• Original Assignee: Sharp Laboratories of America Incorporated (Hon Hai Precision Industry Co., Ltd.)
• Inventors: Campbell, Richard John
• Primary Examiner(s): Moyer, Andrew M
• Assistant Examiner(s): Rosario, Dennis
• Application Number:
  US15/359,314
• Publication Number:
  US 20170076169A1
• Time in Patent Office: 903 Days
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 30/153 using recognition of characters or words
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text

Abstract

A system and method are provided for the correction of a warped page image. The method first accepts a camera image of a page, creates a filtered edge map, and identifies text-likely regions. The filtered edge map and text-likely regions are projected into a polar coordinate system to determine page lines and warped image page curves. An adaptive two-dimensional (2D) ruled mesh piecewise planar approximation of a warped page surface is created. A three-dimensional (3D) model is created using the adaptive 2D ruled mesh and the estimate of the camera focal length estimate. Using the 3D model, a 2D target mesh is created for rectifying the image of the page. In one aspect, the adaptive 2D ruled mesh is projected onto a 3D warped page surface using the estimated camera focal length and an estimated surface normal of each planar strip from the adaptive 2D ruled mesh.

Claims

1. A method for the correction of a warped page image, the method comprising:

accepting a camera image of a page;

creating a filtered edge map and identifying text-likely regions;

projecting the filtered edge map and text-likely regions into a polar coordinate system to determine page lines and warped image page curves;

subsequent to projecting the filtered edge map and text-likely regions into the polar coordinate system, histogramming the filtered and projected edge map into theta bins, computing minimum and maximum theta angles for text-likely regions, localizing a first page line as a closest significant edge bin less than minimum text-likely region theta angle and a second page line as a closest significant edge bin greater than maximum text-likely region theta angle;

creating an adaptive two-dimensional (2D) ruled mesh piecewise planar approximation of a warped page surface;

using the created adaptive two-dimensional (2D) ruled mesh piecewise planar approximation of a warped page surface for estimating a camera focal length;

creating a three-dimensional (3D) model using the adaptive 2D ruled mesh and camera focal length estimate; and

,

using the 3D model, creating a 2D target mesh rectifying the image of the page;

wherein rectifying the image of the page includes using a perspective dewarp to interpolate image values from the adaptive 2D ruled mesh into the image defined by the 2D target rectilinear mesh.

2. The method of claim 1 further comprising:

subsequent to projecting the filtered edge map and text-likely regions into the polar coordinate system, performing a multi-scale edge localization refinement for each determined page curve.

3. The method of claim 2 wherein performing the multi-scale refinement of the each determined page curves includes:

transforming top and bottom page curves into point sets having a first resolution;

scaling the point sets to a second resolution, lower than the first resolution;

creating a gradient image of the second resolution point sets;

identifying zero crossings points in the gradient image as a page edge; and

,

scaling the zero crossing points to the first resolution.

4. The method of claim 1 further comprising:

subsequent to projecting the filtered edge map and text-likely regions into the polar coordinate system, independently fitting a pair of warped image page curve sections to a corresponding pair of Bezier curves.

5. The method of claim 1 wherein creating the adaptive 2D ruled mesh includes creating a plurality of planar strips;

and,

estimating a camera focal length includes independently estimating a camera focal length for each planar strip.

6. The method of claim 5 wherein creating the adaptive 2D ruled mesh includes using a polygonal approximation of Bezier curves and polygonal simplification, with a fit tolerance (fitTH) of 0.5 pixels, to assist in creating the adaptive 2D ruled mesh.

7. The method of claim 5 wherein creating the 3D model includes projecting the adaptive 2D ruled mesh onto a 3D warped page surface using the estimated camera focal length and an estimated surface normal of each planar strip from the adaptive 2D ruled mesh.

8. The method of claim 1 wherein estimating the camera focal length includes using the adaptive 2D ruled mesh of the warped page surface.

9. The method of claim 1 wherein creating the 2D target mesh includes determining a resolution of a 2D target mesh in response to scaling 3D mesh width and height values, where the scaling is determined by measuring an overall 3D page width and height and an input region width and height.

10. The method of claim 1, further comprising:

identifying page gutters for images of adjacent pages, where each gutter is determined as the closest significant theta edge bin to a theta angle halfway between the minimum text-likely region theta angle and the maximum text-likely region angle.

11. The method of claim 1, further comprising determining a contour region of the page by:

filtering curves not between page boundary lines, and dividing remaining curves into a top set defined by curves that are within the region defined by boundary lines L1, L2, and LC1, and a bottom set defined by curves that are within the region defined by the boundary lines L1, L2, and LC2, where L1 is a left boundary, L2 is a right boundary, and LC2 is a bottom boundary;

selecting a top page curve as a top set curve with highest completeness and lowest curvature between the boundary lines L1 and L2; and

,

selecting a bottom page curve as bottom set curve with the highest completeness and lowest curvature between the boundary lines L1 and L2.

12. The method in claim 11 further comprising determining contour regions on two adjacent pages of a book, by:

processing an image of adjacent pages, where each page is divided into upper and lower curve sets using boundary lines and gutter, and upper and lower curve lines are favored that join at the gutter.

13. A system for the correction of a warped page image, the system comprising:

a non-transitory memory;

a processor;

an interface to accept a camera image of a page, and to supply a rectified image of the page;

an image correction application stored in the memory and enabled as a sequence of processor executable steps for;

creating a filtered edge map and identifying text-likely regions;

projecting the filtered edge map and text-likely regions into a polar coordinate system to determine page lines and warped image page curves;

subsequent to projecting the filtered edge map and text-likely regions into the polar coordinate system, histograms the filtered and projected edge map into theta bins, computes minimum and maximum theta angles for text-likely regions, and localizes a first page line as a closest significant edge bin less than minimum text-likely region theta angle and a second page line as a closest significant edge bin greater than maximum text-likely region theta angle;

creating an adaptive two-dimensional (2D) ruled mesh piecewise planar approximation of a warped page surface;

using the created adaptive two-dimensional (2D) ruled mesh piecewise planar approximation of a warped page surface for estimating a camera focal length;

creating a three-dimensional (3D) model using the adaptive 2D ruled mesh and camera focal length estimate; and

,

using the 3D model, creating a 2D target mesh rectifying the image of the page;

wherein the image correction application rectifies the image of the page using a perspective dewarp to interpolate image values from the adaptive 2D ruled mesh into the image defined by the 2D target rectilinear mesh.

14. The system of claim 13 wherein the image correction application, subsequent to projecting the filtered edge map and text-likely regions into the polar coordinate system, performs a multi-scale edge localization refinement for each determined page curve.

15. The system of claim 14 wherein the image correction application performs the multi-scale refinement of the determined page curves by:

transforming top and bottom page curves into point sets having a first resolution;

scaling the point sets to a second resolution, lower than the first resolution;

creating a gradient image of the second resolution point sets;

identifying zero crossings points in the gradient image as a page edge; and

,

scaling the zero crossing points to the first resolution.

16. The system of claim 13 wherein the image correction application, subsequent to projecting the filtered edge map and text-likely regions into the polar coordinate system, independently fits a pair of warped image page curve sections to a corresponding pair of Bezier curves.

17. The system of claim 13 wherein the image correction application creates the adaptive 2D ruled mesh with a plurality of planar strips, and independently estimates a camera focal length for each planar strip.

18. The system of claim 17 wherein the image correction application creates the adaptive 2D ruled mesh using a polygonal approximation of the Bezier curves and polygonal simplification, with a fit tolerance (fitTH) of 0.5 pixels, to assist in creating the adaptive 2D ruled mesh.

19. The system of claim 17 wherein the image correction application creates the 3D model by projecting the adaptive 2D ruled mesh onto a 3D warped page surface using the estimated camera focal length and an estimated surface normal of each planar strip from the adaptive 2D ruled mesh.

20. The system of claim 17 wherein the image correction application further creates the 2D target mesh with the plurality of planar strips by determining a resolution of a 2D target mesh in response to scaling 3D mesh width and height values, where the scaling is determined by measuring an overall 3D page width and height and an input region width and height.

21. The system of claim 13 wherein the image correction application estimates the camera focal length includes using the adaptive 2D ruled mesh of the warped page surface.

22. The system of claim 13 wherein the image correction application identifies page gutters for images of adjacent pages, where each gutter is determined as the closest significant theta edge bin to a theta angle halfway between the minimum text-likely region theta angle and the maximum text-likely region angle.

23. The system of claim 13, further comprising determining a contour region of the page by:

when the image correction application filters curves not between page boundary lines, and divides remaining curves into a top set defined by curves that are within the region defined by boundary lines L1, L2, and LC1, and a bottom set defined by curves that are within the region defined by the boundary lines L1, L2, and LC2, where L1 is a left boundary, L2 is a right boundary, and LC2 is a bottom boundary; and

,

wherein the image correction application selects a top page curve as a top set curve with highest completeness and lowest curvature between the boundary lines L1 and L2, and selects a bottom page curve as bottom set curve with the highest completeness and lowest curvature between the boundary lines L1 and L2.

24. The system of claim 23, further comprising determining contour regions on two adjacent pages of a book, by:

wherein when the image correction application processes an image of adjacent pages, where each page is divided into upper and lower curve sets using boundary lines and gutter, and upper and lower curve lines are favored that join at the gutter.