GENERATING HI-RES DEWARPED BOOK IMAGES

• US 20130242054A1
• Filed: 11/30/2012
• Published: 09/19/2013
• Est. Priority Date: 03/15/2012
• Status: Active Grant

• Current Assignee: Fujifilm Business Innovation Corp. (Fujifilm Holdings Corporation)
• Original Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Inventors: CHIU, Patrick, CUTTER, Michael Patrick, KIM, Chelhwon, CHANDRA, Surendar
• Granted Patent: US 9,992,471 B2
• Time in Patent Office: Days
• US Class Current: 348/46
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/30176 Document
• CPC Class Codes: G06T 2210/44 Morphing
• CPC Class Codes: G06T 2219/021 Flattening
• CPC Class Codes: G06T 2219/2021 Shape modification
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06V 20/62 Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: H04N 13/10 Processing, recording or transmission of stereoscopic or multi-view image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence

Abstract

Systems and methods for generating high resolution dewarped images for an image of a document captured by a 3D stereo digital camera system, or a mobile phone camera capturing a sequence of images, which may improve OCR performance. Example embodiments include a compact stereo camera with two sensors mounted at fixed locations, and a multi-resolution pipeline to process and to dewarp the images using a three dimensional surface model based on curve profiles of the computed depth map. Example embodiments also include a mobile phone including a camera which captures a sequence of images, and a processor which computes a disparity map using the captured sequence of image frames, computes a model of the at least one document page by generating a cylindrical three dimensional geometric surface using the computed disparity map, and renders a dewarped image from the computed model.

Claims

1. A method, comprising:

receiving a stereo photo of at least one document page;

computing a model of the at least one document page by generating a cylindrical three dimensional geometric surface from the stereo photo; and

rendering a dewarped image from the computed model.

2. The method of claim 1, wherein the rendering the dewarped image from the computed model, further comprises:

scaling a first image associated with the stereo photo to produce a second image having a lower resolution than the first image;

dewarping the second image; and

rendering a dewarped representation of the first image based on the dewarping of the second image, the dewarped representation of the first image having a resolution higher than the second image.

3. The method of claim 2, wherein the dewarping of the second image comprises:

projecting a first mesh of the second image to a second mesh of the model; and

flattening the second mesh in the virtual space of the model.

4. The method of claim 3, wherein the rendering the dewarped representation of the first image comprises:

scaling a first rectified mesh of the first mesh to produce a second rectified mesh based on a resolution of the first image; and

applying the second mesh and the second rectified mesh on the first image.

5. The method of claim 4, wherein rendering the dewarped representation of the first image from the first image further comprises:

conducting a first correlation of vertices of a third mesh of the first image to vertices of the second rectified mesh;

conducting a second correlation of vertices of the second rectified mesh to the second mesh;

projecting the second mesh to the first image to render the dewarped representation based on the first correlation and the second correlation.

6. The method of claim 2, further comprising conducting stereo rectification on the second image prior to the dewarping the second image.

7. The method of claim 1, wherein the cylindrical three dimensional geometric surface is constructed using profile curves computed from a depth map.

8. A computer readable storage medium storing instructions for executing a process, the process comprising:

receiving a stereo photo of at least one document page;

computing a model of the at least one document page by generating a cylindrical three dimensional geometric surface from the stereo photo; and

rendering a dewarped image from the computed model.

9. The computer readable storage medium of claim 8, wherein the rendering the dewarped image from the computed model, further comprises:

scaling a first image associated with the stereo photo to produce a second image having a lower resolution than the first image;

dewarping the second image; and

rendering a dewarped representation of the first image based on the dewarping of the second image, the dewarped representation of the first image having a resolution higher than the second image.

10. The computer readable storage medium of claim 9, wherein the dewarping of the second image comprises:

projecting a first mesh of the second image to a second mesh of the model; and

flattening the second mesh in the virtual space of the model.

11. The computer readable storage medium of claim 10, wherein the rendering the dewarped representation of the first image comprises:

scaling a first rectified mesh of the first mesh to produce a second rectified mesh based on a resolution of the first image; and

applying the second mesh and the second rectified mesh on the first image.

12. The computer readable storage medium of claim 11, wherein rendering the dewarped representation of the first image from the first image further comprises:

conducting a first correlation of vertices of a third mesh of the first image to vertices of the second rectified mesh;

conducting a second correlation of vertices of the second rectified mesh to the second mesh;

projecting the second mesh to the first image to render the dewarped representation based on the first correlation and the second correlation.

13. The computer readable storage medium of claim 9, wherein the process further comprises conducting stereo rectification on the second image prior to the dewarping the second image.

14. The computer readable storage medium of claim 8, wherein the cylindrical three dimensional geometric surface is constructed using profile curves computed from a depth map.

15. A device, comprising:

a camera configured to receive a stereo photo of at least one document page; and

a processor configured to compute a model of the at least one document page by generating a cylindrical three dimensional geometric surface from the stereo photo; and

render a dewarped image from the computed model.

16. The device of claim 15, wherein the processor is configured to render the dewarped image from the computed model by:

scaling a first image associated with the stereo photo to produce a second image having a lower resolution than the first image;

dewarping the second image; and

rendering a dewarped representation of the first image based on the dewarping of the second image, the dewarped representation of the first image having a resolution higher than the second image.

17. The device of claim 16, wherein the processor is configured to dewarp of the second image by:

projecting a first mesh of the second image to a second mesh of the model; and

flattening the second mesh in the virtual space of the model.

18. The device of claim 17, wherein the processor is configured to render the dewarped representation of the first image by:

scaling a first rectified mesh of the first mesh to produce a second rectified mesh based on a resolution of the first image; and

applying the second mesh and the second rectified mesh on the first image.

19. The device of claim 18, wherein the processor is configured to render the dewarped representation of the first image from the first image by:

conducting a first correlation of vertices of a third mesh of the first image to vertices of the second rectified mesh;

conducting a second correlation of vertices of the second rectified mesh to the second mesh;

projecting the second mesh to the first image to render the dewarped representation based on the first correlation and the second correlation.

20. The device of claim 16, wherein the processor is further configured to conduct stereo rectification on the second image prior to the dewarping the second image.

21. A mobile device, comprising:

a camera configured to capture a sequence of image frames of at least one document page; and

a processor configured to compute a disparity map using the captured sequence of image frames;

compute a model of the at least one document page by generating a cylindrical three dimensional geometric surface using the computed disparity map; and

render a dewarped image from the computed model.

22. The device of claim 21, wherein the disparity map is computed with an optical flow technique.

23. The device of claim 21, wherein the disparity map is computed with a structure from motion technique.

24. The device of claim 21, wherein the cylindrical three dimensional geometric surface is constructed using profile curves computed from the disparity map.

25. The device of claim 21, wherein the processor is configured to compute the disparity map using a sequence of image frames representing a single linear path of the mobile device over the at least one document page.

27. The device of claim 21, wherein the processor is configured to render multiple dewarped images;

perform optical character recognition on the multiple dewarped images and merge the results thereof.

26. A method comprising:

capturing a sequence of image frames of at least one document page;

computing a disparity map using the captured sequence of image frames;

computing a model of the at least one document page by generating a cylindrical three dimensional geometric surface using the computed disparity map; and

rendering a dewarped image from the computed model.

28. The method of claim 26, further comprising:

rendering multiple dewarped images; and

performing optical character recognition on the multiple dewarped images and merging the results thereof.

29. A computer readable storage medium storing instructions for executing a process, the process comprising:

computing a disparity map using a sequence of image frames of at least one document page;

computing a model of the at least one document page by generating a cylindrical three dimensional geometric surface using the computed disparity map; and

rendering a dewarped image from the computed model.