Method for creating environment map containing information extracted from stereo image pairs

• US 6,507,665 B1
• Filed: 08/25/1999
• Issued: 01/14/2003
• Est. Priority Date: 08/25/1999
• Status: Expired due to Term

• Current Assignee: Eastman Kodak Company
• Original Assignee: Eastman Kodak Company
• Inventors: Chen, Shoupu, Cahill, Nathan D.
• Primary Examiner(s): CHANG, JON CARLTON
• Application Number:
  US09/383,573
• Time in Patent Office: 1,238 Days
• Field of Search: 382/154, 382/284, 382/285, 382/294, 356/12, 359/462, 396/324, 348/42, 348/43, 348/46, 348/47, 348/50, 348/36
• US Class Current: 382/154
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 3/153 using elastic snapping
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0088 Synthesising a monoscopic image signal from stereoscopic images, e.g. synthesising a panoramic or high resolution monoscopic image

Abstract

A technique of synthesizing a panoramic environment map containing intensity and range information of a scene, which includes estimation of orthographic range from a plurality of pairs of stereo images, transformation of orthographic range estimates to perspective or directional range estimates, transformation of planar intensity and range images to cylindrical images, registration of overlapping cylindrical images, and blending of intensity and range information.

Claims

1. A method for generating a panoramic environment map from a plurality of stereo image pairs of a scene, said method comprising the steps of:

(a) acquiring a plurality of stereo image pairs of the scene, wherein there is an overlap region between adjacent stereo image pairs;

(b) generating orthographic range values from each of the stereo image pairs corresponding to the orthogonal distance of image features in the stereo pairs from an image capture point;

(c) transforming the range values into transformed range values referenced to a coordinate system that reflects a common projection of the features from each stereo image pair;

(d) warping the transformed range values onto a cylindrical surface, and forming therefrom a plurality of warped range images;

(e) registering the adjacent warped range images; and

(f) blending the overlap regions of the registered warped range images to generate a panoramic environment map containing range information.

2. A method as claimed in claim 1 wherein the step (a) of acquiring a plurality of stereo image pairs of a scene comprises the steps of:

a) capturing a first panoramic image of the scene from a first point on a vertical axis; and

b) capturing a second panoramic image of the scene from a second point on the vertical axis, the second point being displaced from the first point by a stereoscopic baseline distance.

3. A method as claimed in claim 2 wherein the step (b) of generating orthographic range values comprises deriving range values from the first and second panoramic images.

4. A method as claimed in claim 1 wherein the step (c) of transforming the range values comprises transforming the range values to perspective range distances about an axis through a rear nodal point of a capture device.

5. A method as claimed in claim 1 wherein the step (c) of transforming the range values comprises transforming the range values to directional range distances about a given viewpoint.

6. A method as claimed in claim 1 wherein the step (g) of blending the registered images comprises comparing the structure of the overlap regions for image pixels having common range values and extrapolating other range values for other pixels where common range values exist within a predetermined distance from each other.

7. A method as claimed in claim 1 further comprising the steps of:

(g) acquiring a plurality of stereo intensity image pairs of the scene, wherein there is an overlap region between adjacent stereo intensity image pairs;

(h) warping the intensity values onto a cylindrical surface, and forming therefrom a plurality of warped intensity images;

(i) registering one or more common values in the overlap regions of the adjacent warped intensity images; and

(j) blending the overlap regions of the registered warped intensity images to generate a panoramic environment map containing intensity information.

8. A method as claimed in claim 7 wherein the step (e) of registering the warped range images comprises registering the range images to the same locations as were determined during the step (i) of registering the adjacent warped intensity images.

9. A method as claimed in claim 1 further comprising generating a panoramic environment map containing intensity information, and concatenating the panoramic range and intensity environment maps to generate a panoramic environment map containing both range and intensity information.

10. A computer program product for forming a depth image from at least two images of the scene comprising:

a computer readable storage medium having a computer program stored thereon for performing the steps of;

(a) acquiring a plurality of stereo image pairs of the scene, wherein there is an overlap region between adjacent stereo image pairs;

(b) generating orthographic range values from each of the stereo image pairs corresponding to the orthogonal distance of image features in the stereo pairs from an image capture point;

(c) transforming the range values into transformed range values referenced to a coordinate system that reflects a common projection of the features from each stereo image pair;

(d) warping the transformed range values onto a cylindrical surface, and forming therefrom a plurality of warped range images;

(e) registering the adjacent warped range images; and

(f) blending the overlap regions of the registered warped range images to generate a panoramic environment map containing range information.

11. A computer program product as claimed in claim 10 wherein the step (a) of acquiring a plurality of stereo image pairs of a scene comprises the steps of:

a) capturing a first panoramic image of the scene from a first point on a vertical axis; and

b) capturing a second panoramic image of the scene from a second point on the vertical axis, the second point being displaced from the first point by a stereoscopic baseline distance.

12. A computer program product as claimed in claim 11 wherein the step (b) of generating orthographic range values comprises deriving range values from the first and second panoramic images.

13. A computer program product as claimed in claim 10 wherein the step (c) of transforming the range values comprises transforming the range values to perspective range distances about an axis through a rear nodal point of a capture device.

14. A computer program product as claimed in claim 10 wherein the step (c) of transforming the range values comprises transforming the range values to directional range distances about a given viewpoint.

15. A computer program product as claimed in claim 10 wherein the step (g) of blending the registered images comprises comparing the structure of the overlap regions for image pixels having common range values and extrapolating other range values for other pixels where common range values exist within a predetermined distance from each other.

16. A computer program product as claimed in claim 10 further comprising the steps of:

(g) acquiring a plurality of stereo intensity image pairs of the scene, wherein there is an overlap region between adjacent stereo intensity image pairs;

(h) warping the intensity values onto a cylindrical surface, and forming therefrom a plurality of warped intensity images;

(i) registering one or more common values in the overlap regions of the adjacent warped intensity images; and

(j) blending the overlap regions of the registered warped intensity images to generate a panoramic environment map containing intensity information.

17. A computer program product as claimed in claim 16 wherein the step (e) of registering the warped range images comprises registering the range images to the same locations as were determined during the step (i) of registering the adjacent warped intensity images.

18. A computer program product as claimed in claim 10 further comprising generating a panoramic environment map containing intensity information, and concatenating the panoramic range and intensity environment maps to generate a panoramic environment map containing both range and intensity information.