Three dimensional spatial panorama formation with a range imaging system

• US 20020154812A1
• Filed: 03/12/2001
• Published: 10/24/2002
• Est. Priority Date: 03/12/2001
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Eastman Kodak Company
• Inventors: Chen, Shoupu, Cahill, Nathan D.
• Granted Patent: US 7,194,112 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/221 using the relative movement between cameras and objects
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0088 Synthesising a monoscopic image signal from stereoscopic images, e.g. synthesising a panoramic or high resolution monoscopic image

Abstract

Deriving a three-dimensional panorama from a plurality of images of a scene generated by a range imaging camera of the type that produces ambiguities in range information includes the steps of: (a) acquiring a plurality of adjacent images of the scene, wherein there is an overlap region between the adjacent images and at least some of the adjacent images are range images; (b) providing offset data for the range images in order to recover corrected relative scene spatial information and provide a corrected range image, and (c) deriving a three-dimensional panorama from the corrected range image. In order to provide offset data, a relative range difference is detected between adjacent range images as a constant offset between the adjacent images; and the constant offset is applied to at least one of adjacent range images to correct for ambiguities in the relative ranges of the range images.

Claims

1. A method for deriving a three-dimensional panorama from a plurality of images of a scene generated by a range imaging camera of the type that produces ambiguities in range information, said method comprising the steps of:

(a) acquiring a plurality of adjacent images of the scene, wherein there is an overlap region between the adjacent images and at least some of the adjacent images are range images;

(b) providing offset data for the range images in order to recover corrected relative scene spatial information, wherein the step of providing offset data further comprises;

(i) detecting a relative range difference between adjacent range images as a constant offset between the adjacent images;

(ii) applying the constant offset to at least one of adjacent range images to correct for ambiguities in the relative ranges of the range images, thereby providing corrected range images; and

(c) deriving a three-dimensional panorama from the corrected range images.

2. The method as claimed in claim 1 wherein scene spatial information is provided as image values in a local three-dimensional coordinate system of each of the images and the step (c) of deriving a three-dimensional panorama comprises the steps of:

(i) transforming the image values from each of the local three-dimensional coordinate systems of each of the images to a selected reference three-dimensional world coordinate system, thereby providing transformed range images;

(ii) warping the transformed range images onto a cylindrical surface, and forming a plurality of warped range images;

(iii) registering adjacent warped range images; and

(iv) deriving the three-dimensional panorama using the warped range images.

3. The method as claimed in claim 1 wherein the step (i) of detecting a relative range difference between adjacent range images as a constant offset comprises the steps of:

(i1) predicting the relative range difference by an estimated constant offset;

(i2) warping the range images onto a cylindrical surface using the estimated constant offset, and forming a plurality of warped range images;

(i3) registering adjacent warped range images, thereby producing predicted range values,

(i4) evaluating any error between the predicted range values and the actual range values in the overlap region;

(i5) if the error is unacceptable, using an optimization routine to select another estimated constant offset; and

(i6) repeating the steps (i2) through (i5) until the error is acceptable, thereby producing the constant offset.

4. A method for deriving a three-dimensional panorama from a plurality of images of a scene generated from a range imaging camera of the type that produces ambiguities in range information, said method comprising the steps of:

(a) acquiring a plurality of images of the scene by rotating the camera about a Y-axis (vertical axis), wherein there is an inter-overlap region between adjacent images;

(b) providing offset data for each image to recover corrected relative scene spatial information (X,Y,Z) with respect to a local XYZ coordinate system;

(c) selecting a reference three-dimensional world coordinate system against which spatial information of the scene can be correctly presented;

(d) transforming the corrected relative scene spatial information (X,Y,Z) from each of the local three-dimensional coordinate systems of each of the images to the selected reference three-dimensional world coordinate system, thereby providing transformed (X,Y,Z) images;

(e) warping the transformed (X,Y,Z) images onto a cylindrical surface, and forming a plurality of warped (X,Y,Z) images;

(f) registering adjacent warped (X,Y,Z) images; and

(g) forming a three-dimensional (X,Y,Z) panorama using the warped (X,Y,Z) images.

5. The method claimed in claim 4, wherein the plurality of images includes range images and the step (b) of providing offset data further comprises the steps of:

(i) detecting differences in constant offset between the range images; and

(ii) using the differences to correct for ambiguities between the range images.

6. The method as claimed in claim 4 wherein the plurality of images generated from the range imaging camera includes color images and the three dimensional panorama is in color.

7. The method claimed in claim 4 wherein the reference three-dimensional world coordinate system is an arbitrary three-dimensional coordinate system.

8. The method claimed in claim 7 further comprising the step of selecting the reference three-dimensional world coordinate system from the local three-dimensional coordinate systems or a three-dimensional coordinate system defined elsewhere.

9. The method claimed in claim 4 wherein step (d) of transforming the corrected relative scene spatial information (X,Y,Z) comprises forming a homogeneous transformation matrix.

10. The method as claimed in claim 4 wherein each image is captured as a bundle of associated images, said bundle including a plurality of phase images each incorporating the effect of a predetermined modulation frequency together with a phase offset unique for each image.

11. The method as claimed in claim 10 wherein each range image is generated from a respective plurality of phase images associated with each bundle.

12. The method as claimed in claim 10 wherein the bundle also includes an intensity image.

13. The method as claimed in claim 12 wherein the intensity image is a color image.

18. A computer program product as claimed in claim 17, wherein the plurality of images includes range images and the step (b) of providing offset data farther comprises the steps of:

(i) detecting differences in constant offset between the range images; and

(ii) using the differences to correct for ambiguities between the range images.

19. A computer program product as claimed in claim 12 wherein step (d) of transforming the generated (X,Y,Z) values comprises forming a homogeneous transformation matrix.

14. A three-dimensional panoramic system for producing a sequence of spatial (X,Y,Z) images and a sequence of intensity (R,G,B) images, comprising:

(a) a panoramic three-dimensional scannerless range imaging capture component for acquiring a plurality of image bundles of the scene, wherein each image bundle includes an intensity (R,G,B) image and a plurality of phase images from which a spatial (X,Y,Z) image is derived, and wherein there is an overlap region between adjacent image bundles;

(b) a reference coordinate component for warping image pixels from each image bundle onto a cylindrical surface, thereby generating a plurality of warped images, said reference coordinate component including means for registering one or more common image pixels in the overlap regions of adjacent warped images, thereby providing a plurality of registered warped images;

(c) an image stitching component for stitching the overlap regions of the registered warped images to generate a panorama; and

(d) a graphics display for visually displaying the panorama.

15. A three-dimensional panoramic imaging system, comprising:

(a) a three-dimensional panoramic capturing system, wherein a sequence of spatial (X,Y,Z) images and a sequence of intensity (R,G,B) images are produced,

(b) a reference coordinate transformation system for generating transformed spatial images, said transformation system comprising a general homogenous transformation matrix for transforming each of the spatial images into a common three-dimensional coordinate system from its local three-dimensional coordinate system at which the corresponding intensity image is taken and the original spatial image is computed;

(c) an image stitching system that produces a stitched spatial panorama from the transformed spatial images, and a stitched intensity panorama from the sequence of intensity images; and

(d) a graphics display system for receiving the stitched spatial and intensity panoramas and generating a virtual world reality.

16. A computer program product for deriving a three-dimensional panorama from a plurality of images of a scene generated by a range imaging camera of the type that produces ambiguities in range information, said computer program product comprising:

a computer readable storage medium having a computer program stored thereon for performing the steps of

(a) accessing a plurality of adjacent images of the scene, wherein there is an overlap region between the adjacent images and at least some of the adjacent images are range images;

(b) providing offset data for the range images in order to recover corrected relative scene spatial information, wherein the step of providing offset data further comprises;

(i) detecting a relative range difference between adjacent range images as a constant offset between the adjacent images;

(ii) applying the constant offset to at least one of adjacent range images to correct for ambiguities in the relative ranges of the range images, thereby providing corrected range images; and

(c) deriving a three-dimensional panorama from the corrected range images.

17. A computer program product for deriving a three-dimensional panorama from a plurality of images of a scene generated from a range imaging camera of the type that produces ambiguities in range information, said computer program product comprising:

a computer readable storage medium having a computer program stored thereon for performing the steps of;

(a) accessing a plurality of images of the scene by rotating the camera about a Y-axis (vertical axis), wherein there is an inter-overlap region between adjacent images;

(b) providing offset data for each image to recover corrected relative scene spatial information (X,Y,Z) with respect to a local XYZ coordinate system;

(c) selecting a reference three-dimensional world coordinate system against which spatial information of the scene can be correctly presented;

(d) transforming the corrected relative scene spatial information (X,Y,Z) from each of the local three-dimensional coordinate systems of each of the images to the selected reference three-dimensional world coordinate system, thereby providing transformed (X,Y,Z) images;

(e) warping the transformed (X,Y,Z) images onto a cylindrical surface, and forming a plurality of warped (X,Y,Z) images;

(f) registering adjacent warped (X,Y,Z) images; and

(g) forming a three-dimensional (X,Y,Z) panorama using the warped (X,Y,Z) images.