Method for three dimensional spatial panorama formation

• US 6,677,982 B1
• Filed: 10/11/2000
• Issued: 01/13/2004
• Est. Priority Date: 10/11/2000
• Status: Expired due to Fees

• Current Assignee: Eastman Kodak Company
• Original Assignee: Eastman Kodak Company
• Inventors: Chen, Shoupu, Ray, Lawrence A.
• Primary Examiner(s): DIEP, NHON THANH
• Application Number:
  US09/686,610
• Time in Patent Office: 1,189 Days
• Field of Search: H04/N.700, H04/N 1.300, H04/N 1.500, 348/36, 348/37, 348/38, 348/39, 348/42, 348/47, 348/51, 348/52
• US Class Current: 348/36
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: H04N 13/106 Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

A method, a system, and a computer program product for deriving a three-dimensional panorama from a plurality of stereo image pairs of a scene generated from a plurality of cameras, that includes acquiring a plurality of stereo image pairs of the scene, wherein there is an intra-overlap region between vertically aligned stereo image pairs; acquiring a plurality of stereo image pairs of the scene by rotating the plurality of cameras about a Y-axis, wherein there is an inter-overlap region between adjacent stereo image pairs; providing displacement data for capturing scene depth from a local XYZ image with non-distorted depth information; generating (X,Y,Z) values for each of the stereo image pairs with respect to a local three-dimensional coordinate system wherein the intra-stereo image pair is taken; selecting a reference three-dimensional world coordinate system against which spatial information of the scene can be correctly presented; transforming the generated (X,Y,Z) values from each of the local three-dimensional coordinate systems to the selected reference three-dimensional world coordinate system; warping the transformed (X,Y,Z) images onto a cylindrical surface, and forming a plurality of warped (X,Y,Z) images; registering adjacent warped (X,Y,Z) images; and forming a three-dimensional panorama, i.e., a (X,Y,Z) panorama using the warped (X,Y,Z) images.

Claims

1. A method for deriving a three-dimensional panorama from a plurality of stereo image pairs of a scene generated from a plurality of cameras, the method comprising the steps of:

(a) acquiring a plurality of stereo image pairs of the scene, wherein there is an intra-overlap region between vertically aligned stereo image pairs;

(b) acquiring a plurality of stereo image pairs of the scene by rotating the plurality of cameras about a Y-axis (vertical axis), wherein there is an inter-overlap region between adjacent stereo image pairs;

(c) providing disparity data for every pair (intra-pair) of stereo images to recover scene spatial information (X,Y,Z) with respect to local XYZ coordinate systems;

(d) selecting a reference three-dimensional world coordinate system against which spatial information of the scene can be correctly presented;

(e) transforming the scene spatial information (X,Y,Z) values from each of the local three-dimensional coordinate systems to the selected reference three-dimensional world coordinate system;

(f) warping the transformed (X,Y,Z) images onto a cylindrical surface, and forming a plurality of warped (X,Y,Z) images;

(g) registering adjacent warped (X,Y,Z) images; and

(h) forming a three-dimensional panorama, i.e., a (X,Y,Z) panorama using the warped (X,Y,Z) images.

2. The method claimed in claim 1, and the step of providing disparity data further comprises:

(i) detecting corresponding image pixels in a stereo image pair;

(ii) fixing the local three-dimensional coordinate system relative to the stereo two-dimensional coordinate systems such that image pixels are defined.

3. The method as claimed in claim 1, wherein the three dimensional panorama is in color.

4. The method claimed in claim 1, wherein the reference three-dimensional world coordinate system is an arbitrary three-dimensional coordinate system.

5. The method claimed in claim 4 and further comprising the step of selecting the reference three-dimensional world coordinate system from the local three-dimensional coordinate systems or a three-dimensional coordinate system defined elsewhere.

6. The method claimed in claim 1 wherein step (e) of transforming the scene spatial information (X,Y,Z) values comprises forming a homogeneous transformation matrix.

7. A three-dimensional panoramic system for producing a sequence of spatial (X,Y,Z) images from stereo pairs of intensity (R,G,B) images, comprising:

(a) a panoramic three-dimensional capturing component for acquiring a plurality of stereo (X,Y,Z) image pairs of the scene, wherein there is an overlap region between adjacent stereo (X,Y,Z) image pairs;

(b) a reference coordinate component for warping the (X,Y,Z) image pixels onto a cylindrical surface, and a plurality of warped (X,Y,Z) images, including means for registering one or more common pixels in the overlap regions of the adjacent warped (X,Y,Z) image;

(c) an image stitching component for stitching the overlap regions of a registered warped (X,Y,Z) image to generate a (X,Y,Z) panorama; and

(d) a graphics display for visually displaying the (X,Y,Z) panorama.

8. A three-dimensional panoramic image homogenous system, comprising:

a three-dimensional panoramic capturing system, wherein a sequence of spatial (X,Y,Z) images and a sequence of intensity (R,G,B) images are produced;

a reference coordinate transformation system such that each of the spatial images is transformed to a common three-dimensional coordinate system from its local three-dimensional coordinate system at which the corresponding intensity image is taken and the original spatial image is computed;

an image stitching system that produces a stitched spatial panorama, from the transformed spatial images, and a stitched intensity panorama from the sequence of intensity images; and

a graphics display system for receiving the stitched spatial and intensity panoramas, and generating a virtual world reality.

9. A computer program product for constructing a complete 3-D panorama from at least two scene images, comprising:

a computer readable storage medium having a computer program stored thereon for performing the steps of;

(a) acquiring a plurality of stereo image pairs of the scene, wherein there is an intra-overlap region between vertically aligned stereo image pairs;

(b) acquiring a plurality of stereo image pairs of the scene by rotating the plurality of cameras about a Y-axis (vertical axis), wherein there is an inter-overlap region between adjacent stereo image pairs;

(c) providing disparity data for every pair (intra-pair) of stereo images to recover scene spatial information (X,Y,Z) with respect to local XYZ coordinate systems;

(d) selecting a reference three-dimensional world coordinate system against which spatial information of the scene can be correctly presented;

(e) transforming the scene spatial information (X,Y,Z) values from each of the local three-dimensional coordinate systems to the selected reference three-dimensional world coordinate system;

(f) warping the transformed (X,Y,Z) images onto a cylindrical surface, and forming a plurality of warped (X,Y,Z) images;

(g) registering adjacent warped (X,Y,Z) images; and

(h) forming a three-dimensional panorama, i.e., a (X,Y,Z) panorama using the warped (X,Y,Z) images.

10. A computer program product as claimed in claim 9, wherein recovering scene spatial information (X,Y,Z) values comprises:

(i) detecting corresponding image pixels in a stereo image pair;

(ii) fixing the local three-dimensional coordinate system relative to the stereo two-dimensional coordinate systems where image pixels are defined.

11. A computer program product as claimed in claim 9, comprising generating a complete panoramic scene panorama containing both (R,G,B) and (X,Y,Z) panoramas.

12. A computer program product as claimed in claim 9, wherein the reference three-dimensional world coordinate system is an arbitrary three-dimensional coordinate system.

13. A computer program product as claimed in claim 12, further comprises selecting the reference three-dimensional world coordinate system from the local three-dimensional coordinate systems or a three-dimensional coordinate system defined elsewhere.

14. A computer program product as claimed in claim 9, wherein step (e) of transforming the scene spatial information (X,Y,Z) values comprises forming a homogeneous transformation matrix.