Method and system for reconstructing 3D interactive walkthroughs of real-world environments

• US 20020176635A1
• Filed: 04/16/2002
• Published: 11/28/2002
• Est. Priority Date: 04/16/2001
• Status: Active Grant

• Current Assignee: Lucent Technologies, Inc. (Nokia Corporation)
• Original Assignee: Lucent Technologies, Inc. (Nokia Corporation)
• Inventors: Aliaga, Daniel G., Carlbom, Ingrid B.
• Granted Patent: US 6,831,643 B2
• Time in Patent Office: Days
• US Class Current: 382/284
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 10/16 using multiple overlapping images; Image stitching

Abstract

An omnidirectional video camera captures images of the environment while moving along several intersecting paths forming an irregular grid. These paths define the boundaries of a set of image loops within the environment. For arbitrary viewpoints within each image loop, a 4D plenoptic function may be reconstructed from the group of images captured at the loop boundary. For an observer viewpoint, a strip of pixels is extracted from an image in the loop in front of the observer and paired with a strip of pixels extracted from another image on the opposite side of the image loop. A new image is generated for an observer viewpoint by warping pairs of such strips of pixels according to the 4D plenoptic function, blending each pair, and then stitching the resulting strips of pixels together.

Claims

1. A method of generating images of an environment comprising:

stitching together image data obtained at multiple points along boundaries of an image loop to generate an image for a viewpoint within the boundaries of the image loop, the boundaries of the image loop formed by intersecting image paths within the environment, the viewpoint and the image loop being on an observer plane.

2. The method of claim 1, wherein the image data is obtained by,

capturing images, each having a center-of-projection of known position and orientation, along intersecting paths within the observer plane, the intersecting paths defining boundaries of at least one image loop.

3. The method of claim 1, wherein the stitching step comprises:

defining an array of line segments through the viewpoint corresponding to a field-of-view (FOV) from the viewpoint, each line segment of the FOV lying on the image plane and intersecting the image loop; and

combining, for each line segment of the FOV, data from the images of the loop.

4. The method of claim 3, wherein each line segment is a stitching line.

5. The method of claim 3, wherein the combining step includes,

extracting, for each line segment, strips of pixels from the image data, each strip of pixels being extracted from the images recorded along the boundaries.

6. The method of claim 5, wherein the extracting step for a strip of pixels at an intersection of a line segment with the boundary of the image loop includes,

obtaining two captured images on the boundary of the image loop closest to the intersection point;

for each of the two captured images,

obtaining two strips of pixels, which are closest to being parallel to the line segment; and

blending the two strips of pixels to produce a single extracted strip of pixels;

blending the extracted strips of pixels from each of the two captured images to produce a single strip of pixels for the intersection of the line segment with the boundary of the image loop.

7. The method of claim 5, wherein the extracting step extracts the strips of pixels from omnidirectional images recorded along the boundaries, each omnidirectional image having a 360°

horizontal FOV of the environment and an effective center-of-projection.

8. The method of claim 5, wherein

the extracting step extracts the strips of pixels from separate images recorded along the boundaries, and each strip of pixels recording light rays corresponding to a sample of the environment perpendicular to a plane of the image loop and viewed from a center-of-projection on the image loop plane.

9. The method of claim 5, wherein the combining step combines the extracted strips of pixels according to a mapping, which correlates identified environment features within the images recorded along the boundaries of the loop.

10. The method of claim 9, wherein the identified environment features include those features located outside the image loop, whose projection onto each image recorded along the boundaries is successfully tracked.

11. The method of claim 3, wherein the combining step comprises:

extracting a first image from the images recorded along the boundaries, the first image having a center-of-projection located approximately at a first intersection point of the line segment with the boundaries of the image loop;

extracting a second image from the images recorded along the boundaries, the second omnidirectional image having a center-of-projection located approximately at a second intersection point of the line segment with the boundaries of the image loop;

extracting a strip of pixels from each of the first and second images;

establishing a mapping, which correlates identified environment features within the first and second images recorded along the boundaries;

computing a parametric position of the viewpoint along the line segment; and

creating a strip of pixels for a novel viewpoint by warping and blending the extracted strips of pixels of the first and second omnidirectional images according to the parametric position and the mapping.

12. The method of claim 11, wherein

the generated image comprises one of a planar projection and cylindrical projection;

and the creating step creates the strip of pixels as an image column.

13. The method of claim 11, wherein the creating step includes,

optimizing at least one of the extracted strips of pixels to correct errors resulting from at least one of the following:

incorrectly estimated image rotation, and misalignment of image features.

15. The method of claim 14, wherein the creating step includes,

defining the secondary surface to be a surface, which is perpendicular to the observer plane, and which surrounds a visually unobstructed area of the observer plane in the environment.

17. The method of claim 16, wherein the plenoptic function includes a mapping for each of a plurality of line segments intersecting the viewpoint, each mapping identifying light rays emanating to the viewpoint.

18. The method of claim 17, wherein the generating step includes,

extracting two strips of pixels from two separate images captured along the boundaries of the image loop;

and

warping and blending the two extracted strips of pixels to produce a single extracted strip of pixels based on the mappings of the plenoptic function.

19. The method of claim 17, further comprising,

computing and storing the mappings in the computer-readable storage device, said computing and storing steps being performed prior to the inputting step.

20. The method of claim 19, wherein the step of computing a mapping comprises:

selecting, for each image captured along the boundaries of the image loop, a set of strips of pixels; and

for each selected strip of pixels,

extracting a strip of pixels from a separate image captured along the boundaries of the selected image loop, the extracted strip of pixels being substantially parallel to the determined strip of pixels, and

establishing correlating identified environmental features within the selected strip of pixels and the extracted strip of pixels.

14. A method of generating an image comprising:

creating a four-dimensional plenoptic function based on samples of the environment, the plenoptic function including a plurality of coefficients representing the intensity of light rays arriving at an observer located at a two-dimensional point (x, y) of the observer plane, the light rays intersecting a secondary surface at (u, v), and the coefficients being indexed according to the observer plane point (x, y) and the secondary surface point (u, v); and

generating an image from the plurality of coefficients.

16. A method for generating an image of a three-dimensional environment in a computer-implemented interactive walkthrough application, comprising:

inputting a viewpoint and orientation;

selecting a predefined image loop containing the viewpoint, boundaries of the image loop being defined according to a set of intersecting paths along which images have been captured in the environment;

retrieving, from a computer-readable storage device, images captured along the boundaries of the determined image loop; and

generating an image corresponding to the viewpoint and orientation by combining pixels of the retrieved images according to a four-dimensional plenoptic function representing the intensity values of light rays emanating to the viewpoint.

21. A method of gathering image data of an environment comprising:

recording images of the environment at various points of intersecting paths within a plane of the environment, at least a portion of the intersecting paths defining boundaries of an image loop;

determining a three-dimensional position and orientation relative to the environment corresponding to each image.

22. The method of claim 21, wherein the step of recording images comprises:

using a hemispherical field-of-view omnidirectional camera with an effective center-of-projection to record the images.

23. The method of claim 22, wherein the step of recording images further comprises:

conveying the omnidirectional camera to each of the various points along the intersecting paths, the omnidirectional camera being conveyed at a fixed height.

24. The method of claim 23, wherein the steps of using the omnidirectional camera and conveying the omnidirectional camera are performed by wired or wireless control.

26. The apparatus of claim 25, wherein the means for stitching includes,

a data storage device for storing image data of the environment captured along the boundaries of the image loop;

means for generating a mapping for each of a plurality of line segments intersecting the viewpoint, each mapping correlating environment features within the image data; and

27. The apparatus of claim 26, wherein the means for stitching further includes,

means for defining a plurality of line segments extending through the viewpoint, each line segment intersecting the image loop;

means for storing the generated mapping in the data storage device;

means for retrieving from the data storage device, for each line segment, a mapping; and

means for generating a strip of pixels corresponding to the line segment by combining pixels from the stored image data according to the mapping.

25. An apparatus for generating an image of a three-dimensional environment comprising:

means for determining a viewpoint and orientation within the environment; and

means for stitching together image data obtained at multiple points along boundaries of an image loop containing the viewpoint in order to generate an image corresponding to the viewpoint and the orientation, the boundaries of the image loop formed by intersecting image paths within the environment, the viewpoint and the image loop being on an observer plane.