Scene capturing and view rendering based on a longitudinally aligned camera array

• US 20050052543A1
• Filed: 10/21/2004
• Published: 03/10/2005
• Est. Priority Date: 06/28/2000
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Li, Jiang, Shum, Heung-Yeung
• Granted Patent: US 7,538,797 B2
• Time in Patent Office: Days
• US Class Current: 348/222.100
• CPC Class Codes: H04N 13/221 using the relative movement between cameras and objects
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking

Abstract

In one embodiment, a longitudinal camera array is rotated through a capture cylinder, with each camera in the array capturing multiple images as the array rotates. These images can be looking outward along the radials of the cylinder, or alternatively looking tangential to the cylinder. The longitudinal camera array allows the surrounding scene to be captured from multiple different planes that are substantially parallel to the ends of the capture cylinder, allowing for more accurate subsequent rendering of the scene. A view of the scene can be subsequently rendered by determining a location and direction of view of an observer, and then selecting one or more of the multiple lateral and longitudinally adjacent capture images, as well as one or more pixels within that capture image(s), to use to determine a display value for the pixel.

Claims

1. A method for generating a rendered view image of a 3D scene for display by a computing system, the method comprising:

generating a series of concentric mosaic array images each of which comprises a plurality of slit images that collectively depict the surrounding 3D scene; and

generating a rendered view image of the 3D scene from an observer viewpoint within circular regions defined by the series of concentric mosaics using the plurality of slit images.

2. A method as recited in claim 1, further comprising receiving, as the plurality of slit images, images that were captured by one or more cameras.

3. A method as recited in claim 1, further comprising generating a new rendered view image of the 3D scene in response to movement of the observer viewpoint in three dimensions.

4. A method as recited in claim 1, wherein the 3D scene is defined by a capture cylinder, and wherein the observer viewpoint is able to move within the capture cylinder but is constrained such that the field of view from the observer viewpoint does not exceed the capture cylinder.

5. A method as recited in claim 1, further comprising concurrently generating a second rendered view image of the 3D scene, wherein the rendered view image corresponds to a viewing position of one eye of an eye pair and the second rendered view image corresponds to a viewing position of another eye of the eye pair.

6. A method as recited in claim 1, wherein the 3D scene is defined by a capture cylinder based on the plurality of slit images, and wherein generating the rendered view image further comprises:

determining an intersection point between the capture cylinder and a viewing ray passing through a pixel of the rendered view image in a direction of viewing of an observer from the observer viewpoint; and

using the intersection point to determine which one or more of the plurality of slit images to use to determine a display value for the pixel.

7. One or more computer-readable media having stored thereon a computer program to generate a rendered view image of a 3D scene for display, wherein the computer program, when executed by one or more processors of a computer, causes the one or more processors to:

generate a series of concentric mosaic array images each of which comprises a plurality of slit images that collectively depict the surrounding 3D scene; and

generate a rendered view image of the 3D scene from an observer viewpoint within circular regions defined by the series of concentric mosaics using the plurality of slit images.

8. One or more computer-readable media as recited in claim 7, the plurality of slit images having been captured by one or more cameras.

9. One or more computer-readable media as recited in claim 7, wherein the computer program further causes the one or more processors to generate a new rendered view image of the 3D scene in response to movement of the observer viewpoint in three dimensions.

10. One or more computer-readable media as recited in claim 7, wherein the 3D scene is defined by a capture cylinder, and wherein the observer viewpoint is able to move within the capture cylinder but is constrained such that the field of view from the observer viewpoint does not exceed the capture cylinder.

11. One or more computer-readable media as recited in claim 7, wherein the computer program further causes the one or more processors to concurrently generate a second rendered view image of the 3D scene, wherein the rendered view image corresponds to a viewing position of one eye of an eye pair and the second rendered view image corresponds to a viewing position of another eye of the eye pair.

12. One or more computer-readable media as recited in claim 7, wherein the 3D scene is defined by a capture cylinder based on the plurality of slit images, and wherein to generate the rendered view image is to:

determine an intersection point between the capture cylinder and a viewing ray passing through a pixel of the rendered view image in a direction of viewing of an observer from the observer viewpoint; and

use the intersection point to determine which one or more of the plurality of slit images to use to determine a display value for the pixel.

13. A system comprising:

means for generating a series of concentric mosaic array images each of which comprises a plurality of slit images that collectively depict the surrounding 3D scene; and

means for generating a rendered view image of the 3D scene from an observer viewpoint within circular regions defined by the series of concentric mosaics using the plurality of slit images.

14. A system as recited in claim 13, further comprising means for receiving, as the plurality of slit images, images that were captured by one or more cameras.

15. A system as recited in claim 13, wherein the means for generating are further for generating a new rendered view image of the 3D scene in response to movement of the observer viewpoint in three dimensions.

16. A system as recited in claim 13, wherein the 3D scene is defined by a capture cylinder, and wherein the observer viewpoint is able to move within the capture cylinder but is constrained such that the field of view from the observer viewpoint does not exceed the capture cylinder.

17. A system as recited in claim 13, further comprising means for concurrently generating a second rendered view image of the 3D scene, wherein the rendered view image corresponds to a viewing position of one eye of an eye pair and the second rendered view image corresponds to a viewing position of another eye of the eye pair.

18. A system as recited in claim 13, wherein the 3D scene is defined by a capture cylinder based on the plurality of slit images, and wherein the means for generating the rendered view image further comprises:

means for determining an intersection point between the capture cylinder and a viewing ray passing through a pixel of the rendered view image in a direction of viewing of an observer from the observer viewpoint; and

means for using the intersection point to determine which one or more of the plurality of slit images to use to determine a display value for the pixel.

19. A system for generating a rendered view image of a 3D scene using concentric mosaic arrays, the system comprising:

a general purpose computing device;

a computer program comprising one or more program modules executable by the computing device; and

wherein the program modules comprise a concentric mosaic array rendering module, a concentric mosaic image capture module for capturing and combining a set of vertically aligned images obtained at a plurality of radial locations within the 3D scene into a concentric mosaic array and a concentric mosaic array scene combination module for generating a rendered view image of a 3D scene as seen by an observer having a point of view within the 3D scene.

20. A system as recited in claim 19, wherein the set of vertically aligned images were captured by one or more cameras.

21. A system as recited in claim 19, wherein the concentric mosaic array scene combination module is further for generating a new rendered view image of the 3D scene in response to movement of the observer in three dimensions.

22. A system as recited in claim 19, wherein the 3D scene is defined by a capture cylinder, and wherein the observer is able to move within the capture cylinder but is constrained such that the field of view of the observer does not exceed the capture cylinder.

23. A system as recited in claim 19, wherein the concentric mosaic array scene combination module is further for concurrently generating a second rendered view image of the 3D scene, wherein the rendered view image corresponds to a viewing position of one eye of an eye pair and the second rendered view image corresponds to a viewing position of another eye of the eye pair.