GENERATING FREE VIEWPOINT VIDEO USING STEREO IMAGING

• US 20130095920A1
• Filed: 10/13/2011
• Published: 04/18/2013
• Est. Priority Date: 10/13/2011
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Patiejunas, Kestutis, Mitra, Kanchan, Kang, Sing Bing, Zitnick, Charles Lawrence III, Eraker, David, Winder, Simon, Eshet, Yaron, Mital, Amit, Sweeney, Patrick, Kirk, Adam G., Harnett, David
• Application Number:
  US13/273,213
• Publication Number:
  US 20130095920A1
• Time in Patent Office: Days
• US Class Current: 463/31
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

Methods and systems for generating free viewpoint video using an active infrared (IR) stereo module are provided. The method includes computing a depth map for a scene using an active IR stereo module. The depth map may be computed by projecting an IR dot pattern onto the scene, capturing stereo images from each of two or more synchronized IR cameras, detecting dots within the stereo images, computing feature descriptors corresponding to the dots in the stereo images, computing a disparity map between the stereo images, and generating the depth map using the disparity map. The method also includes generating a point cloud for the scene using the depth map, generating a mesh of the point cloud, and generating a projective texture map for the scene from the mesh of the point cloud. The method further includes generating the video for the scene using the projective texture map.

Claims

1. A method for generating a video using an active infrared (IR) stereo module, comprising:

computing a depth map for a scene using the active IR stereo module, wherein computing the depth map comprises;

projecting an IR dot pattern onto the scene;

capturing stereo images from each of two or more synchronized IR cameras;

detecting a plurality of dots within the stereo images;

computing a plurality of feature descriptors corresponding to the plurality of dots in the stereo images;

computing a disparity map between the stereo images; and

generating a depth map for the scene using the disparity map;

generating a point cloud for the scene in three-dimensional space using the depth map;

generating a mesh of the point cloud;

generating a projective texture map for the scene from the mesh of the point cloud; and

generating the video for the scene using the projective texture map.

2. The method of claim 1, wherein the video is a Free Viewpoint Video (FVV).

3. The method of claim 1, comprising:

displaying the video on a display device; and

enabling space-time navigation by a user during video playback.

4. The method of claim 1, comprising capturing stereo images from each of two or more synchronized IR cameras using one or more IR projectors, one or more synchronized RGB camera, or any combination thereof.

5. The method of claim 1, comprising:

computing a depth map for each of two or more synchronized active IR stereo modules;

generating a point cloud for the scene in three-dimensional space for each of the two or more synchronized active IR stereo modules;

combining point clouds generated by the two or more synchronized active IR stereo modules;

creating a mesh of combined point clouds; and

generating the video by creating a projective texture map on the mesh.

6. The method of claim 5, wherein computing the depth map for each of two or more synchronized active IR stereo modules comprises:

projecting an IR dot pattern onto a scene;

generating a synchronization signal for genlocking of the two or more synchronized active IR stereo modules; and

confirming that each of the two or more synchronized active IR stereo modules has received the synchronization signal and, if confirmation is received, generating the depth map for the scene for each of the two or more synchronized active IR stereo modules.

7. The method of claim 1, wherein generating the point cloud for the scene in three-dimensional space using the depth map comprises converting the depth map into a three-dimensional point cloud.

8. The method of claim 1, wherein generating the mesh of the point cloud comprises converting the point cloud into a geometric mesh that is a three-dimensional representation of objects in the scene.

9. The method of claim 1, wherein generating the projective texture map for the scene comprises generating the projective texture map by projecting RGB image data from the active IR stereo module onto the mesh of the point cloud.

10. The method of claim 1, wherein generating the video by creating the projective texture map comprises using image-based rendering methods to combine the projective texture map with real images to create synthetic viewpoints between real images.

11. A system for generating a video using an active infrared (IR) stereo module, comprising:

a processor configured to implement random stereo modules, wherein the random stereo modules comprise;

a depth map computation module configured to compute a depth map for a scene using the active IR stereo module, wherein the active IR stereo module comprises three or more synchronized cameras and an IR dot pattern projector;

a point cloud generation module configured to generate a point cloud for the scene in three-dimensional space using the depth map;

a point cloud mesh generation module configured to generate a mesh of the point cloud;

a projective texture map generation module configured to generate a projective texture map for the scene from the mesh of the point cloud; and

a video generation module configured to generate the video for the scene using the projective texture map.

12. The system of claim 11, comprising:

a processor configured to implement random stereo modules, wherein the random stereo modules comprise;

a video display module configured to display the video on a display device; and

a video playback module configured to enable space-time navigation by a user during video playback.

13. The system of claim 11, wherein the system comprises a conferencing system for generating a real-time video using one or more active IR stereo modules in a room.

14. The system of claim 11, wherein the system comprises a gaming system for generating a real-time video using one or more active IR stereo modules connected to a gaming device.

15. The system of claim 14, wherein the three or more synchronized cameras comprise two or more synchronized IR cameras and one or more synchronized RGB camera.

16. One or more non-volatile computer-readable storage media for storing computer readable instructions, the computer-readable instructions providing a stereo module system for generating a video using an active infrared (IR) stereo module when executed by one or more processing devices, the computer-readable instructions comprising code configured to:

compute a depth map for a scene using the active IR stereo module, wherein computing the depth map comprises;

projecting an IR dot pattern onto the scene;

capturing stereo images from each of two or more synchronized IR cameras;

detecting a plurality of dots within the stereo images;

computing a plurality of feature descriptors corresponding to the plurality of dots in the stereo images;

computing a disparity map between the stereo images; and

generating the depth map for the scene using the disparity map;

generate a point cloud for the scene in three-dimensional space using the depth map;

generate a mesh of the point cloud;

generate a projective texture map for the scene from the mesh of the point cloud; and

generate the video by combining the projective texture map with real images.

17. The non-volatile computer-readable storage media of claim 16, wherein the computer-readable instructions comprise code further configured to:

display the video on a display device; and

enable space-time navigation by a user during video playback.

18. The non-volatile computer-readable storage media of claim 16, wherein the active IR stereo module comprises two or more synchronized IR cameras, one or more synchronized RGB camera, or any combination thereof.

19. The non-volatile computer-readable storage media of claim 16, wherein the computer-readable instructions comprise code further configured to:

compute a depth map for each of two or more synchronized active IR stereo modules;

generate a point cloud for the scene in three-dimensional space for each of the two or more synchronized active IR stereo modules;

combine point clouds generated by the two or more synchronized active IR stereo modules;

create a mesh of combined point clouds; and

generate the video by creating a projective texture map for the scene.

20. The non-volatile computer-readable storage media of claim 19, wherein the code configured to compute the depth map for each of the two or more synchronized active IR stereo modules further comprises code configured to:

project an IR dot pattern onto the scene;

generate a synchronization signal for genlocking of the two or more synchronized active IR stereo modules; and

confirm that each of the two or more synchronized active IR stereo modules has received the synchronization signal and, if confirmation is received, generating the depth map for the scene for each of the two or more synchronized active IR stereo modules.