SYSTEM FOR EXECUTING 3D PROPAGATION FOR DEPTH IMAGE-BASED RENDERING

• US 20110115886A1
• Filed: 11/17/2010
• Published: 05/19/2011
• Est. Priority Date: 11/18/2009
• Status: Active Grant

• Current Assignee: Board of Trustees of The University of Illinois
• Original Assignee: Board of Trustees of The University of Illinois
• Inventors: Patel, Sanjay Jeram, Do, Minh Ngoc, Nguyen, Quang Huu
• Granted Patent: US 8,643,701 B2
• Time in Patent Office: Days
• US Class Current: 348/47
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2207/20028 Bilateral filtering
• CPC Class Codes: G06T 5/003 Deblurring; Sharpening vibration or motion blur correction for cameras comprising an electronic image sensor H04N23/682
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/30 Image reproducers optical systems for producing stereoscopic or other three-dimensional effects G02B30/00
• CPC Class Codes: H04N 2213/003 Aspects relating to the "2D+depth" image format

Abstract

A system is disclosed for executing depth image-based rendering of a 3D image by a computer having a processor and that is coupled with one or more color cameras and at least one depth camera. The color cameras and the depth camera are positionable at different arbitrary locations relative to a scene to be rendered. In some examples, the depth camera is a low resolution camera and the color cameras are high resolution. The processor is programmed to propagate depth information from the depth camera to an image plane of each color camera to produce a propagated depth image at each respective color camera, to enhance the propagated depth image at each color camera with the color and propagated depth information thereof to produce corresponding enhanced depth images, and to render a complete, viewable image from one or more enhanced depth images from the color cameras. The processor may be a graphics processing unit.

Claims

1. A method for executing depth image-based rendering of a 3D image from one or more color cameras and at least one depth camera by a computer having a processor and memory, the method comprising:

propagating, by the computer, depth information from the at least one depth camera to an image plane of each color camera to produce a propagated depth image at each respective color camera, where propagating includes 3D warping with the depth camera considered as a first reference view and each color camera respectively being considered as a first target view, where the one or more color cameras and the at least one depth camera are positioned arbitrarily at different locations relative to a scene to be rendered;

enhancing, by the computer, the propagated depth image at each color camera with the color and propagated depth information thereof to produce corresponding enhanced depth images; and

rendering, by the computer, a complete image viewable on a screen of the computer from one or more enhanced depth images from the one or more color cameras.

2. The method of claim 1, where the at least one depth camera is a low resolution camera and the one or more color cameras are high resolution.

3. The method of claim 1, where the computer includes one or more graphics processing units (GPUs) and shared memory for access by the one or more GPUs, the method further comprising:

executing the propagating, the enhancing, and the rendering by the one or more GPUs.

4. The method of claim 3, where the one or more GPUs comprise one or more general purpose GPUs (GPGPUs) configured with compute unified device architecture (CUDA).

5. The method of claim 1, where the enhancing comprises executing the following processing techniques on the propagated depth image at each color camera to produce the corresponding enhanced depth images:

detect occluded pixels in the propagated depth images;

occlusion removal to replace values of the occluded pixels with newly interpolated values;

depth-color bilateral filtering (DCBF) to combine color and propagated depth information to calculate values for unknown depth pixels in the propagated depth image while preserving edges of an object in the scene;

directional disocclusion filling to fill areas of unknown depth pixels caused by the disocclusion in the propagated depth image, the filling direction originating from a plurality of epipoles related to each color camera; and

depth edge enhancement to sharpen depth edges surrounding objects in the propagated depth images.

6. The method of claim 5, where the complete image is rendered at an arbitrary virtual view position, and where rendering comprises:

propagating depth and color information from each enhanced depth image to a second target view defined by a virtual camera to produce the rendered image through 3D warping, where the enhanced depth images of the color cameras are merged to the virtual camera in which the color cameras are second reference views;

detecting and removing occlusions from the rendered image; and

processing the rendered image with a median filter to fill and denoise the rendered image.

7. The method of claim 5, where occlusion removal comprises, for a plurality of known depth pixels in the propagated depth image:

selecting a known depth pixel A;

partitioning a support window around pixel A into fourths; and

replacing the value of pixel A by a newly interpolated value if there are at least three partitions each having at least one pixel B such that the depth difference between pixels A and B is greater than a predetermined threshold value.

8. The method of claim 7, where the direction of the disocclusion filling is defined by a vector pointing from each respective epipole to a center of the target propagated depth image.

9. The method of claim 5, where depth edge enhancement comprises:

computing depth gradients in vertical, horizontal, and in at least one diagonal direction with Sobel operators;

for each pixel P whose depth edge gradients are greater than a predetermined value, searching within a search window among the neighbor pixels for a best color-matched pixel that has a smallest Euclidian color distance with pixel P; and

replacing the depth value of the pixel P by the depth value of the best color-matched pixel.

10. A system for executing depth image-based rendering from one or more color cameras and at least one depth camera, the system comprising:

a computer coupled with the one or more color cameras and the at least one depth camera, the computer including a processor and memory, where the one or more color cameras and the at least one depth camera are positioned arbitrarily at different locations in 3D space relative to a scene to be rendered;

where the processor is programmed to;

propagate depth information from the at least one depth camera to an image plane of each color camera to produce a propagated depth image at each respective color camera of the scene to be rendered, where to propagate the depth information includes 3D warping with the depth camera considered as a first reference view and each color camera respectively being considered as a first target view;

enhance the propagated depth image at each color camera with the color and propagated depth information thereof to produce corresponding enhanced depth images; and

render a complete image at a virtual view that is viewable on a screen of the computer from one or more enhanced depth images from the one or more color cameras.

11. The system of claim 10, where a resolution of the at least one depth camera is the same or higher than that of the one or more color cameras.

12. The system of claim 10, where a resolution of the at least one depth camera is the same as or lower than that of the one or more color cameras.

13. The system of claim 10, the computer further including one or more graphics processing units (GPU) having data parallel architectures, the computer also including shared memory for access by the one or more GPU, the one or more GPU programmed to execute the propagation, the enhancement, and the rendering steps.

14. The system of claim 13, where the one or more GPU comprises one or more general purpose GPU (GPGPU) configured with compute unified device architecture (CUDA).

15. The system of claim 13, where the one or more GPU is further programmed to, in execution of the enhancement step, execute the following processing techniques on the propagated depth image at each color camera to produce the corresponding enhanced depth images:

detect occluded pixels in the propagated depth images;

occlusion removal to replace values of the occluded pixels with newly interpolated values;

depth-color bilateral filtering (DCBF) to combine color and propagated depth information to calculate values for unknown depth pixels in the propagated depth image while preserving edges of an object in the scene;

directional disocclusion filling to fill areas of unknown depth pixels caused by the disocclusion in the propagated depth image, the filling direction originating from a plurality of epipoles related to each color camera; and

depth edge enhancement to sharpen depth edges surrounding objects in the propagated depth images.

16. The system of claim 15, where the one or more GPU is configured to render the complete 3D image at an arbitrary virtual view position, and where, to execute the rendering, the one or more GPU further programmed to:

propagate depth and color information from each enhanced depth image to a second target view defined by a virtual camera to produce the rendered image through 3D warping, where the enhanced depth images of the color cameras are merged to the virtual camera in which the color cameras are second reference views;

detect and remove occlusions from the rendered image; and

process the rendered image with a median filter to fill and denoise the rendered image.

17. The system of claim 15, where, to execute the occlusion removal for a plurality of known pixels, the one or more GPU further programmed to:

select a known depth pixel A;

partition a support window around pixel A into fourths; and

replace the value of pixel A by a newly interpolated value if there are at least three partitions each having at least one pixel B such that the depth difference between pixels A and B is greater than a predetermined threshold value.

18. The system of claim 17, where the direction of disocclusion filling is defined by a vector pointing from each respective epipole to a center of the target propagated depth image.

19. The system of claim 15, where, to execute the depth edge enhancement, the one or more GPU further programmed to:

compute depth gradients in vertical, horizontal, and in at least one diagonal direction with Sobel operators;

for each pixel P whose depth edge gradients are greater than a predetermined value, search within a search window among the neighbor pixels for a best color-matched pixel that has a smallest Euclidian color distance with pixel P; and

replace the depth value of the pixel P by the depth value of the best color-matched pixel.

20. A computer-readable storage medium comprising a set of instructions for executing depth image-based rendering of a 3D image from one or more color cameras and at least one depth camera by a computer having a processor and memory, the computer-readable medium comprising:

instructions to direct the processor to propagate depth information from the at least one depth camera to an image plane of each color camera to produce a propagated depth image at each respective color camera, where the one or more color cameras and the at least one depth camera are positioned arbitrarily at different locations relative to a scene to be rendered;

instructions to direct the processor to enhance the propagated depth image at each color camera with the color and propagated depth information thereof to produce corresponding enhanced depth images; and

instructions to direct the processor to render a complete image viewable on a screen of the computer from one or more enhanced depth images from the one or more color cameras.

21. The computer-readable storage medium of claim 20, where to propagate the depth information includes 3D warping with the depth camera considered as a first reference view and each color camera respectively being considered as a first target view.

22. The computer-readable storage medium of claim 21, where a resolution of the at least one depth camera is the same or higher than that of the one or more color cameras.

23. The computer-readable storage medium of claim 21, where a resolution of the at least one depth camera is the same as or lower than that of the one or more color cameras.

24. The computer-readable storage medium of claim 21, where the computer includes one or more graphics processing units (GPUs) having data parallel architectures, the computer also including shared memory for access by the one or more GPUs, the computer-readable storage medium further comprising:

instructions to direct the one or more GPUs to execute the propagation, the enhancement, and the rendering steps.

25. The computer-readable storage medium of claim 24, where the one or more GPUs comprise one or more general purpose GPUs (GPGPUs) programmed with instructions to execute compute unified device architecture (CUDA).