System for executing 3D propagation for depth image-based rendering

• US 9,654,765 B2
• Filed: 02/03/2014
• Issued: 05/16/2017
• Est. Priority Date: 11/18/2009
• Status: Active Grant

• Current Assignee: Board of Trustees of The University of Illinois
• Original Assignee: Board of Trustees of The University of Illinois
• Inventors: Nguyen, Quang H, Do, Minh N, Patel, Sanjay J
• Primary Examiner(s): Kim, Hee-Yong
• Application Number:
  US14/171,141
• Publication Number:
  US 20140293010A1
• Time in Patent Office: 1,198 Days
• Field of Search: 348 47
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2207/20028 Bilateral filtering
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/30 Image reproducers optical systems for producing stereoscopic or other three-dimensional effects G02B30/00
• CPC Class Codes: H04N 2213/003 Aspects relating to the "2D+depth" image format

Abstract

A system is disclosed for executing depth image-based rendering of a 3D image by a computer having a processor and that is coupled with one or more color cameras and at least one depth camera. The color cameras and the depth camera are positionable at different arbitrary locations relative to a scene to be rendered. In some examples, the depth camera is a low resolution camera and the color cameras are high resolution. The processor is programmed to propagate depth information from the depth camera to an image plane of each color camera to produce a propagated depth image at each respective color camera, to enhance the propagated depth image at each color camera with the color and propagated depth information thereof to produce corresponding enhanced depth images, and to render a complete, viewable image from one or more enhanced depth images from the color cameras. The processor may be a graphics processing unit.

Claims

1. A method for executing depth image-based rendering of a 3D image from one or more color cameras and at least one depth camera by at least one computer having a processor and memory, the method comprising:

propagating, using the at least one computer, depth information from the at least one depth camera to an image plane of each color camera to produce a propagated depth image for each color camera;

enhancing, using the at least one computer, the propagated depth image for each color camera with the color and propagated depth information thereof to produce an enhanced depth image for each color camera, wherein, for the propagated depth image for each color camera, the enhancing comprises (i) depth-color bilateral filtering (DCBF) and (ii) filling holes along a specific direction that is determined based on a line originating from an epipole; and

creating, using the at least one computer, a complete image from the one or more enhanced depth images.

2. The method of claim 1, wherein, for the propagated depth image for each color camera, the enhancing further comprises:

detection of occluded pixels in the propagated depth images;

occlusion removal to replace values of the occluded pixels with newly interpolated values; and

depth edge enhancement to sharpen depth edges surrounding objects in the propagated depth images,

wherein the DCBF combines color and propagated depth information to calculate values for unknown depth pixels in the propagated depth image while preserving edges of an object in the scene; and

wherein the filling of holes along the specific direction comprises filling areas of unknown depth pixels caused by disocclusion in the propagated depth image.

3. The method of claim 2, wherein the complete image is rendered at a virtual view position, and wherein rendering comprises:

propagating depth and color information from each enhanced depth image to a second target view defined by a virtual camera to produce the rendered image through 3D warping, wherein the enhanced depth images of the color cameras are merged to the virtual camera in which the color cameras are second reference views;

detecting and removing occlusions from the rendered image; and

processing the rendered image with a median filter to fill and denoise the rendered image.

4. The method of claim 2, wherein the occlusion removal comprises, for a plurality of known depth pixels in the propagated depth image:

selecting a known depth pixel A;

partitioning a support window around pixel A into fourths; and

replacing the value of pixel A by a newly interpolated value if there are at least three partitions each having at least one pixel B such that the depth difference between pixels A and B is greater than a predetermined threshold value.

5. The method of claim 2, wherein the depth edge enhancement comprises:

computing depth gradients in vertical, horizontal, and at least one diagonal direction with Sobel operators;

for each pixel P whose depth edge gradients are greater than a predetermined value, searching within a search window among the neighbor pixels for a best color-matched pixel that has a smallest Euclidian color distance with pixel P; and

replacing the depth value of the pixel P by the depth value of the best color-matched pixel.

6. The method of claim 1, wherein DCBF is used to fill holes along the specific direction.

7. The method of claim 1, further comprising:

enhancing, using the at least one computer, the complete image to produce an enhanced complete image; and

rendering, using the at least one computer, the enhanced complete image viewable on a display.

8. A system for executing depth image-based rendering of a 3D image from one or more color cameras and at least one depth camera, the system comprising:

one or more color cameras and the at least one depth camera connected to at least one processor and memory and instructions that, when executed, cause the at least one processor to;

propagate depth information from the at least one depth camera to an image plane of each color camera to produce a propagated depth image for each color camera;

enhance the propagated depth image for each color camera with the color and propagated depth information thereof to produce an enhanced depth image for each color camera, wherein, for the propagated depth image for each color camera, the enhancing comprises (i) depth-color bilateral filtering (DCBF) and (ii) filling holes along a specific direction that is determined based on a line originating from an epipole; and

create a complete image from the one or more enhanced depth images.

9. The system of claim 8, wherein, for the propagated depth image for each color camera, the enhancing further comprises:

detection of occluded pixels in the propagated depth images;

occlusion removal to replace values of the occluded pixels with newly interpolated values; and

depth edge enhancement to sharpen depth edges surrounding objects in the propagated depth images,

wherein the DCBF combines color and propagated depth information to calculate values for unknown depth pixels in the propagated depth image while preserving edges of an object in the scene; and

wherein the filling of holes along the specific direction comprises filling areas of unknown depth pixels caused by disocclusion in the propagated depth image.

10. The system of claim 9, wherein the instructions, when executed, further cause the at least one processor to:

render the complete image at a virtual view position at least in part by;

propagating depth and color information from each enhanced depth image to a second target view defined by a virtual camera to produce the rendered image through 3D warping, wherein the enhanced depth images of the color cameras are merged to the virtual camera in which the color cameras are second reference views;

detecting and removing occlusions from the rendered image; and

processing the rendered image with a median filter to fill and denoise the rendered image.

11. The system of claim 9, wherein the occlusion removal comprises, for a plurality of known depth pixels in the propagated depth image:

selecting a known depth pixel A;

partitioning a support window around pixel A into fourths; and

replacing the value of pixel A by a newly interpolated value if there are at least three partitions each having at least one pixel B such that the depth difference between pixels A and B is greater than a predetermined threshold value.

12. The system of claim 9, wherein the depth edge enhancement comprises:

computing depth gradients in vertical, horizontal, and at least one diagonal direction with Sobel operators;

for each pixel P whose depth edge gradients are greater than a predetermined value, searching within a search window among the neighbor pixels for a best color-matched pixel that has a smallest Euclidian color distance with pixel P; and

replacing the depth value of the pixel P by the depth value of the best color-matched pixel.

13. The system of claim 8, wherein DCBF is used to fill holes along the specific direction.

14. The system of claim 8, further comprising instructions that, when executed, cause the at least one processor to:

enhance the complete image to produce an enhanced complete image; and

render the enhanced complete image viewable on a display.

15. A non-transitory computer-readable medium comprising a set of instructions for executing depth image-based rendering of a 3D image from one or more color cameras and at least one depth camera by at least one processor and memory, wherein the instructions, when executed, cause the at least one processor to:

propagate depth information from the at least one depth camera to an image plane of each color camera to produce a propagated depth image for each color camera;

enhance the propagated depth image for each color camera with the color and propagated depth information thereof to produce an enhanced depth image for each color camera, wherein, for the propagated depth image for each color camera, the enhancing comprises (i) depth-color bilateral filtering (DCBF) and (ii) filling holes along a specific direction that is determined based on a line originating from an epipole; and

create a complete image from the one or more enhanced depth images.

16. The non-transitory computer-readable medium of claim 15, wherein DCBF is used to fill holes along the specific direction.

17. The non-transitory computer-readable medium of claim 15, further comprising instructions that, when executed, cause the at least one processor to:

enhance the complete image to produce an enhanced complete image; and

render the enhanced complete image viewable on a display.