Generating a depth map from a two-dimensional source image for stereoscopic and multiview imaging

• US 20070024614A1
• Filed: 07/25/2006
• Published: 02/01/2007
• Est. Priority Date: 07/26/2005
• Status: Active Grant

• Current Assignee: Her Majesty The Queen In Right of Canada As Represented By The Minister of National Defence
• Original Assignee: Her Majesty The Queen In Right of Canada As Represented By The Minister of National Defence
• Inventors: Zhang, Liang, Tam, Wa
• Granted Patent: US 8,384,763 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/564 from contours
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems

Abstract

Depth maps are generated from a monoscopic source images and asymmetrically smoothed to a near-saturation level. Each depth map contains depth values focused on edges of local regions in the source image. Each edge is defined by a predetermined image parameter having an estimated value exceeding a predefined threshold. The depth values are based on the corresponding estimated values of the image parameter. The depth map is used to process the source image by a depth image based rendering algorithm to create at least one deviated image, which forms with the source image a set of monoscopic images. At least one stereoscopic image pair is selected from such a set for use in generating different viewpoints for multiview and stereoscopic purposes, including still and moving images.

Claims

1. A method for generating a depth map containing an array of pixels, from a monoscopic source image, comprising the steps of:

(a) identifying a subset of the array of pixels representing an edge of at least one local region of the source image, the edge being defined by a predetermined image parameter having an estimated value exceeding a predefined threshold; and

(b) assigning to each pixel within said subset, a depth value based on the corresponding estimated value of the image parameter.

2. The method of claim 1, wherein the image parameter is a transition in depth.

3. The method of claim 1, wherein the image parameter is a transition in an image property selected from the group consisting of luminance, contrast, texture, and color.

4. The method of claim 1,

wherein step (a) is performed by the steps of:

determining from a finite set of scales a minimum reliable scale; and

estimating gradient magnitude for each pixel of the source image by using the minimum reliable scale; and

wherein step (b) is performed by the steps of;

recording the estimated gradient magnitude as the depth value;

partitioning the total area of the depth map into a plurality of windows of a predetermined size; and

filling the depth map in regions with missing depth values, by inserting maximum depth values within each window.

5. The method of claim 1,

wherein step (a) is performed by applying an operator to the source image to detect location of the edge, the operator having a suitably selected input threshold value to obtain a binary depth value distribution for use by step (b), the input threshold selected from an empirically pre-determined range so as to make the depth map lie between being too barren and too finely textured;

and

wherein step (b) is performed by the steps of;

amplifying the binary depth value distribution by a predetermined factor; and

expanding spatial location of each depth value by a predetermined number of pixels to increase width of the identified subset of the array of pixels representing the edge.

6. The method of claim 5, wherein the operator is a Sobel operator and the input threshold range is 0.04 to 0.10.

7. The method of claim 1, further comprising the step of using a depth image based rendering (DIBR) algorithm to create at least one deviated image by processing the source image based on the depth map, such that the source image and the deviated image form at least one stereoscopic image pair.

8. The method of claim 7, wherein the DIBR algorithm performs the steps of:

selecting a value for zero-parallax setting between nearest and farthest clipping planes of the depth map, so selected as to meet viewing preferences;

providing a depth range value and a corresponding focal length for each stereoscopic image pair; and

filling each residual vacant spot, by using an average of all neighboring pixels.

9. The method of claim 1, further comprising the steps of:

using a depth image based rendering (DIBR) algorithm to create a plurality of deviated images by processing the source image based on the depth map; and

selecting from the source image and the plurality of deviated images more than one stereoscopic image pairs, so as to give an impression of being captured from different camera positions.

10. The method of claim 1, further comprising the step of

(c) smoothing the depth map to a near-saturation level, so selected as to minimize dis-occluded regions around each edge.

11. The method of claim 10 further comprising the step of using a depth image based rendering (DIBR) algorithm to create at least one deviated image by processing the source image based on the smoothed depth map, such that the source image and the at least one deviated image form at least one stereoscopic image pair.

12. The method of claim 10, wherein step (c) uses a 2D Gaussian filter defined by a pair of parameter values for window size and standard deviation so chosen for both horizontal and vertical orientations as to determine a type of smoothing selected from the group consisting of:

i) uniform smoothing, wherein each of the parameter values is similar in the horizontal and vertical orientations;

ii) asymmetrical smoothing, wherein each of the parameter values is substantially larger in the vertical than in the horizontal orientation; and

iii) adaptive smoothing, wherein each of the parameter values follows a respective predefined function of the depth values.

13. A method for generating a smoothed depth map from a monoscopic source image, comprising the steps of:

(a) deriving a depth map from the monoscopic source image; and

(b) smoothing the depth map to a near-saturation level around an area corresponding to at least one local region of the source image defined by a change in depth exceeding a predefined threshold, so as to minimize dis-occlusions around said area, wherein range and strength of smoothing are substantially higher in the vertical than the horizontal orientation.

14. A method for creating at least one deviated image from a monoscopic source image, to form with the source image at least one stereoscopic image pair, the method comprising the steps of:

(a) deriving a sparse depth map containing an array of pixels, from a raw depth map corresponding to the source image, by

identifying a subset of the array of pixels representing an edge of at least one local region of the raw depth map, the edge being defined by a predetermined image parameter having an estimated value exceeding a predefined threshold; and

assigning to each pixel within said subset, a depth value based on the corresponding estimated value of the image parameter; and

(b) applying a depth image based rendering (DIBR) algorithm to the depth map to obtain the at least one deviated image.

15. A system for generating a stereoscopic view from a monoscopic source image, the system comprising a tandem chain of:

an edge analyzer for receiving the source image and deriving a depth map therefrom, the depth map containing depth values of at least one edge of a local region of the source image, the edge being defined by a predetermined image parameter having an estimated value exceeding a predefined threshold, wherein each depth value is based on the corresponding estimated value of the image parameter;

a DIBR processor for processing the source image based on the depth map to render at least one deviated image to form with the source image at least one stereoscopic image pair; and

a 3D display for generating at least one stereoscopic view from the at least one stereoscopic image pair.

16. The system of claim 15, further comprising an asymmetric smoother positioned between the edge analyzer and the DIBR processor, for smoothing the depth map to a near-saturation level.

17. A system for generating 3D motion pictures from a sequence of monoscopic source images, the system comprising a tandem chain of:

an edge analyzer for receiving each source image and deriving a corresponding depth map therefrom;

a DIBR processor for processing each source image with the corresponding depth map to render at least one corresponding deviated image forming with the source image at least one stereoscopic image pair; and

a 3D display device for sequentially generating at least one stereoscopic view from each stereoscopic image pair.

18. The system of claim 17,

wherein the depth map contains an array of depth values;

and

wherein the edge analyzer comprises means for identifying an edge of at least one local region of the source image, and for assigning each depth value, based on an estimated value of a predetermined image parameter defining the edge.

19. A depth map derived from a monoscopic source image, containing depth values of at least one edge of a local region of the source image, the edge being defined by a predetermined image parameter having an estimated value exceeding a predefined threshold, wherein each depth value is based on the corresponding estimated value of the image parameter.

20. The depth map of claim 19 smoothed to a near-saturation level to minimize dis-occlusions around each edge, wherein range and strength of smoothing are substantially higher in the vertical than the horizontal orientation.