DEVICE AND METHOD FOR ESTIMATIMING DEPTH MAP, AND METHOD FOR GENERATING INTERMEDIATE IMAGE AND METHOD FOR ENCODING MULTI-VIEW VIDEO USING THE SAME

• US 20090129667A1
• Filed: 10/29/2008
• Published: 05/21/2009
• Est. Priority Date: 11/16/2007
• Status: Active Grant

• Current Assignee: Gwangju Institute of Science and Technology, KT Corporation
• Original Assignee: Gwangju Institute of Science and Technology
• Inventors: Lee, Sang Beom, Oh, Kwang Jung, Ho, Yo Sung
• Granted Patent: US 8,351,685 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/61 in combination with predictive coding

Abstract

The present invention relates to a device and a method for estimating a depth map, and a method for making an intermediate image and a method for encoding multi-view video using the same. More particularly, the present invention relates to a device and a method for estimating a depth map that are capable of acquiring a depth map that reduces errors and complexity, and is resistant to external influence by dividing an area into segments on the basis of similarity, acquiring a segment-unit initial depth map by using a three-dimensional warping method and a self adaptation function to which an extended gradient map is reflected, and refining the initial depth map by performing a belief propagation method by the segment unit, and achieving smoother view conversion and improved encoding efficiency by generating an intermediate image with the depth map and utilizing the intermediate image for encoding a multi-view video, and a method for generating the intermediate image and a method for encoding the multi-view video using the same.

Claims

1. A device for estimating a depth map using multi-view images, comprising:

a multi-view image storing unit that stores the multi-view images input through a multi-view video camera;

a segment setting unit that sets a segment for pixels having a difference in the intensity between adjacent pixels is equal to or less than a threshold value in a target image where the depth map is to be estimated among the multi-view images stored in the multi-view image storing unit;

a three-dimensional warping unit that acquires a warped image by performing three-dimensional warping for the segment set through the segment setting unit in a view of a reference image adjacent to the target image; and

an initial depth value searching unit that estimate only one depth value for each segment at the time of estimating the depth value of the warped target image on the basis of the reference image.

2. The device of claim 1,

wherein the initial depth value searching unit estimates the depth value for each segment by a self adaptation function that considers a function depending on the difference in the intensity between the pixels of the segment and a function depending on a gradient value for the pixels of the segment.

3. The device of claim 2,

wherein a weighted value considering a difference between the mean intensity of the segment of the target image and the mean intensity of the segment of the reference image corresponding to the segment of the target image is applied to the self adaptation function.

4. The device of claim 2,

wherein the self adaptation function is set by additionally considering a depth value searched in an image temporally earlier than the target image.

5. The device of claim 3,

wherein the self adaptation function is set by additionally considering a depth value searched in an image temporally earlier than the target image.

6. The device of claim 1 further comprising:

a depth value refining unit that refines the depth value determined through the initial depth value searching unit by using the belief propagation method by the unit of the segment through set through the segment setting unit.

7. A method for estimating a depth map using multi-view images, comprising:

(a) storing the multi-view images input through a multi-view video camera;

(b) setting a segment for pixels having a difference in the intensity between adjacent pixels is equal to or less than a threshold value in a target image where the depth map is to be estimated among the multi-view images stored through the step (a);

(c) acquiring a warped image by performing three-dimensional warping for the segment set at step (b) on the basis of a reference image adjacent to the target image; and

(d) estimating only one depth value for each segment at the time of estimating the depth value of the warped target image on the basis of the reference image.

8. The method of claim 7,

wherein in step (d), the depth value for each segment is estimated by a self adaptation function that considers function depending on the difference in the intensity between the pixels of the segment and a function depending on a gradient value for the pixels of the segment.

9. The method of claim 8,

wherein a weighted value considering a difference between the mean intensity of the segment of the target image and the mean intensity of the segment of the reference image is applied to the self adaptation function.

10. The method of claim 8,

wherein the self adaptation function is set by additionally considering a depth value searched in an image temporally earlier than the target image.

11. The method of claim 9,

wherein the self adaptation function is set by additionally considering a depth value searched in an image temporally earlier than the target image.

12. The method of claim 8, further comprising:

using the belief propagation method by the unit of the segment set through the step (b) to refine the depth value determined through the step (d).

13. A method for generating an intermediate image,

wherein after a depth map is estimated according to claim 8, an intermediate-view image is generated by the depth map.

14. A method for encoding images of different views by using an intermediate-view image in a multi-view video, comprising:

(A) inputting a target image to be encoded in the multi-view video;

(B) adding an intermediate image generated according to claim 11 to a reference image list;

(C) determining a motion vector of the target image on the basis of a reference image of the reference image list; and

(D) acquiring a difference between the target image and the reference image motion-compensated by the motion vector determined at step (C).