Depth Map Coding to Reduce Rendered Distortion

• US 20110292043A1
• Filed: 11/23/2009
• Published: 12/01/2011
• Est. Priority Date: 02/13/2009
• Status: Active Grant

• Current Assignee: Interdigital Vc Holdings, Inc. (InterDigital, Inc.)
• Original Assignee: Thomson Licensing (Vantiva SA)
• Inventors: Lai, Po-Lin, Tian, Dong, Kim, Woo-Shik
• Granted Patent: US 9,066,075 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/158 Switching image signals
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 19/463 by compressing encoding parameters before transmission
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

Several implementations relate to depth map coding. In one implementation, a depth coding rate, that results from coding one or more portions of a depth map using a particular coding mode, is determined. The depth map can be used to render video for a different view than that of the depth map. A depth map distortion, that results from coding the one or more portions of the depth map using the particular coding mode, is determined. A value of distortion for the rendered video, based on the depth map distortion and on a particular relationship between the depth map distortion and values of distortion for the rendered video, is determined. It is determined whether to use the particular coding mode to code the one or more portions of the depth map, and the determination is based on the value of distortion for the rendered video and the depth coding rate.

Claims

1. A method comprising:

determining a depth coding rate that results from coding one or more portions of a depth map using a particular coding mode, wherein the depth map is for a particular view and the depth map can be used to render video for a different view than the particular view to produce a rendered video;

determining a depth map distortion that results from coding the one or more portions of the depth map using the particular coding mode;

determining a value of distortion for the rendered video based on the depth map distortion and based on a particular relationship between the depth map distortion and values of distortion for the rendered video; and

determining whether to use the particular coding mode to code the one or more portions of the depth map based on the value of distortion for the rendered video and the depth coding rate.

2. The method of claim 1 wherein determining the value of distortion for rendered video is performed without using the rendered video.

3. The method of claim 1 wherein the particular relationship between the depth map distortion and the value of distortion for rendered video is a linear relationship.

4. The method of claim 1, wherein the particular relationship is determined using camera parameters.

5. The method of claim 1, wherein the particular relationship is indicated by a global parameter that applies to an entire picture of video.

6. The method of claim 5, wherein the global parameter is estimated using a difference between at least a portion of the entire picture and a plurality of translated versions of at least the portion of the entire picture, to obtain a plurality of data points of pixel value distortion due to different translations.

7. The method of claim 6, further comprising processing the plurality of data points to determine an estimate of the global parameter that linearly relates pixel distortions and the different translations.

8. The method of claim 5, wherein the global parameter is scaled according to weights applied to a number of reference views used to render an intermediate view.

9. The method of claim 1, wherein the particular relationship is indicated by a local parameter that applies to one or more portions, but less than all, of a picture of video.

10. The method of claim 9, wherein the picture is partitioned into a plurality of regions based on one or more parameters, and the method further comprises estimating the local parameter for a region of the plurality of regions, the region including the one or more portions, using a difference between at least part of the region and a plurality of translated versions of at least the part of the region, to obtain a plurality of data points of pixel value distortion due to different translations.

11. The method of claim 10, further comprising processing the plurality of data points for the region to determine an estimate of the local parameter for the region, and the estimate of the local parameter linearly relates pixel distortions and the different translations for the region.

12. The method of claim 11, further comprising using the estimate of the local parameter for the region to determine a distortion for a macroblock that belongs to the region.

13. The method of claim 9, wherein the local parameter is scaled according to weights applied to a number of reference views used to render an intermediate view.

14. The method of claim 1, wherein the particular relationship is indicated, for different scenes, by a stored set of values for different types of scenes.

15. The method of claim 1, wherein at least a portion of the depth map is coded using skip mode when a corresponding picture portion to which the depth map corresponds is encoded using skip mode.

16. The method of claim 1 wherein determining whether to use the particular coding mode based on the value of distortion for the rendered video and the depth coding rate comprises determining a cost associated with the particular coding mode as a function of the value of distortion and the depth coding rate.

17. The method of claim 1 further comprising coding the one or more portions of the depth map using the particular coding mode.

18. The method of claim 1 wherein determining the value of distortion for the rendered video comprises determining the value by estimating the value.

19. An apparatus comprising:

means for determining a depth coding rate that results from coding one or more portions of a depth map using a particular coding mode, wherein the depth map is for a particular view and the depth map can be used to render video for a different view than the particular view to produce a rendered video;

means for determining a depth map distortion that results from coding the one or more portions of the depth map using the particular coding mode;

means for determining a value of distortion for the rendered video based on the depth map distortion and based on a particular relationship between the depth map distortion and values of distortion for the rendered video; and

means for determining whether to use the particular coding mode to code the one or more portions of the depth map based on the value of distortion for the rendered video and the depth coding rate.

20. The apparatus of claim 19, wherein the apparatus is implemented in a video encoder.

21. A processor readable medium having stored thereon instructions for causing a processor to perform at least the following:

determining a depth coding rate that results from coding one or more portions of a depth map using a particular coding mode, wherein the depth map is for a particular view and the depth map can be used to render video for a different view than the particular view to produce a rendered video;

determining a depth map distortion that results from coding the one or more portions of the depth map using the particular coding mode;

determining a value of distortion for the rendered video based on the depth map distortion and based on a particular relationship between the depth map distortion and values of distortion for the rendered video; and

determining whether to use the particular coding mode to code the one or more portions of the depth map based on the value of distortion for the rendered video and the depth coding rate.

22. An apparatus, comprising a processor configured to perform at least the following:

determining a depth coding rate that results from coding one or more portions of a depth map using a particular coding mode, wherein the depth map is for a particular view and the depth map can be used to render video for a different view than the particular view to produce a rendered video;

determining a depth map distortion that results from coding the one or more portions of the depth map using the particular coding mode;

determining a value of distortion for the rendered video based on the depth map distortion and based on a particular relationship between the depth map distortion and values of distortion for the rendered video; and

determining whether to use the particular coding mode to code the one or more portions of the depth map based on the value of distortion for the rendered video and the depth coding rate.

23. An apparatus comprising an encoder for performing at least the following operations:

determining a depth coding rate that results from coding one or more portions of a depth map using a particular coding mode, wherein the depth map is for a particular view and the depth map can be used to render video for a different view than the particular view to produce a rendered video;

determining a depth map distortion that results from coding the one or more portions of the depth map using the particular coding mode;

determining a value of distortion for the rendered video based on the depth map distortion and based on a particular relationship between the depth map distortion and values of distortion for the rendered video; and

determining whether to use the particular coding mode to code the one or more portions of the depth map based on the value of distortion for the rendered video and the depth coding rate.

24. An apparatus comprising:

an encoder for performing at least the following operations;

determining a depth coding rate that results from coding one or more portions of a depth map using a particular coding mode, wherein the depth map is for a particular view and the depth map can be used to render video for a different view than the particular view to produce a rendered video,

determining a depth map distortion that results from coding the one or more portions of the depth map using the particular coding mode,

determining a value of distortion for the rendered video based on the depth map distortion and based on a particular relationship between the depth map distortion and values of distortion for the rendered video,

determining whether to use the particular coding mode to code the one or more portions of the depth map based on the value of distortion for the rendered video and the depth coding rate; and

a modulator for modulating a signal that includes a coding of the one or more portions of the depth map using the particular coding mode.