FILTERING AND EDGE ENCODING

• US 20120200669A1
• Filed: 10/14/2010
• Published: 08/09/2012
• Est. Priority Date: 10/14/2009
• Status: Active Grant

• Current Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Original Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Inventors: Liu, Shujie, Lai, Wang Lin, Tian, Dong
• Granted Patent: US 10,198,792 B2
• Time in Patent Office: Days
• US Class Current: 348/43
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20028 Bilateral filtering
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/182 the unit being a pixel
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 2213/003 Aspects relating to the "2D+depth" image format

Abstract

Several implementations relate, for example, to depth encoding and/or filtering for 3D video (3DV) coding formats. A sparse dyadic mode for partitioning macroblocks (MBs) along edges in a depth map is provided as well as techniques for trilateral (or bilateral) filtering of depth maps that may include adaptive selection between filters sensitive to changes in video intensity and/or changes in depth. One implementation partitions a depth picture, and then refines the partitions based on a corresponding image picture. Another implementation filters a portion of a depth picture based on values for a range of pixels in the portion. For a given pixel in the portion that is being filtered, the filter weights a value of a particular pixel in the range by a weight that is based on one or more of location distance, depth difference, and image difference.

Claims

1. A method comprising:

accessing a portion of a depth picture, the depth picture including information indicating depth of corresponding locations in an image picture;

partitioning the portion of the depth picture into multiple partitions;

refining the multiple partitions based on the image picture to provide refined partitions; and

encoding the portion of the depth picture based on the refined partitions to produce an encoded portion.

2. The method of claim 1 wherein refining comprises refining the multiple partitions based on intensity values of the image picture.

3. The method of claim 2 wherein refining based on intensity values comprises refining based on differences between intensity values in the image picture.

4. The method of claim 3 wherein refining based on differences comprises, for a given pixel location in the partition:

forming a similarity area for the given pixel, the similarity area including a particular pixel location if a difference is below a threshold, the difference being between (i) the intensity of the pixel in the image at the particular pixel location and (ii) the intensity of the pixel in the image that corresponds to the given pixel location;

determining the partition for which the similarity area has most overlap; and

assigning the given pixel location to the determined partitioned.

5. The method of claim 1 further comprising assigning a single separate value to each of the refined partitions as a representative value for the intensity of the pixels in the refined partition, wherein encoding comprises encoding the single separate values.

6. The method of claim 5 wherein assigning the single separate values comprises:

using the upper left value of a first of the refined partitions as the single value for the first of the refined partitions; and

using the lower right value of a second of the refined partitions as the single value for the second of the refined partitions.

7. The method of claim 5, wherein:

partitioning comprises partitioning the portion of the depth picture into multiple partitions according to one of multiple partitioning options, and encoding comprises encoding an indicator of the one of the multiple partitioning options.

8. The method of claim 1 wherein:

partitioning comprises partitioning the portion of the depth picture into multiple partitions according to one of multiple partitioning options, and encoding comprises encoding an indicator of the one of the multiple partitioning options.

9. The method of claim 8 wherein partitioning comprises selecting one of the following partitioning options:

(1) equal sized right and left halves, (2) equal sized top and bottom halves, (3) equal sized segments divided by a forward diagonal line extending between two corners of the portion, and (4) equal sized segments divided by a backward diagonal line extending between two corners of the portion.

10. The method of claim 8, wherein partitioning comprises selecting a partitioning option that minimizes a distortion function.

11. The method of claim 1 wherein accessing a portion of a depth picture comprises accessing a portion of the depth picture that represents an edge between picture regions.

12. An apparatus comprising:

means for accessing a portion of a depth picture, the depth picture including information indicating depth of corresponding locations in an image picture;

means for partitioning the portion of the depth picture into multiple partitions;

means for refining the multiple partitions based on the image picture to provide refined partitions; and

means for encoding the portion of the depth picture based on the refined partitions to produce an encoded portion.

13. A processor readable medium having stored thereon instructions for causing a processor to perform at least the following:

accessing a portion of a depth picture, the depth picture including information indicating depth of corresponding locations in an image picture;

partitioning the portion of the depth picture into multiple partitions;

refining the multiple partitions based on the image picture to provide refined partitions; and

encoding the portion of the depth picture based on the refined partitions to produce an encoded portion.

14. An apparatus, comprising a processor configured to perform at least the following:

accessing a portion of a depth picture, the depth picture including information indicating depth of corresponding locations in an image picture;

partitioning the portion of the depth picture into multiple partitions;

refining the multiple partitions based on the image picture to provide refined partitions; and

encoding the portion of the depth picture based on the refined partitions to produce an encoded portion.

15. An apparatus comprising one or more encoders configured to perform at least the following:

accessing a portion of a depth picture, the depth picture including information indicating depth of corresponding locations in an image picture;

partitioning the portion of the depth picture into multiple partitions refining the multiple partitions based on the image picture to provide refined partitions; and

encoding the portion of the depth picture based on the refined partitions to produce an encoded portion.

16. An apparatus comprising:

one or more encoders configured to perform at least the following;

accessing a portion of a depth picture, the depth picture including information indicating depth of corresponding locations in an image picture;

partitioning the portion of the depth picture into multiple partitions;

refining the multiple partitions based on the image picture to provide refined partitions; and

encoding the portion of the depth picture based on the refined partitions to produce an encoded portion; and

a modulator configured to modulate a signal including the encoded portion.

17. A method comprising

accessing a portion of an encoded depth picture, the depth picture including information indicating depth of corresponding locations in an image picture, and the encoding including an encoding of one or more value indicators indicating intensity values for multiple partitions of the portion;

identifying at least one of the multiple partitions;

refining the at least one partition based on the image picture to provide a refined partition;

decoding the encoding of the one or more value indicators; and

determining values for pixels in the refined partition based on the one or more decoded value indicators.

18. The method of claim 17 wherein:

the one or more value indicators comprises, for a refined partition, an indicator of a single value that provides a representative value for the intensity of pixels in the refined partition, and

determining values for pixels in the refined partition comprises determining the values based on the representative value.

19. The method of claim 17 wherein:

the one or more value indicators comprises, for a refined partition, multiple indicators of the intensity of pixels in the refined partition, and

determining values for pixels in the refined partition comprises determining the values based on the representative value.

20. The method of claim 17 wherein identifying at least one of the multiple partitions is based on a default partitioning.

21. The method of claim 17 wherein:

the accessed encoding further includes an encoding of a partitioning indicator indicating one of multiple partitioning options used in partitioning the portion into the multiple partitions,

the method further comprises decoding the encoding of the partitioning indicator, and

identifying at least one of the multiple partitions is based on the decoded partitioning indicator.

27. The method of claim 17 further comprising:

filtering the refined partition to produce a filtered depth portion, wherein a given pixel in the filtered depth portion is based on values for a range of pixels in the refined partition and producing the given pixel includes weighting a value of a particular pixel in the range by a weight that is based on;

(i) a distance from a location of the particular pixel to a location of a pixel in the refined partition that corresponds to the given pixel,

(ii) a depth difference between a value of the particular pixel and a value of the pixel in the refined partition that corresponds to the given pixel, and

(iii) an image difference between a value of a pixel in the image picture corresponding to the location of the particular pixel, and a value of a pixel in the image picture corresponding to the location of the given pixel.

32. The method of claim 27 wherein filtering provides the filtered depth portion with edge detail, from the corresponding portion of the image picture, that is absent from the portion.

39. The method of claim 27, wherein factor (i) is based on a function that decays exponentially as the distance between the particular pixel and the given pixel increases.

40. The method of claim 39, wherein the function is

e⁻

∥

p-q∥

/2,

where p is the location of the particular pixel and q is the location of the given pixel.

41. The method of claim 27, wherein factor (iii) is based on a function that decays exponentially as the difference between the color values of the particular pixel and the given pixel increases.

42. The method of claim 39, wherein the function is

43. The method of claim 27, wherein filtering is performed iteratively, such that at each iteration, neighboring samples are used to obtain weighted average results.

44. The method of claim 43, wherein a user can select a number of iterations and a window size that determines how many neighboring samples are considered.

22. An apparatus comprising:

means for accessing a portion of an encoded depth picture, the depth picture including information indicating depth of corresponding locations in an image picture, and the encoding including an encoding of one or more value indicators indicating intensity values for multiple partitions of the portion;

means for identifying at least one of the multiple partitions;

means for refining the at least one partition based on the image picture to provide a refined partition;

means for decoding the encoding of the one or more value indicators; and

means for determining values for pixels in the refined partition based on the one or more decoded value indicators.

23. A processor readable medium having stored thereon instructions for causing a processor to perform at least the following:

accessing a portion of an encoded depth picture, the depth picture including information indicating depth of corresponding locations in an image picture, and the encoding including an encoding of one or more value indicators indicating intensity values for multiple partitions of the portion;

identifying at least one of the multiple partitions;

refining the at least one partition based on the image picture to provide a refined partition;

decoding the encoding of the one or more value indicators; and

determining values for pixels in the refined partition based on the one or more decoded value indicators.

24. An apparatus, comprising a processor configured to perform at least the following:

accessing a portion of an encoded depth picture, the depth picture including information indicating depth of corresponding locations in an image picture, and the encoding including an encoding of one or more value indicators indicating intensity values for multiple partitions of the portion;

identifying at least one of the multiple partitions;

refining the at least one partition based on the image picture to provide a refined partition;

decoding the encoding of the one or more value indicators; and

determining values for pixels in the refined partition based on the one or more decoded value indicators.

25. An apparatus, comprising a decoder configured to perform at least the following:

accessing a portion of an encoded depth picture, the depth picture including information indicating depth of corresponding locations in an image picture, and the encoding including an encoding of one or more value indicators indicating intensity values for multiple partitions of the portion;

identifying at least one of the multiple partitions;

refining the at least one partition based on the image picture to provide a refined partition;

decoding the encoding of the one or more value indicators; and

determining values for pixels in the refined partition based on the one or more decoded value indicators.

26. An apparatus comprising:

a demodulator configured to demodulate a signal including a portion of an encoded depth picture, the depth picture including information indicating depth of corresponding locations in an image picture, and the encoding including an encoding of one or more value indicators indicating intensity values for multiple partitions of the portion; and

a decoder configured to perform at least the following;

accessing the encoding of the portion of the depth picture, identifying at least one of the multiple partitions;

refining the at least one partition based on the image picture to provide a refined partition;

decoding the encoding of the one or more value indicators; and

determining values for pixels in the refined partition based on the one or more decoded value indicators.

49. The apparatus of claim 26 further comprising a filter configured to perform at least the following:

filtering the refined partition to produce a filtered depth portion, wherein a given pixel in the filtered depth portion is based on values for a range of pixels in the refined partition and producing the given pixel includes weighting a value of a particular pixel in the range by a weight that is based on;

(i) a distance from a location of the particular pixel to a location of a pixel in the refined partition that corresponds to the given pixel,

(ii) a depth difference between a value of the particular pixel and a value of the pixel in the refined partition that corresponds to the given pixel, and

(iii) an image difference between a value of a pixel in the image picture corresponding to the location of the particular pixel, and a value of a pixel in the image picture corresponding to the location of the given pixel.

53. The apparatus of claim 49, wherein the filter is used as an in-loop filter to filter a reconstructed depth map to provide a reference image.

28. (canceled)

29. (canceled)

30. (canceled)

31. (canceled)

33. (canceled)

34. (canceled)

35. (canceled)

36. (canceled)

37. (canceled)

38. (canceled)

45. (canceled)

46. (canceled)

47. (canceled)

48. (Canceled)

50. (canceled)

51. (canceled)

52. (canceled)

54. (canceled)

55. (canceled)

56. (canceled)

57. (Canceled)

58. (Canceled)

59. (Canceled)

60. (Canceled)

61. (Canceled)

62. (Canceled)

63. (Canceled)