REFINED DEPTH MAP

• US 20110142138A1
• Filed: 08/06/2009
• Published: 06/16/2011
• Est. Priority Date: 08/20/2008
• Status: Active Grant

• Current Assignee: InterDigital Madison Patent Holdings (InterDigital, Inc.)
• Original Assignee: Thomson Licensing (Vantiva SA)
• Inventors: Yin, Peng, Tian, Dong, Pandit, Purvin Bibhas
• Granted Patent: US 9,179,153 B2
• Time in Patent Office: Days
• US Class Current: 375/240.24
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/162 User input
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 2213/005 Aspects relating to the "3D+depth" image format

Abstract

Various implementations are described. Several implementations relate to a refined depth map. According to one aspect, depth information for a picture in a set of pictures is accessed. Modified depth information for the picture is accessed. A refinement is determined that characterizes a difference between the depth information and the modified depth information. The refinement, and the depth information, is provided for use in processing one or more pictures in the set of pictures.

Claims

1. A method comprising:

accessing depth information for a picture in a set of pictures;

accessing modified depth information for the picture;

determining a refinement that characterizes a difference between the depth information and the modified depth information; and

providing the refinement, and the depth information, for use in processing one or more pictures in the set of pictures.

2. The method of claim 1 wherein:

the set of pictures includes a reference picture from a reference view and a target picture from a second view, the reference picture for use as a reference in encoding the target picture, the reference view being at a reference location and the second view being at a second location different from the reference location,

the picture is the reference picture, and

the refinement and the depth information are provided for use in at least one of encoding or decoding a picture in the set or rendering a further picture.

3. The method of claim 2 further comprising:

encoding the target picture based on the modified depth information; and

providing the encoded target picture along with the refinement and the depth information.

4. The method of claim 3 wherein encoding the target picture based on the modified depth information comprises:

warping the modified depth information from the reference location to the second location;

determining, based on the warped modified depth information, a correspondence between pixels in the reference picture and pixels in the target picture; and

encoding the target picture based on the correspondence.

5. The method of claim 3 wherein encoding the target picture based on the modified depth information comprises:

warping the reference picture, based on the modified depth information, from the reference location to the second location; and

encoding the target picture based on the warped reference picture.

6. The method of claim 3 further comprising:

modifying the depth information for the reference picture to produce the modified depth information, the modifying being done so as to reduce a cost function for encoding the target picture, the cost function combining rate and distortion.

7. The method of claim 6 wherein the cost function accounts for one or more of:

rate due to encoding the refinement,

rate due to encoding the target picture based on a warping of the reference picture that is derived using the modified depth information rather than the depth information, or

distortion due to encoding the target picture based on a warping of the reference picture that is derived using the modified depth information rather than the depth information.

8. The method of claim 6 wherein modifying the depth information comprises iteratively modifying the depth information as long as the cost function at each iteration satisfies one or more criteria.

9. The method of claim 8, wherein the one or more criteria comprise one or more of the following:

the cost function is not more than a threshold amount lower than a last cost function;

)

the cost function is not below a threshold; and

the cost function has not gone up, or has not gone up more than a threshold number of times, or has not gone up more than a threshold amount or percentage.

10. The method of claim 1 further comprising modifying the depth information for the picture to produce the modified depth information.

11. The method of claim 1 wherein providing the refinement comprises encoding the refinement and providing the encoded refinement.

12. The method of claim 11 wherein encoding the refinement comprises predictively encoding the refinement using a previous refinement as a reference.

13. The method of claim 1 wherein determining the refinement comprises setting the refinement equal to the difference between the depth information and the modified depth information.

14. The method of claim 1 wherein the picture is a depth map that includes the depth information.

15. The method of claim 2 wherein the set of pictures includes a further target picture that is from a third view, the reference picture also for use as a reference in encoding the further target picture, the third view being at a third location that is different from the reference location and the second location, and the method further comprises:

accessing a third-view modified depth information for the picture;

determining a third-view refinement that characterizes a difference between the depth information and the third-view modified depth information; and

providing the third-view refinement, and the depth information, for use in processing one or more pictures in the set of pictures.

16. The method of claim 15 further comprising:

encoding the further target picture based on the third-view modified depth information; and

providing the encoded further target picture along with the third-view refinement and the depth information.

17. The method of claim 16 wherein encoding the further target picture based on the third-view modified depth information comprises:

warping the reference picture, based on the third-view modified depth information, from the reference location to the third location; and

encoding the further target picture based on the third-view warped reference picture.

18. The method of claim 1 wherein:

the refinement include separate information at one of more levels of a hierarchy for the picture, the hierarchy including a pixel level, a partition level, a macroblock level, a slice level, and a picture level.

19. The method of claim 18 wherein the refinement includes a separate value for each pixel represented by the refinement.

20. The method of claim 18 wherein the refinement includes a separate value for each macroblock represented by the refinement, and the separate value for a given macroblock applies to each pixel within the given macroblock.

21. The method of claim 1 wherein the depth information for the picture, the modified depth information for the picture, and the refinement apply only to a portion of the picture.

22. An apparatus comprising:

means for accessing depth information for a picture in a set of pictures;

means for accessing modified depth information for the picture;

means for determining a refinement that characterizes a difference between the depth information and the modified depth information; and

means for providing the refinement, and the depth information, for use in processing one or more pictures in the set of pictures.

23. A processor readable medium having stored thereon instructions for causing a processor to perform at least the following:

accessing depth information for a picture in a set of pictures;

accessing modified depth information for the picture;

determining a refinement that characterizes a difference between the depth information and the modified depth information; and

providing the refinement, and the depth information, for use in processing one or more pictures in the set of pictures.

24. An apparatus, comprising a processor configured to perform at least the following:

accessing depth information for a picture in a set of pictures;

accessing modified depth information for the picture;

determining a refinement that characterizes a difference between the depth information and the modified depth information; and

providing the refinement, and the depth information, for use in processing one or more pictures in the set of pictures.

25. An apparatus comprising:

an accessing unit for accessing depth information for a picture in a set of pictures, and modified depth information for the picture; and

a depth refiner for determining a refinement that characterizes a difference between the depth information and the modified depth information, and providing the refinement, and the depth information, for use in processing one or more pictures in the set of pictures.

26. The apparatus of claim 25 wherein the apparatus is implemented in a video encoder.

27. An apparatus comprising:

an accessing unit for accessing depth information for a picture in a set of pictures, and modified depth information for the picture;

a depth refiner for determining a refinement that characterizes a difference between the depth information and the modified depth information, and providing the refinement, and the depth information, for use in processing one or more pictures in the set of pictures; and

a modulator for modulating a signal, the signal including the refinement and the depth information.

28. A method comprising:

accessing depth information for a picture in a set of pictures;

accessing a refinement that characterizes a difference between the depth information for the picture and modified depth information for the picture;

determining the modified depth information for the picture; and

processing one or more pictures in the set of pictures using the modified depth information.

29. The method of claim 28 wherein:

the set of pictures includes a reference picture from a reference view and a target picture from a second view, the reference picture for use as a reference in decoding the target picture, the reference view being at a reference location and the second view being at a second location different from the reference location,

the picture is the reference picture, and

the depth information and the modified depth information are used in at least one of decoding a picture in the set or rendering a further picture.

30. The method of claim 29 further comprising:

accessing an encoding of the target picture;

decoding the accessed encoding of the target picture based on the modified depth information; and

providing the decoded target picture for processing or display.

31. The method of claim 30 wherein decoding the accessed encoding of the target picture based on the modified depth information comprises:

accessing decoded reference pictures;

warping the modified depth information from the reference location to the second location;

determining, based on the warped modified depth information, a correspondence between pixels in the decoded reference picture and pixels locations in the target picture; and

decoding the accessed encoding of the target picture based on the correspondence.

32. The method of claim 30 wherein decoding the accessed encoding of the target picture based on the modified depth information comprises:

accessing decoded reference pictures;

warping the decoded reference picture, based on the modified depth information, from the reference location to the second location; and

decoding the accessed encoding of the target picture based on the warped reference picture.

33. The method of claim 32 wherein the modified depth information reduces, as compared to the depth information, a cost function for encoding the target picture, the cost function combining rate and distortion.

34. The method of claim 33 wherein the cost function accounts for one or more of:

rate due to encoding the refinement,

rate due to encoding the target picture based on a warping of the reference picture that is derived using the modified depth information rather than the depth information, and

distortion due to encoding the target picture based on a warping of the reference picture that is derived using the modified depth information rather than the depth information.

35. The method of claim 28 wherein the accessed refinement comprises an encoded refinement.

36. The method of claim 35 wherein the encoded refinement has been predictively encoded using a previous refinement as a reference.

37. The method of claim 28 wherein the accessed refinement is equal to the difference between the depth information and the modified depth information.

38. The method of claim 28 wherein the picture is a depth map that includes the depth information.

39. The method of claim 28 wherein the set of pictures includes a further target picture that is from a third view, the reference picture also has been used as a reference in encoding the further target picture, the third view being at a third location that is different from the reference location and the second location, and the method further comprises:

accessing a third-view refinement that characterizes a difference between the depth information and a third-view modified depth information;

determining the third-view modified depth information for the picture; and

processing one or more pictures in the set of pictures using the third-view modified depth information.

40. The method of claim 39 further comprising:

decoding the further target picture based on the third-view modified depth information; and

providing the decoded further target picture for processing or display.

41. The method of claim 40 wherein decoding the further target picture based on the third-view modified depth information comprises:

warping the reference picture, based on the third-view modified depth information, from the reference location to the third location; and

decoding the further target picture based on the third-view warped reference picture.

42. The method of claim 28 wherein:

the refinement include separate information at one of more levels of a hierarchy for the picture, the hierarchy including a pixel level, a partition level, a macroblock level, a slice level, and a picture level.

43. The method of claim 42 wherein the refinement includes a separate value for each pixel represented by the refinement.

44. The method of claim 42 wherein the refinement includes a separate value for each macroblock represented by the refinement, and the separate value for a given macroblock applies to each pixel within the given macroblock.

45. The method of claim 28 wherein the depth information for the picture, the modified depth information for the picture, and the refinement apply only to a portion of the picture.

46. The method of claim 28 further comprising:

rendering a picture from a virtual view using the reference picture and the depth information.

47. An apparatus comprising:

means for accessing depth information for a picture in a set of pictures;

means for accessing a refinement that characterizes a difference between, the depth information for the picture and modified depth information for the picture;

means for determining the modified depth information for the picture; and

means for processing one or more pictures in the set of pictures using the modified depth information.

48. A processor readable medium having stored thereon instructions for causing a processor to perform at least the following:

accessing depth information for a picture in a set of pictures;

accessing a refinement that characterizes a difference between the depth information for the picture and modified depth information for the picture;

determining the modified depth information for the picture; and

processing one or more pictures in the set of pictures using the modified depth information.

49. An apparatus, comprising a processor configured to perform at least the following:

accessing depth information for a picture in a set of pictures;

accessing a refinement that characterizes a difference between the depth information for the picture and modified depth information for the picture;

determining the modified depth information for the picture; and

processing one or more pictures in the set of pictures using the modified depth information.

50. An apparatus comprising:

an accessing unit for accessing depth information for a picture in a set of pictures, and a refinement that characterizes a difference between the depth information for the picture and modified depth information for the picture; and

a depth refiner for determining the modified depth information for the picture, and for processing one or more pictures in the set of pictures using the modified depth information.

51. The apparatus of claim 50 wherein the apparatus is implemented in at least one of a video encoder and a video decoder.

52. An apparatus comprising:

a demodulator for receiving and demodulating a signal, the signal including coded video information for a set of pictures, depth information for a picture in the set of pictures, and a refinement that characterizes a difference between the depth information for the picture and modified depth information for the picture; and

an accessing unit for accessing depth information for a picture in a set of pictures, and a refinement that characterizes a difference between the depth information for the picture and modified depth information for the picture; and

a depth refiner for determining the modified depth information for the picture, and for processing one or more pictures in the set of pictures using the modified depth information.

53. A video signal formatted to include information, the video signal comprising:

a depth refinement portion including a refinement that characterizes a difference between depth information for a picture and modified depth information for the picture.

54. A video signal structure comprising:

a depth refinement portion including a refinement that characterizes a difference between depth information for a picture and modified depth information for the picture.

55. A processor readable medium having stored thereon a video signal structure, comprising:

a depth refinement portion including a refinement that characterizes a difference between depth information for a picture and modified depth information for the picture.

56. The processor readable medium of claim 55 wherein the video signal structure stored on the processor-readable medium further includes:

a picture portion including an encoding of the picture.

57. The processor readable medium of claim 55 wherein the video signal structure stored on the processor-readable medium further includes:

a depth information portion including the depth information for the picture, wherein the depth information and the refinement together allow a determination of the modified depth information for the picture.

58. The processor readable medium of claim 57 wherein:

the depth information in the video signal structure is useful for rendering, based on the picture, one or more additional pictures for viewing by a user, and the modified depth information, that is based on the refinement and the depth information in the video signal structure, is useful for decoding an encoding of the picture.