CODING MULTIVIEW VIDEO PLUS DEPTH CONTENT

• US 20120229602A1
• Filed: 03/07/2012
• Published: 09/13/2012
• Est. Priority Date: 03/10/2011
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Chen, Ying, Zhang, Rong, Karczewicz, Marta
• Granted Patent: US 9,565,449 B2
• Time in Patent Office: Days
• US Class Current: 348/43
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding

Abstract

This disclosure describes techniques for coding 3D video block units. In one example, a video encoder is configured to receive one or more texture components from at least a portion of an image representing a view of three dimensional video data, receive a depth map component for at least the portion of the image, code a block unit indicative of pixels of the one or more texture components for a portion of the image and the depth map component. The coding comprises receiving texture data for a temporal instance of a view of video data, receiving depth data corresponding to the texture data for the temporal instance of the view of video data, and encapsulating the texture data and the depth data in a view component for the temporal instance of the view, such that the texture data and the depth data are encapsulated within a common bitstream.

Claims

1. A method of coding video data, the method comprising:

receiving a texture view component comprising texture data for a temporal instance of a view of video data;

receiving a depth view component comprising depth data corresponding to the texture data for the temporal instance of the view of video data; and

encapsulating the texture view component and the depth view component in a view component for the temporal instance of the view, such that the texture view component and the depth view component are encapsulated within a common bitstream.

2. The method of claim 1, wherein the texture data comprises coded slices of a coded frame of texture data and wherein the depth data comprises coded slices of a coded frame of depth data.

3. The method of claim 1, further comprising:

forming the common bitstream to include the texture view component of the view component followed consecutively by the depth view component of the view component; and

forming a delimiter network abstraction layer (NAL) unit,

wherein encapsulating comprises providing the delimiter NAL unit between the texture view component and the depth view component in an access unit of the common bitstream.

4. The method of claim 1, wherein encapsulating comprises:

encapsulating the depth view component as a network abstraction layer (NAL) unit with a NAL unit type that is different from NAL unit types used to encapsulate the texture view component.

5. The method of claim 1, further comprising signaling camera parameters for one or more cameras related to views of video data in the bitstream, wherein signaling the camera parameters comprises:

determining intrinsic camera parameters shared by the views corresponding to the one or more cameras, the intrinsic camera parameters comprising one or more of focal length and principle point offset;

determining extrinsic camera parameters shared by the views corresponding to the one or more cameras;

determining view-specific camera parameters including real-world horizontal locations of the one or more cameras; and

forming a sequence level data structure including values indicative of the intrinsic camera parameters, the extrinsic camera parameters, and the view-specific camera parameters.

6. The method of claim 1, further comprising signaling data representing conversion from real-world depth (z) values to depth values represented in the depth data as coded frames, wherein the data representing the conversion comprises data representing at least one of an indication of a linear function for converting the real-world depth (z) values to depth values of the depth view component, an indication of an inverse linear function for converting the real-world depth (z) values to depth values to depth values of the depth view component, data representing a range of the real-world depth (z) values, and an indication of a lookup table used to signal the real-word depth (z) values.

7. The method of claim 6, wherein signaling the data representing the conversion comprises forming a NAL unit comprising a unique NAL unit type value and the data representing the conversion, the method further comprising signaling a view parameter set in the common stream in one or more access units, wherein the view parameter set includes information indicative of the range of the real-world depth (z) values.

8. A device for processing video data comprising a video coder configured to receive a texture view component comprising texture data for a temporal instance of a view of video data, receive a depth view component comprising depth data corresponding to the texture data for the temporal instance of the view of video data, and encapsulate the view component texture and the depth view component in a view component for the temporal instance of the view, such that the texture view component and the depth view component are encapsulated within a common bitstream.

9. The device of claim 8, wherein the texture data comprises coded slices of a coded frame of texture data and wherein the depth data comprises coded slices of a coded frame of depth data.

10. The device of claim 8, wherein the video coder is further configured to:

form the common bitstream to include the texture view component of the view component followed consecutively by the depth view component of the view component; and

form a delimiter network abstraction layer (NAL) unit,

wherein encapsulating comprises providing the delimiter NAL unit between the texture view component and the depth view component in an access unit of the common bitstream.

11. The device of claim 8, wherein to encapsulate, the video coder is further configured to:

encapsulate the depth view component as a network abstraction layer (NAL) unit with a NAL unit type that is different from NAL unit types used to encapsulate the texture view component.

12. The device of claim 8, wherein the video coder is further configured to signal camera parameters for one or more cameras related to views of video data in the bitstream, wherein to signal the camera parameters the video coder is configured to:

determine intrinsic camera parameters shared by the views corresponding to the one or more cameras, the intrinsic camera parameters comprising one or more of focal length and principle point offset;

determine extrinsic camera parameters shared by the views corresponding to the one or more cameras;

determine view-specific camera parameters including real-world horizontal locations of the one or more cameras; and

form a sequence level data structure including values indicative of the intrinsic camera parameters, the extrinsic camera parameters, and the view-specific camera parameters.

13. The device of claim 8, further wherein the video coder is further configured to signal data representing conversion from real-world depth (z) values to depth values represented in the depth data as coded frames, wherein the data representing the conversion comprises data representing at least one of an indication of a linear function for converting the real-world depth (z) values to depth values of the depth view component, an indication of an inverse linear function for converting the real-world depth (z) values to depth values to depth values of the depth view component, data representing a range of the real-world depth (z) values, and an indication of a lookup table used to signal the real-word depth (z) values.

14. The device of claim 13, wherein to signal the data representing the conversion the video coder is configured to form a NAL unit comprising a unique NAL unit type value and the data representing the conversion, and signal a view parameter set in the common stream in one or more access units, wherein the view parameter set includes information indicative of the range of the real-world depth (z) values.

15. A computer program product comprising a computer-readable storage medium having stored thereon instructions that, when executed, cause a processor of a video encoding device to:

receive a texture view component comprising texture data for a temporal instance of a view of video data;

receive a depth view component comprising depth data corresponding to the texture data for the temporal instance of the view of video data; and

encapsulate the view component texture and the depth view component in a view component for the temporal instance of the view, such that the texture view component and the depth view component are encapsulated within a common bitstream.

16. The computer program product of claim 15, wherein the texture data comprises coded slices of a coded frame of texture data and wherein the depth data comprises coded slices of a coded frame of depth data.

17. The computer program product of claim 15, further comprising instructions that cause the processor to:

form the common bitstream to include the texture view component of the view component followed consecutively by the depth view component of the view component; and

form a delimiter network abstraction layer (NAL) unit,

wherein encapsulating comprises providing the delimiter NAL unit between the texture view component and the depth view component in an access unit of the common bitstream.

18. The computer program product of claim 15, wherein the instructions to encapsulate comprise instructions that cause the processor to:

encapsulate the depth view component as a network abstraction layer (NAL) unit with a NAL unit type that is different from NAL unit types used to encapsulate the texture view component.

19. The computer program product of claim 15, further comprising instructions that cause the processor to signal camera parameters for one or more cameras related to views of video data in the bitstream, wherein the instructions to signal the camera parameters comprise instructions that cause the processor to:

determine intrinsic camera parameters shared by the views corresponding to the one or more cameras, the intrinsic camera parameters comprising one or more of focal length and principle point offset;

determine extrinsic camera parameters shared by the views corresponding to the one or more cameras;

determine view-specific camera parameters including real-world horizontal locations of the one or more cameras; and

form a sequence level data structure including values indicative of the intrinsic camera parameters, the extrinsic camera parameters, and the view-specific camera parameters.

20. The computer program product of claim 15, further comprising instructions that cause the processor to signal data representing conversion from real-world depth (z) values to depth values represented in the depth data as coded frames, wherein the data representing the conversion comprises data representing at least one of an indication of a linear function for converting the real-world depth (z) values to depth values of the depth view component, an indication of an inverse linear function for converting the real-world depth (z) values to depth values to depth values of the depth view component, data representing a range of the real-world depth (z) values, and an indication of a lookup table used to signal the real-word depth (z) values.

21. The computer program product of claim 20, wherein the instructions that cause the processor to signal the data representing the conversion comprise instructions that cause the processor to form a NAL unit comprising a unique NAL unit type value and the data representing the conversion, and signal a view parameter set in the common stream in one or more access units, wherein the view parameter set includes information indicative of the range of the real-world depth (z) values.

22. A device for processing video data, comprising:

means for receiving a texture view component comprising texture data for a temporal instance of a view of video data;

means for receiving a depth view component comprising depth data corresponding to the texture data for the temporal instance of the view of video data; and

means for encapsulating the texture view component and the depth view component in a view component for the temporal instance of the view, such that the texture view component and the depth view component are encapsulated within a common bitstream.

23. The device of claim 22, wherein the texture data comprises coded slices of a coded frame of texture data and wherein the depth data comprises coded slices of a coded frame of depth data.

24. The device of claim 22, further comprising:

means for forming the common bitstream to include the texture view component of the view component followed consecutively by the depth view component of the view component; and

means for forming a delimiter network abstraction layer (NAL) unit,

wherein encapsulating comprises providing the delimiter NAL unit between the texture view component and the depth view component in an access unit of the common bitstream.

25. The device of claim 22, wherein the means for encapsulating comprise:

means for encapsulating the depth view component as a network abstraction layer (NAL) unit with a NAL unit type that is different from NAL unit types used to encapsulate the texture view component.

26. The device of claim 22, further comprising means for signaling camera parameters for one or more cameras related to views of video data in the bitstream, wherein the means for signaling the camera parameters comprises:

means for determining intrinsic camera parameters shared by the views corresponding to the one or more cameras, the intrinsic camera parameters comprising one or more of focal length and principle point offset;

means for determining extrinsic camera parameters shared by the views corresponding to the one or more cameras;

means for determining view-specific camera parameters including real-world horizontal locations of the one or more cameras; and

means for forming a sequence level data structure including values indicative of the intrinsic camera parameters, the extrinsic camera parameters, and the view-specific camera parameters.

27. The device of claim 22, further comprising means for signaling data representing conversion from real-world depth (z) values to depth values represented in the depth data as coded frames, wherein the data representing the conversion comprises data representing at least one of an indication of a linear function for converting the real-world depth (z) values to depth values of the depth view component, an indication of an inverse linear function for converting the real-world depth (z) values to depth values to depth values of the depth view component, data representing a range of the real-world depth (z) values, and an indication of a lookup table used to signal the real-word depth (z) values.

28. The device of claim 27, wherein the means for signaling the data representing the conversion comprises means for forming a NAL unit comprising a unique NAL unit type value and the data representing the conversion, and means for signaling a view parameter set in the common stream in one or more access units, wherein the view parameter set includes information indicative of the range of the real-world depth (z) values.

29. A method of coding video data, the method comprising:

receiving a common bitstream encapsulating a texture view component and a depth view component in a view component for a temporal instance of the view of video data, wherein the texture view component comprises texture data for the temporal instance of a view and the depth view component comprising depth data corresponding to the texture data for the temporal instance of the view of video data; and

separating the texture view component and the depth view component for the temporal instance of the view of video data.

30. The method of claim 29, wherein the texture data comprises coded slices of a coded frame of texture data and wherein the depth data comprises coded slices of a coded frame of depth data.

31. The method of claim 29, wherein the common bitstream includes a delimiter NAL unit between the texture view component and the depth view component in the common bitstream.

32. The method of claim 29, wherein the depth view component is encapsulated as a network abstraction layer (NAL) unit with a NAL unit type that is different from NAL unit types used to encapsulate the texture view component.

33. The method of claim 29, further comprising obtaining signaled camera parameters for one or more cameras related to views of video data in the bitstream, wherein the bitstream encapsulates SPS data structure including values indicative of intrinsic camera parameters shared by the views corresponding to the one or more cameras, the intrinsic camera parameters comprising one or more of focal length and principle point offset, extrinsic camera parameters shared by the views corresponding to the one or more cameras, and view-specific camera parameters including real-world horizontal locations of the one or more cameras.

34. The method of claim 29, further comprising obtaining signaled data representing conversion from real-world depth (z) values to depth values represented in the depth data as coded frames, wherein the data representing the conversion comprises data representing at least one of an indication of a linear function for converting the real-world depth (z) values to depth values of the depth view component, an indication of an inverse linear function for converting the real-world depth (z) values to depth values to depth values of the depth view component, data representing a range of the real-world depth (z) values, and an indication of a lookup table used to signal the real-word depth (z) values.

35. A device for processing video data comprising a video coder configured to receive a common bitstream encapsulating a texture view component and a depth view component in a view component for a temporal instance of the view of video data, wherein the texture view component comprises texture data for the temporal instance of a view and the depth view component comprising depth data corresponding to the texture data for the temporal instance of the view of video data, and separate the texture view component and the depth view component for the temporal instance of the view of video data.

36. The device of claim 35, wherein the texture data comprises coded slices of a coded frame of texture data and wherein the depth data comprises coded slices of a coded frame of depth data.

37. The device of claim 35, wherein the common bitstream includes a delimiter NAL unit between the texture view component and the depth view component in the common bitstream.

38. The device of claim 35, wherein the depth view component is encapsulated as a network abstraction layer (NAL) unit with a NAL unit type that is different from NAL unit types used to encapsulate the texture view component.

39. The device of claim 35, wherein the video coder is further configured to obtain signaled camera parameters for one or more cameras related to views of video data in the bitstream, wherein the bitstream encapsulates SPS data structure including values indicative of intrinsic camera parameters shared by the views corresponding to the one or more cameras, the intrinsic camera parameters comprising one or more of focal length and principle point offset, extrinsic camera parameters shared by the views corresponding to the one or more cameras, and view-specific camera parameters including real-world horizontal locations of the one or more cameras.

40. The device of claim 35, wherein the video coder is further configured to obtain signaled data representing conversion from real-world depth (z) values to depth values represented in the depth data as coded frames, wherein the data representing the conversion comprises data representing at least one of an indication of a linear function for converting the real-world depth (z) values to depth values of the depth view component, an indication of an inverse linear function for converting the real-world depth (z) values to depth values to depth values of the depth view component, data representing a range of the real-world depth (z) values, and an indication of a lookup table used to signal the real-word depth (z) values.

41. A computer program product comprising a computer-readable storage medium having stored thereon instructions that, when executed, cause a processor of a video decoding device to:

receive a common bitstream encapsulating a texture view component and a depth view component in a view component for a temporal instance of the view of video data, wherein the texture view component comprises texture data for the temporal instance of a view and the depth view component comprising depth data corresponding to the texture data for the temporal instance of the view of video data; and

separate the texture view component and the depth view component for the temporal instance of the view of video data.

42. The computer program product of claim 41, wherein the texture data comprises coded slices of a coded frame of texture data and wherein the depth data comprises code slices of a coded frame of depth data.

43. The computer program product of claim 41, wherein the common bitstream includes a delimiter NAL unit between the texture view component and the depth view component in the common bitstream.

44. The computer program product of claim 41, wherein the depth view component is encapsulated as a network abstraction layer (NAL) unit with a NAL unit type that is different from NAL unit types used to encapsulate the texture view component.

45. The computer program product of claim 41, further comprising instructions that cause the processor to obtain signaled camera parameters for one or more cameras related to views of video data in the bitstream, wherein the bitstream encapsulates SPS data structure including values indicative of intrinsic camera parameters shared by the views corresponding to the one or more cameras, the intrinsic camera parameters comprising one or more of focal length and principle point offset, extrinsic camera parameters shared by the views corresponding to the one or more cameras, and view-specific camera parameters including real-world horizontal locations of the one or more cameras.

46. The computer program product of claim 41, further comprising instructions that cause the processor to obtain signaled data representing conversion from real-world depth (z) values to depth values represented in the depth data as coded frames, wherein the data representing the conversion comprises data representing at least one of an indication of a linear function for converting the real-world depth (z) values to depth values of the depth view component, an indication of an inverse linear function for converting the real-world depth (z) values to depth values to depth values of the depth view component, data representing a range of the real-world depth (z) values, and an indication of a lookup table used to signal the real-word depth (z) values.

47. A device for processing video data comprising:

means for receiving a common bitstream encapsulating a texture view component and a depth view component in a view component for a temporal instance of the view of video data, wherein the texture view component comprises texture data for the temporal instance of a view and the depth view component comprising depth data corresponding to the texture data for the temporal instance of the view of video data; and

means for separating the texture view component and the depth view component for the temporal instance of the view of video data.

48. The device of claim 47, wherein the texture data comprises coded slices of a coded frame of texture data and wherein the depth data comprises coded slices of a coded frame of depth data.

49. The device of claim 47, wherein the common bitstream includes a delimiter NAL unit between the texture view component and the depth view component in the common bitstream.

50. The device of claim 47, wherein depth data is encapsulated as a network abstraction layer (NAL) unit with a NAL unit type that is different from NAL unit types used to encapsulate the texture data.

51. The device of claim 47, further comprising means for obtaining signaled camera parameters for one or more cameras related to views of video data in the bitstream, wherein the bitstream encapsulates SPS data structure including values indicative of intrinsic camera parameters shared by the views corresponding to the one or more cameras, the intrinsic camera parameters comprising one or more of focal length and principle point offset, extrinsic camera parameters shared by the views corresponding to the one or more cameras, and view-specific camera parameters including real-world horizontal locations of the one or more cameras.

52. The device of claim 47, further comprising means for obtaining signaled data representing conversion from real-world depth (z) values to depth values represented in the depth data as coded frames, wherein the data representing the conversion comprises data representing at least one of an indication of a linear function for converting the real-world depth (z) values to depth values of the depth view component, an indication of an inverse linear function for converting the real-world depth (z) values to depth values to depth values of the depth view component, data representing a range of the real-world depth (z) values, and an indication of a lookup table used to signal the real-word depth (z) values.