ENCAPSULATING THREE-DIMENSIONAL VIDEO DATA IN ACCORDANCE WITH TRANSPORT PROTOCOLS

• US 20110032338A1
• Filed: 05/13/2010
• Published: 02/10/2011
• Est. Priority Date: 08/06/2009
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Soliman, Samir S., Martinez Bauza, Judit, Raveendran, Vijayalakshmi R.
• Granted Patent: US 8,878,912 B2
• Time in Patent Office: Days
• US Class Current: 348/51
• CPC Class Codes: H04N 13/10 Processing, recording or transmission of stereoscopic or multi-view image signals
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 21/2381 Adapting the multiplex stream to a specific network, e.g. an Internet Protocol [IP] network transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/4122 additional display device, e.g. video projector digital output for controlling a plurality of local displays G06F3/1423
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/43615 Interfacing a Home Network, e.g. for connecting the client to a plurality of peripherals home Audio Video Interoperability [HAVI] data switching networks H04L12/2805
• CPC Class Codes: H04N 21/4363 Adapting the video stream to a specific local network, e.g. a Bluetooth® network
• CPC Class Codes: H04N 21/43637 involving a wireless protocol, e.g. Bluetooth, RF or wireless LAN [IEEE 802.11] arrangements for wireless networking or broadcasting of information in indoor or near-field type systems H04B10/114
• CPC Class Codes: H04N 21/4381 Recovering the multiplex stream from a specific network, e.g. recovering MPEG packets from ATM cells transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/440218 by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4 conversion of standards in analogue television systems H04N7/01
• CPC Class Codes: H04N 21/44231 Monitoring of peripheral device or external card, e.g. to detect processing problems in a handheld device or the failure of an external recording device configuring of peripheral devices in general G06F9/4411; reporting information sensed by appliance or service execution status of appliance services in a home automation network H04L12/2823; monitoring the status of connected device in data switched networks H04L43/0817
• CPC Class Codes: H04N 21/6437 Real-time Transport Protocol [RTP]
• CPC Class Codes: H04N 21/816 involving special video data, e.g 3D video

Abstract

In general, techniques are described for encapsulating three dimensional video data in accordance with a transport protocol. As one example, an apparatus comprising a multimedia processing module, a transport protocol module and a wireless module implement the techniques. The multimedia processing module generates a video data segment, an audio data segment and a depth data segment of 3D video content. The transport protocol module encapsulates each of the video data, audio data and depth data segments in different ones of a plurality of packets according to a transport protocol and adds metadata to at least one of the plurality of packets for enhancing playback of the 3D video content. The wireless module transmits the packets to a 3D display device external from the apparatus.

Claims

1. An apparatus comprising:

a multimedia processing module that generates a video data segment, an audio data segment and a depth data segment of 3D video content;

a transport protocol module that encapsulates each of the video data, audio data and depth data segments in different ones of a plurality of packets according to a transport protocol and adds metadata to at least one of the plurality of packets for enhancing playback of the 3D video content; and

a wireless module that transmits the packets to a 3D display device external from the apparatus.

2. The apparatus of claim 1, wherein the transport protocol module adds the metadata within an optional data field in a header of the at least one of the plurality of packets.

3. The apparatus of claim 1,

wherein the transport protocol comprises an application layer transport protocol, and

wherein the application layer transport protocol comprises a real-time transport protocol (RTP).

4. The apparatus of claim 1, wherein the metadata comprises depth metadata defining one or more of a depth range, a depth resolution, a camera model parameter, an optimal viewing parameter and a compression type.

5. The apparatus of claim 1, wherein the metadata comprises occlusion metadata defining one or more of an occlusion resolution and a compression type.

6. The apparatus of claim 1, wherein the metadata comprises global effects metadata defining one or more of a plurality of global effects parameters, wherein global effects parameters describe video effects related to one or more of rain, haze, snow and smoke, including a density, a location and a repetition rate.

7. The apparatus of claim 1, wherein the multimedia processing module comprises a transformation module, and

the apparatus further comprising:

a parameter discovery module that determines one or more parameters that define capabilities supported by the 3D display device, wherein the one or more parameters include an input format supported by the 3D display device and configures the transformation module to transform two-dimensional (2D) video data into 3D video data so as to generate the 3D video data in accordance with the determined input format,

wherein the wireless module receive 2D video data, and

wherein the transformation module transforms the 2D video data to 3D video data so that the 3D video data complies with the determined input format.

8. The apparatus of claim 1, wherein the multimedia processing module comprises a transformation module, and

the apparatus further comprising:

a parameter discovery module that configures the transformation module to generate three-dimensional (3D) video data in accordance with parameters defining capabilities supported by the 3D display device,

wherein the transformation module transforms video data to generate the 3D video data, and

wherein the wireless module forwards the 3D video data to the 3D display device; and

a human visual system (HVS) feedback module that determines, while the transformation module transforms the video data, one or more metrics using an HVS model that reflects a quality of 3D visualization of the generated 3D video data with respect to a human visual system and, again while the transformation module transforms the video data, reconfigures the one or more modules based on the determined one or more metrics to refine the generation of the 3D video data.

9. The apparatus of claim 1, further comprising

a module to store the video data segment;

and

a wireless display host module that determines one or more display parameters of the display device external from the apparatus, and prepares the video data segment to generate 3D video data based on the determined display parameters.

10. A method comprising:

generating, with a multimedia processing module of an apparatus, a video data segment, an audio data segment and a depth data segment of 3D video content;

encapsulating, with a transport protocol module of the apparatus, each of the video data, audio data and depth data segments in different ones of a plurality of packets according to a transport protocol and adds metadata to at least one of the plurality of packets for enhancing playback of the 3D video content; and

transmitting, with a wireless module of the apparatus, the packets to a 3D display device external from the apparatus.

11. The method of claim 10, further comprising adding the metadata within an optional data field in a header of the at least one of the plurality of packets.

12. The method of claim 10,

wherein the transport protocol comprises an application layer transport protocol, and

wherein the application layer transport protocol comprises a real-time transport protocol (RTP).

13. The method of claim 10, wherein the metadata comprises depth metadata defining one or more of a depth range, a depth resolution, a camera model parameter, an optimal viewing parameter and a compression type.

14. The method of claim 10, wherein the metadata comprises occlusion metadata defining one or more of an occlusion resolution and a compression type.

15. The method of claim 10, wherein the metadata comprises global effects metadata defining one or more of a plurality of global effects parameters, wherein global effects parameters describe video effects related to one or more of rain, haze, snow and smoke, including a density, a location and a repetition rate.

16. The method of claim 10, further comprising:

determining one or more parameters that define capabilities supported by the 3D display device, wherein the one or more parameters include an input format supported by the 3D display device and configures the transformation module to transform two-dimensional (2D) video data into 3D video data so as to generate the 3D video data in accordance with the determined input format;

receiving 2D video data; and

transforming the 2D video data to 3D video data so that the 3D video data complies with the determined input format.

17. The method of claim 10, further comprising:

configuring a transformation module of the multimedia processing module to generate three-dimensional (3D) video data in accordance with parameters defining capabilities supported by the 3D display device;

transforming, with the configured transformation module, video data to generate the 3D video data,

wherein transmitting the packets comprises forwarding the 3D video data to the 3D display device; and

while the transformation module transforms the video data, determining, with a human visual system (HVS) feedback module of the apparatus, one or more metrics using an HVS model that reflects a quality of 3D visualization of the generated 3D video data with respect to a human visual system; and

while the transformation module transforms the video data, reconfiguring the transformation module based on the determined one or more metrics to refine the generation of the 3D video data.

18. The method of claim 10, further comprising

storing the video data segment;

determining one or more display parameters of the display device external from the apparatus with a wireless display host; and

preparing, with the wireless display host, the video data segment to generate 3D video data based on the determined display parameters.

19. An apparatus comprising:

means for generating a video data segment, an audio data segment and a depth data segment of 3D video content;

means for encapsulating each of the video data, audio data and depth data segments in different ones of a plurality of packets according to a transport protocol and adds metadata to at least one of the plurality of packets for enhancing playback of the 3D video content; and

means for transmitting the packets to a 3D display device external from the apparatus.

20. The apparatus of claim 19, further comprising means for adding the metadata within an optional data field in a header of the at least one of the plurality of packets.

21. The apparatus of claim 19,

wherein the transport protocol comprises an application layer transport protocol, and

wherein the application layer transport protocol comprises a real-time transport protocol (RTP).

22. The apparatus of claim 19, wherein the metadata comprises depth metadata defining one or more of a depth range, a depth resolution, a camera model parameter, an optimal viewing parameter and a compression type.

23. The apparatus of claim 19, wherein the metadata comprises occlusion metadata defining one or more of an occlusion resolution and a compression type.

24. The apparatus of claim 19, wherein the metadata comprises global effects metadata defining one or more of a plurality of global effects parameters, wherein global effects parameters describe video effects related to one or more of rain, haze, snow and smoke, including a density, a location and a repetition rate.

25. The apparatus of claim 19, further comprising:

means for determining one or more parameters that define capabilities supported by the 3D display device, wherein the one or more parameters include an input format supported by the 3D display device and configures the transformation module to transform two-dimensional (2D) video data into 3D video data so as to generate the 3D video data in accordance with the determined input format;

means for receiving 2D video data; and

means for transforming the 2D video data to 3D video data so that the 3D video data complies with the determined input format.

26. The apparatus of claim 19, further comprising:

means for configuring a transformation module of the apparatus to generate three-dimensional (3D) video data in accordance with parameters defining capabilities supported by the 3D display device,

wherein the configured transformation module transforming video data to generate the 3D video data,

wherein the means for transmitting the packets comprises means for forwarding the 3D video data to the 3D display device; and

while the transformation module transforms the video data, means for determining one or more metrics using a human visual system (HVS) model that reflects a quality of 3D visualization of the generated 3D video data with respect to a human visual system; and

while the transformation module transforms the video data, means for reconfiguring the transformation module based on the determined one or more metrics to refine the generation of the 3D video data.

27. The apparatus of claim 19, further comprising

means for storing the video data segment;

means for determining one or more display parameters of the display device external from the apparatus; and

means for preparing the video data segment to generate 3D video data based on the determined display parameters.

28. A computer-readable storage medium comprising instructions that cause a processor to:

generate a video data segment, an audio data segment and a depth data segment of 3D video content;

encapsulate each of the video data, audio data and depth data segments in different ones of a plurality of packets according to a transport protocol and adds metadata to at least one of the plurality of packets for enhancing playback of the 3D video content; and

transmit the packets to a 3D display device external from the apparatus.

29. The computer-readable storage medium of claim 28, wherein the instructions further cause the processor to add the metadata within an optional data field in a header of the at least one of the plurality of packets.

30. The computer-readable storage medium of claim 28,

wherein the transport protocol comprises an application layer transport protocol, and

wherein the application layer transport protocol comprises a real-time transport protocol (RTP).

31. The computer-readable storage medium of claim 28, wherein the metadata comprises depth metadata defining one or more of a depth range, a depth resolution, a camera model parameter, an optimal viewing parameter and a compression type.

32. The computer-readable storage medium of claim 28, wherein the metadata comprises occlusion metadata defining one or more of an occlusion resolution and a compression type.

33. The computer-readable storage medium of claim 28, wherein the metadata comprises global effects metadata defining one or more of a plurality of global effects parameters, wherein global effects parameters describe video effects related to one or more of rain, haze, snow and smoke, including a density, a location and a repetition rate.

34. The computer-readable storage medium of claim 28, wherein the instructions further cause the processor to:

determine one or more parameters that define capabilities supported by the 3D display device, wherein the one or more parameters include an input format supported by the 3D display device and configures the transformation module to transform two-dimensional (2D) video data into 3D video data so as to generate the 3D video data in accordance with the determined input format;

receive 2D video data; and

transform the 2D video data to 3D video data so that the 3D video data complies with the determined input format.

35. The computer-readable storage medium of claim 28, wherein the instructions further cause the processor to:

configure a transformation module of the multimedia processing module to generate three-dimensional (3D) video data in accordance with parameters defining capabilities supported by the 3D display device,

wherein the configured transformation module transforms video data to generate the 3D video data;

forward the 3D video data to the 3D display device; and

while the transformation module transforms the video data, determine one or more metrics using a human visual system (HVS) model that reflects a quality of 3D visualization of the generated 3D video data with respect to a human visual system; and

while the transformation module transforms the video data, reconfigure the transformation module based on the determined one or more metrics to refine the generation of the 3D video data.

36. The computer-readable storage medium of claim 28, wherein the instructions further cause the processor to:

store the video data segment;

determine one or more display parameters of the display device external from the apparatus with a wireless display host; and

prepare, with the wireless display host, the video data segment to generate 3D video data based on the determined display parameters.