PROVIDING FRAME PACKING TYPE INFORMATION FOR VIDEO CODING

• US 20120020413A1
• Filed: 04/07/2011
• Published: 01/26/2012
• Est. Priority Date: 07/21/2010
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Chen, Ying, Karczewicz, Marta, Wang, Yong
• Granted Patent: US 9,596,447 B2
• Time in Patent Office: Days
• US Class Current: 375/240.26
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards
• CPC Class Codes: H04N 21/816 involving special video data, e.g 3D video
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 21/85406 involving a specific file format, e.g. MP4 format

Abstract

This disclosure includes techniques for signaling characteristics of a representation of multimedia content at a representation layer, such as frame packing arrangement information for the representation. In one example, an apparatus for receiving video data includes a processing unit configured to receive information indicative of whether a bitstream includes a packed frame of video data, wherein the packed frame comprises two frames corresponding to different views of a scene for three-dimensional playback, and wherein the information is present in a representation layer external to a codec layer of the bitstream, automatically determine whether the apparatus is capable of decoding and rendering the bitstream based on an analysis of the received information and decoding and rendering capabilities of the device, and retrieve the bitstream when the processing unit determines that the device is capable of decoding and rendering the bitstream.

Claims

1. A method of providing video data, the method comprising:

receiving a packed frame of video data, wherein the packed frame comprises two frames corresponding to different views of a scene for three-dimensional playback; and

providing information at a representation layer to a client device, wherein the information indicates a frame packing arrangement for the packed frame.

2. The method of claim 1, wherein providing the information comprises providing the information in response to a request from the client device for representation layer data.

3. The method of claim 1, wherein providing the information comprises:

forming a media presentation description data structure that describes three-dimensional video characteristics of one or more representations of multimedia content, at least one of the representations comprising the packed frame; and

sending the media presentation description data structure to the client device.

4. The method of claim 1, further comprising:

receiving a request for data from a representation of multimedia content, the representation comprising the packed frame, from the client device when the client device is configured to decode packed frames having the frame packing arrangement of the packed frame; and

in response to the request, sending the data to the client device.

5. The method of claim 1, further comprising sending information at a codec layer for a representation of multimedia content, the representation comprising the packed frame, to the client device, wherein the information comprises a frame packing arrangement supplemental enhancement information (SEI) message, and wherein the frame packing arrangement SEI message is separate from the information provided at the representation layer.

6. The method of claim 1, further comprising providing an indication of an aspect sub-sampling ratio for each of the views.

7. The method of claim 6, wherein providing the indication of the aspect sub-sampling ratio comprises including information for the indication in at least one of a codec layer, a file format layer in a media file container format, and a transport layer.

8. The method of claim 7, wherein the transport layer comprises one of MPEG-2 Transport Stream and HTTP streaming.

9. The method of claim 6, wherein the indication of the aspect sub-sampling ratio comprises an indication that the views have an asymmetric spatial resolution.

10. The method of claim 1, wherein providing the information at the representation layer comprises sending the information to the client device in accordance with a streaming network protocol.

11. An apparatus for providing video data, the apparatus comprising a processing unit configured to receive a packed frame of video data, wherein the packed frame comprises two frames corresponding to different views of a scene for three-dimensional playback, and provide information at a representation layer to a client device, wherein the information indicates a frame packing arrangement for the packed frame.

12. The apparatus of claim 11, wherein the processing unit is configured to provide the information in response to a request from the client device for representation layer data.

13. The apparatus of claim 11, wherein to provide the information, the processing unit is configured to form a media presentation description data structure that describes three-dimensional video characteristics of one or more representations of multimedia content, at least one of the representations comprising the packed frame, and send the media presentation description data structure to the client device.

14. The apparatus of claim 11, wherein the processing unit is further configured to receive a request for data from a representation of multimedia content, the representation comprising the packed frame, from the client device when the client device is configured to decode packed frames having the frame packing arrangement of the packed frame, and, in response to the request, send the data to the client device.

15. The apparatus of claim 11, wherein the processing unit is configured to send information at a codec layer for a representation of multimedia content, the representation comprising the packed frame, to the client device, wherein the information comprises a frame packing arrangement supplemental enhancement information (SEI) message, and wherein the frame packing arrangement SEI message is separate from the information provided at the representation layer.

16. The apparatus of claim 11, wherein the apparatus comprises at least one of:

an integrated circuit;

a microprocessor; and

a wireless communication device that includes the processing unit.

17. An apparatus for providing video data, the apparatus comprising:

means for receiving a packed frame of video data, wherein the packed frame comprises two frames corresponding to different views of a scene for three-dimensional playback; and

means for providing information at a representation layer to a client device, wherein the information indicates a frame packing arrangement for the packed frame.

18. The apparatus of claim 17, wherein the means for providing the information comprises means for providing the information in response to a request from the client device for representation layer data.

19. The apparatus of claim 17, wherein the means for providing the information comprises:

means for forming a media presentation description data structure that describes three-dimensional video characteristics of one or more representations of multimedia content, at least one of the representations comprising the packed frame; and

means for sending the media presentation description data structure to the client device.

20. The apparatus of claim 17, further comprising:

means for receiving a request for data from a representation of multimedia content, the representation comprising the packed frame, from the client device when the client device is configured to decode packed frames having the frame packing arrangement of the packed frame; and

means for sending the data to the client device in response to the request.

21. The apparatus of claim 17, further comprising means for sending information at a codec layer for a representation of multimedia content, the representation comprising the packed frame, to the client device, wherein the information comprises a frame packing arrangement supplemental enhancement information (SEI) message, and wherein the frame packing arrangement SEI message is separate from the information provided at the representation layer.

22. A computer program product comprising a computer-readable storage medium having stored thereon instructions that, when executed, cause a processor of a device for providing video data to:

receive a packed frame of video data, wherein the packed frame comprises two frames corresponding to different views of a scene for three-dimensional playback; and

provide information at a representation layer to a client device, wherein the information indicates a frame packing arrangement for the packed frame.

23. The computer program product of claim 22, wherein the instructions that cause the processor to provide the information comprise instructions that cause the processor to provide the information in response to a request from the client device for representation layer data.

24. The computer program product of claim 22, wherein the instructions that cause the processor to provide the information comprise instructions that cause the processor to:

form a media presentation description data structure that describes three-dimensional video characteristics of one or more representations of multimedia content, at least one of the representations comprising the packed frame; and

send the media presentation description data structure to the client device.

25. The computer program product of claim 22, further comprising instructions that cause the processor to:

receive a request for data from a representation of multimedia content, the representation comprising the packed frame, from the client device when the client device is configured to decode packed frames having the frame packing arrangement of the packed frame; and

in response to the request, send the data to the client device.

26. The computer program product of claim 22, further comprising instructions that cause the processor to send information at a codec layer for a representation of multimedia content, the representation comprising the packed frame, to the client device, wherein the information comprises a frame packing arrangement supplemental enhancement information (SEI) message, and wherein the frame packing arrangement SEI message is separate from the information provided at the representation layer.

27. A method of retrieving video data, the method comprising:

receiving information indicative of whether a bitstream includes a packed frame of video data, wherein the packed frame comprises two frames corresponding to different views of a scene for three-dimensional playback, and wherein the information is present in a representation layer external to a codec layer of the bitstream;

automatically determining, by a processing unit of a device, whether the device is capable of decoding and rendering the bitstream based on an analysis of the received information by the processing unit and decoding and rendering capabilities of the device; and

retrieving the bitstream when the processing unit determines that the device is capable of decoding and rendering the bitstream.

28. The method of claim 27, wherein automatically determining comprises determining whether a video decoder of the device is configured to decode packed frames that are packed in a manner as indicated by the received information.

29. The method of claim 27, wherein the bitstream comprises a first bitstream, the method further comprising, when the processing unit determines that the device is not capable of decoding and rendering three-dimensional video data, retrieving a second bitstream, wherein the second bitstream comprises full resolution frames of video data for two-dimensional playback, and wherein the first bitstream and the second bitstream correspond to the same multimedia content.

30. The method of claim 29, further comprising, prior to retrieving the second bitstream, receiving information indicating that the second bitstream includes the full resolution frames of video data for two-dimensional playback, wherein the information is present in a representation layer external to a codec layer of the second bitstream.

31. The method of claim 27, wherein the bitstream comprises a first bitstream, the method further comprising, when the processing unit determines that the device is not capable of decoding and rendering the first bitstream due to a frame packing type of the first bitstream, retrieving a second bitstream, wherein the second bitstream comprises a packed frame comprising two frames corresponding to different views of a scene for three-dimensional playback, and wherein the two frames of the packed frame have a frame packing type supported by the device.

32. The method of claim 31, further comprising, prior to retrieving the second bitstream, receiving information indicating that the second bitstream includes the packed frame having the supported frame packing type.

33. The method of claim 27, further comprising decoding the bitstream with a video decoder of the device, wherein the video decoder is separate from the processing unit, wherein decoding the bitstream comprises using second information indicative of whether the bitstream includes the packed frame, and wherein the second information is present in the codec layer of the bitstream.

34. The method of claim 27, further comprising receiving an indication of an aspect sub-sampling ratio for each of the views.

35. The method of claim 34, wherein receiving the indication of the aspect sub-sampling ratio comprises receiving information for the indication in at least one of a codec layer, a file format layer in a media file container format, and a transport layer.

36. The method of claim 34, wherein the indication of the aspect sub-sampling ratio comprises an indication that the views have an asymmetric spatial resolution.

37. An apparatus for retrieving video data, the apparatus comprising a processing unit configured to receive information indicative of whether a bitstream includes a packed frame of video data, wherein the packed frame comprises two frames corresponding to different views of a scene for three-dimensional playback, and wherein the information is present in a representation layer external to a codec layer of the bitstream, automatically determine whether the apparatus is capable of decoding and rendering the bitstream based on an analysis of the received information and decoding and rendering capabilities of the device, and retrieve the bitstream when the processing unit determines that the device is capable of decoding and rendering the bitstream.

38. The apparatus of claim 37, further comprising a video decoder, wherein the processing unit is configured to automatically determine whether the video decoder is configured to decode packed frames that are packed in a manner as indicated by the received information.

39. The apparatus of claim 37, wherein the bitstream comprises a first bitstream, and wherein the processing unit is configured to, after determining that the device is not capable of decoding and rendering three-dimensional video data, retrieve a second bitstream, wherein the second bitstream comprises full resolution frames of video data for two-dimensional playback, and wherein the first bitstream and the second bitstream correspond to the same multimedia content.

40. The apparatus of claim 39, wherein the processing unit is configured to, prior to retrieving the second bitstream, receive information indicating that the second bitstream includes the full resolution frames of video data for two-dimensional playback, wherein the information is present in a representation layer external to a codec layer of the second bitstream.

41. The apparatus of claim 37, wherein the bitstream comprises a first bitstream, and wherein the processing unit is configured to, after determining that the device is not capable of decoding and rendering the first bitstream due to a frame packing type of the first bitstream, retrieve a second bitstream, wherein the second bitstream comprises a packed frame comprising two frames corresponding to different views of a scene for three-dimensional playback, and wherein the two frames of the packed frame have a frame packing type supported by the device.

42. The apparatus of claim 41, wherein the processing unit is configured to, prior to retrieving the second bitstream, receive information indicating that the second bitstream includes the packed frame having the supported frame packing type.

43. The apparatus of claim 37, further comprising a video decoder configured to decode the bitstream, wherein to decode the bitstream, the video decoder is configured to use second information indicative of whether the bitstream includes the packed frame, wherein the second information is present in the codec layer of the bitstream, and wherein the video decoder is separate from the processing unit.

44. The apparatus of claim 37, wherein the apparatus comprises at least one of:

an integrated circuit;

a microprocessor; and

a wireless communication device that includes the processing unit.

45. An apparatus for retrieving video data, the apparatus comprising:

means for receiving information indicative of whether a bitstream includes a packed frame of video data, wherein the packed frame comprises two frames corresponding to different views of a scene for three-dimensional playback, and wherein the information is present in a representation layer external to a codec layer of the bitstream;

means for automatically determining whether the device is capable of decoding and rendering the bitstream based on an analysis of the received information by the processing unit and decoding and rendering capabilities of the device; and

means for retrieving the bitstream when the processing unit determines that the device is capable of decoding and rendering the bitstream.

46. The apparatus of claim 45, wherein the means for automatically determining comprises means for determining whether the apparatus is configured to decode packed frames that are packed in a manner as indicated by the received information prior to retrieving the second bitstream.

47. The apparatus of claim 45, wherein the bitstream comprises a first bitstream, further comprising means for retrieving a second bitstream when the means for determining determines that the device is not capable of decoding and rendering three-dimensional video data, wherein the second bitstream comprises full resolution frames of video data for two-dimensional playback, and wherein the first bitstream and the second bitstream correspond to the same multimedia content.

48. The apparatus of claim 47, further comprising means for receiving information indicating that the second bitstream includes the full resolution frames of video data for two-dimensional playback prior to retrieving the second bitstream, wherein the information is present in a representation layer external to a codec layer of the second bitstream.

49. The apparatus of claim 45, wherein the bitstream comprises a first bitstream, further comprising means for retrieving a second bitstream when the means for determining determines that the device is not capable of decoding and rendering the first bitstream due to a frame packing type of the first bitstream, wherein the second bitstream comprises a packed frame comprising two frames corresponding to different views of a scene for three-dimensional playback, and wherein the two frames of the packed frame have a frame packing type supported by the device.

50. The apparatus of claim 49, further comprising means for receiving information indicating that the second bitstream includes the packed frame having the supported frame packing type prior to retrieving the second bitstream.

51. The apparatus of claim 45, further comprising means for decoding the bitstream with a video decoder of the device, wherein the means for decoding the bitstream comprise means for using second information indicative of whether the bitstream includes the packed frame, wherein the second information is present in the codec layer of the bitstream, and wherein the means for decoding is separate from the means for determining.

52. A computer program product comprising a computer-readable storage medium having stored thereon instructions that, when executed, cause a processor of a device for retrieving video data to:

receive information indicative of whether a bitstream includes a packed frame of video data, wherein the packed frame comprises two frames corresponding to different views of a scene for three-dimensional playback, and wherein the information is present in a representation layer external to a codec layer of the bitstream;

automatically determine whether the device is capable of decoding and rendering the bitstream based on an analysis of the received information by the processing unit and decoding and rendering capabilities of the device; and

retrieve the bitstream when the processing unit determines that the device is capable of decoding and rendering the bitstream.

53. The computer program product of claim 52, wherein the instructions that cause the processor to automatically determine comprise instructions that cause the processor to determine whether a video decoder of the device is configured to decode packed frames that are packed in a manner as indicated by the received information.

54. The computer program product of claim 52, wherein the bitstream comprises a first bitstream, further comprising instructions that cause the processor to, when the processing unit determines that the device is not capable of decoding and rendering three-dimensional video data, retrieve a second bitstream, wherein the second bitstream comprises full resolution frames of video data for two-dimensional playback, and wherein the first bitstream and the second bitstream correspond to the same multimedia content.

55. The computer program product of claim 54, further comprising instructions that cause the processor to, prior to retrieving the second bitstream, receive information indicating that the second bitstream includes the full resolution frames of video data for two-dimensional playback, wherein the information is present in a representation layer external to a codec layer of the second bitstream.

56. The computer program product of claim 52, wherein the bitstream comprises a first bitstream, further comprising instructions that cause the processor to, when the processing unit determines that the device is not capable of decoding and rendering the first bitstream due to a frame packing type of the first bitstream, retrieve a second bitstream, wherein the second bitstream comprises a packed frame comprising two frames corresponding to different views of a scene for three-dimensional playback, and wherein the two frames of the packed frame have a frame packing type supported by the device.

57. The computer program product of claim 56, further comprising instructions that cause the processor to, prior to retrieving the second bitstream, receive information indicating that the second bitstream includes the packed frame having the supported frame packing type.

58. The computer program product of claim 52, further comprising instructions that cause the processor to send the bitstream to a video decoder of the device to cause the video decoder to decode the bitstream, wherein the video decoder is separate from the processor, wherein the video decoder is configured to decode the bitstream using second information indicative of whether the bitstream includes the packed frame, and wherein the second information is present in the codec layer of the bitstream.