Providing frame packing type information for video coding

• US 9,596,447 B2
• Filed: 04/07/2011
• Issued: 03/14/2017
• Est. Priority Date: 07/21/2010
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Chen, Ying, Karczewicz, Marta, Wang, Yong
• Primary Examiner(s): Ren, Zhubing
• Application Number:
  US13/082,051
• Publication Number:
  US 20120020413A1
• Time in Patent Office: 2,168 Days
• US Class Current: 1/1
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards
• CPC Class Codes: H04N 21/816 involving special video data, e.g 3D video
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 21/85406 involving a specific file format, e.g. MP4 format

Abstract

This disclosure includes techniques for signaling characteristics of a representation of multimedia content at a representation layer, such as frame packing arrangement information for the representation. In one example, an apparatus for receiving video data includes a processing unit configured to receive information indicative of whether a bitstream includes a packed frame of video data, wherein the packed frame comprises two frames corresponding to different views of a scene for three-dimensional playback, and wherein the information is present in a representation layer external to a codec layer of the bitstream, automatically determine whether the apparatus is capable of decoding and rendering the bitstream based on an analysis of the received information and decoding and rendering capabilities of the device, and retrieve the bitstream when the processing unit determines that the device is capable of decoding and rendering the bitstream.

Claims

1. A method of retrieving video data, the method comprising:

receiving, by a client device, a manifest file for media content from a server device that is separate from the client device, wherein the manifest file describes a plurality of representations of a representation layer that is external to a codec layer for the media content, each of the representations including one or more files including video data, wherein each of the representations differs from the other representations by at least one of encoding characteristics, rendering characteristics, or bitrate and act as alternatives to each other, wherein the manifest file does not include the codec layer that includes the one or more files including the video data, wherein the manifest file does not include any of the files including the video data, wherein the representation layer is separate from and external to the codec layer such that the manifest file is separate from the one or more files including the video data, wherein the manifest file includes information indicative of whether the representations include a packed frame of video data, wherein the packed frame comprises two frames corresponding to different views of a scene for three-dimensional playback, and wherein receiving the manifest file comprises receiving the manifest file from the server device before receiving any of the files including the video data of any of the representations from the server device;

prior to requesting the video data of one of the representations that the manifest file indicates includes the packed frame, automatically determining, by a processing unit of the client device, whether the client device is capable of decoding and rendering each of the views of the video data of the one of the representations based on an analysis of the manifest file by the processing unit and decoding and rendering capabilities of the client device; and

requesting, by the client device, the video data of the one of the representations from the server device when the processing unit determines that the client device is capable of decoding and rendering each of the views of the one of the representations.

2. The method of claim 1, wherein automatically determining comprises determining whether a video decoder of the client device is configured to decode packed frames that are packed in a manner as indicated by the received information.

3. The method of claim 1, wherein the one of the representations comprises a first representation, the method further comprising, when the processing unit determines that the client device is not capable of decoding and rendering three-dimensional video data, retrieving a second representation of the representations instead of the first representation, wherein the manifest file indicates that the second representation comprises full resolution frames of video data for two-dimensional playback, and wherein the first representation and the second representation correspond to the same media content.

4. The method of claim 3, further comprising, prior to retrieving the second representation, determining that the manifest file indicates that the second representation includes the full resolution frames of video data for two-dimensional playback.

5. The method of claim 1, wherein the one of the representations comprises a first representation, the method further comprising, when the processing unit determines that the client device is not capable of decoding and rendering the first representation due to a frame packing type of the first representation, retrieving a second representation of the representations, wherein the manifest file indicates that the second representation comprises a packed frame comprising two frames corresponding to different views of a scene for three-dimensional playback, and wherein the two frames of the packed frame have a frame packing type supported by the client device.

6. The method of claim 5, further comprising, prior to retrieving the second representation, determining that the manifest file includes information indicating that the second representation includes the packed frame having the supported frame packing type.

7. The method of claim 1, further comprising decoding the video data of the one of the representations with a video decoder of the client device, wherein the video decoder is separate from the processing unit, wherein decoding the video data of the one of the representations comprises using second information indicative of whether the one of the representations includes the packed frame, and wherein the second information is present in the codec layer.

8. The method of claim 1, further comprising receiving an indication of an aspect sub-sampling ratio for each of the views.

9. The method of claim 8, wherein receiving the indication of the aspect sub-sampling ratio comprises receiving information for the indication in at least one selected from the group consisting of a codec layer, a file format layer in a media file container format, and a transport layer.

10. The method of claim 8, wherein the indication of the aspect sub-sampling ratio comprises an indication that the views have an asymmetric spatial resolution.

11. The method of claim 1, wherein the manifest file comprises a media presentation description (MPD).

12. The method of claim 1, wherein the one of the representations comprises a first representation of the plurality of representations the method further comprising, when the processing unit determines that the client device is not capable of decoding and rendering each of the views of the first representation:

selecting a second, different representation of the plurality of representations that the manifest file indicates does not include packed frames; and

retrieving video data of the second, different representation.

13. An apparatus for retrieving video data, the apparatus comprising a client device including a processing unit configured to:

receive a manifest file for media content from a server device that is separate from the client device, wherein the manifest file describes a plurality of representations of a representation layer that is external to a codec layer for the media content, each of the representations including one or more files including video data, wherein each of the representations differs from the other representations by at least one of encoding characteristics, rendering characteristics, or bitrate and act as alternatives to each other, wherein the manifest file does not include the codec layer that includes the one or more files including the video data, wherein the manifest file does not include any of the files including the video data, wherein the representation layer is separate from and external to the codec layer such that the manifest file is separate from the one or more files including the video data, wherein the manifest file includes information indicative of whether the representations include a packed frame of video data, wherein the packed frame comprises two frames corresponding to different views of a scene for three-dimensional playback, and wherein the processing unit is configured to receive the manifest file from the server device before receiving any of the files including the video data of any of the representations from the server device;

prior to requesting the video data of one of the representations that the manifest file indicates includes the packed frame, automatically determine whether the apparatus is capable of decoding and rendering each of the views of the video data of the one of the representations based on an analysis of the manifest file by the processing unit and decoding and rendering capabilities of the client device, and

request the video data of the one of the representations from the server device when the processing unit determines that the client device is capable of decoding and rendering each of the views of the one of the representations.

14. The apparatus of claim 13, further comprising a video decoder, wherein the processing unit is configured to automatically determine whether the video decoder is configured to decode packed frames that are packed in a manner as indicated by the received information.

15. The apparatus of claim 13, wherein the one of the representations comprises a first representation, and wherein the processing unit is configured to, after determining that the apparatus is not capable of decoding and rendering three-dimensional video data, retrieve a second representation of the representations instead of the first representation, wherein the manifest file indicates that the second representation comprises full resolution frames of video data for two-dimensional playback, and wherein the first representation and the second representation correspond to the same media content.

16. The apparatus of claim 15, wherein the processing unit is configured to, prior to retrieving the second representation, determine that the manifest file indicates that the second representation includes the full resolution frames of video data for two-dimensional playback.

17. The apparatus of claim 13, wherein the one of the representations comprises a first representation, and wherein the processing unit is configured to, after determining that the apparatus is not capable of decoding and rendering the first representation due to a frame packing type of the first representation, retrieve a representation of the representations, wherein the manifest file indicates that the second representation comprises a packed frame comprising two frames corresponding to different views of a scene for three-dimensional playback, and wherein the two frames of the packed frame have a frame packing type supported by the client device.

18. The apparatus of claim 17, wherein the processing unit is configured to, prior to retrieving the second representation, determine that the manifest file includes information indicating that the second representation includes the packed frame having the supported frame packing type.

19. The apparatus of claim 13, further comprising a video decoder configured to decode the video data of the one of the representations, wherein to decode the video data of the one of the representations, the video decoder is configured to use second information indicative of whether the one of the representations includes the packed frame, wherein the second information is present in the codec layer, and wherein the video decoder is separate from the processing unit.

20. The apparatus of claim 13, wherein the apparatus comprises at least one of:

an integrated circuit;

a microprocessor;

or

a wireless communication device that includes the processing unit.

21. The apparatus of claim 13, wherein the manifest file comprises a media presentation description (MPD).

22. The apparatus of claim 13, the one of the representations comprises a first representation of the plurality of representations, and wherein the processing unit is further configured to, when the processing unit determines that the apparatus is not capable of decoding and rendering each of the views of the first representation:

select a second, different representation of the plurality of representations that the manifest file indicates does not include packed frames; and

retrieve video data of the second, different representation.

23. An apparatus for retrieving video data, the apparatus including a client device comprising:

means for receiving a manifest file for media content from a server device that is separate from the client device, wherein the manifest file describes a plurality of representations of a representation layer that is external to a codec layer for the media content, each of the representations including one or more files including video data, wherein each of the representations differs from the other representations by at least one of encoding characteristics, rendering characteristics, or bitrate and act as alternatives to each other, wherein the manifest file does not include the codec layer that includes the one or more files including the video data, wherein the manifest file does not include any of the files including the video data, wherein the representation layer is separate from and external to the codec layer such that the manifest file is separate from the one or more files including the video data, wherein the manifest file includes information indicative of whether the representations include a packed frame of video data, wherein the packed frame comprises two frames corresponding to different views of a scene for three-dimensional playback, and wherein the means for receiving the manifest file comprises means for receiving the manifest file from the server device before receiving any of the files including the video data of any of the representations from the server device;

means for automatically determining, prior to requesting the video data of one of the representations that the manifest file indicates includes the packed frame, whether the apparatus is capable of decoding and rendering each of the views of the video data of the one of the representations based on an analysis of the manifest file by the processing unit and decoding and rendering capabilities of the client device; and

means for requesting the video data of the one of the representations from the server device when the processing unit determines that the client device is capable of decoding and rendering each of the views of the one of the representations.

24. The apparatus of claim 23, wherein the means for automatically determining comprises means for determining whether the apparatus is configured to decode packed frames that are packed in a manner as indicated by the received information.

25. The apparatus of claim 23, wherein the one of the representations comprises a first representation, further comprising means for retrieving a second representation of the representations instead of the first representation when the means for determining determines that the apparatus is not capable of decoding and rendering three-dimensional video data, wherein the manifest file indicates that the second representation comprises full resolution frames of video data for two-dimensional playback, and wherein the first representation and the second representation correspond to the same media content.

26. The apparatus of claim 25, further comprising means for determining that the manifest file indicates that the second representation includes the full resolution frames of video data for two-dimensional playback prior to retrieving the second representation.

27. The apparatus of claim 23, wherein the one of the representations comprises a first representation, further comprising means for retrieving a second representation when the means for determining determines that the apparatus is not capable of decoding and rendering the first representation due to a frame packing type of the first representation, wherein the manifest file indicates that the second representation comprises a packed frame comprising two frames corresponding to different views of a scene for three-dimensional playback, and wherein the two frames of the packed frame have a frame packing type supported by the apparatus.

28. The apparatus of claim 27, further comprising means for determining that the manifest file includes information indicating that the second representation includes the packed frame having the supported frame packing type prior to retrieving the second representation.

29. The apparatus of claim 23, further comprising means for decoding the video data of the one of the representations, wherein the means for decoding the video data of the one of the representations comprise means for using second information indicative of whether the one of the representations includes the packed frame, wherein the second information is present in the codec layer, and wherein the means for decoding is separate from the means for determining.

30. The apparatus of claim 23, wherein the manifest file comprises a media presentation description (MPD).

31. The apparatus of claim 23, wherein the one of the representations comprises a first representation of the plurality of representations, the apparatus further comprising:

means for selecting a second, different representation of the plurality of representations that the manifest file indicates does not include packed frames when the apparatus is not capable of decoding and rendering each of the views of the first representation; and

means for retrieving video data of the second, different representation when the apparatus is not capable of decoding and rendering each of the views of the first representation.

32. A computer program product comprising a non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause a processor of a client device for retrieving video data to:

receive a manifest file for media content from a server device that is separate from the client device, wherein the manifest file describes a plurality of representations of a representation layer that is external to a codec layer for the media content, each of the representations including one or more files including video data, wherein each of the representations differs from the other representations by at least one of encoding characteristics, rendering characteristics, or bitrate and act as alternatives to each other, wherein the manifest file does not include the codec layer that includes the one or more files including the video data, wherein the manifest file does not include any of the files including the video data, wherein the representation layer is separate from and external to the codec layer such that the manifest file is separate from the one or more files including the video data, wherein the manifest file includes information indicative of whether the representations include a packed frame of video data, wherein the packed frame comprises two frames corresponding to different views of a scene for three-dimensional playback, and wherein the instructions that cause the processor to receive the manifest file comprise instructions that cause the processor to receive the manifest file from the server device before receiving any of the files including the video data of any of the representations from the server device;

prior to requesting the video data of one of the representations that the manifest file indicates includes the packed frame, automatically determining, by a processing unit of the client device, whether the client device is capable of decoding and rendering each of the views of the video data of the one of the representations based on an analysis of the manifest file by the processing unit and decoding and rendering capabilities of the client device; and

request the video data of the one of the representations from the server device when the processing unit determines that the client device is capable of decoding and rendering each of the views of the one of the representations.

33. The computer program product of claim 32, wherein the instructions that cause the processor to automatically determine comprise instructions that cause the processor to determine whether a video decoder of the client device is configured to decode packed frames that are packed in a manner as indicated by the received information.

34. The computer program product of claim 32, wherein the one of the representations comprises a first representation, further comprising instructions that cause the processor to, when the client device is not capable of decoding and rendering three-dimensional video data, retrieve a second representation of the representations instead of the first representation, wherein the manifest file indicates that the second representation comprises full resolution frames of video data for two-dimensional playback, and wherein the first representation and the second representation correspond to the same media content.

35. The computer program product of claim 34, further comprising instructions that cause the processor to, prior to retrieving the second representation, determine that the manifest file indicates that the second representation includes the full resolution frames of video data for two-dimensional playback.

36. The computer program product of claim 32, wherein the one of the representations comprises a first representation, further comprising instructions that cause the processor to, when the client device is not capable of decoding and rendering the first representation due to a frame packing type of the first representation, retrieve a second representation of the representations, wherein the manifest file indicates that the second representation comprises a packed frame comprising two frames corresponding to different views of a scene for three-dimensional playback, and wherein the two frames of the packed frame have a frame packing type supported by the client device.

37. The computer program product of claim 36, further comprising instructions that cause the processor to, prior to retrieving the second representation, determine that the manifest file includes information indicating that the second representation includes the packed frame having the supported frame packing type.

38. The computer program product of claim 32, further comprising instructions that cause the processor to send the video data of the one of the representations to a video decoder of the client device to cause the video decoder to decode the video data of the one of the representations, wherein the video decoder is separate from the processor, wherein the video decoder is configured to decode the video data of the one of the representations using second information indicative of whether the one of the representations includes the packed frame, and wherein the second information is present in the codec layer.

39. The computer program product of claim 32, wherein the manifest file comprises a media presentation description (MPD).

40. The computer program product of claim 32, wherein the one of the representations comprises a first representation of the plurality of representations, further comprising instructions that cause the processor to, when the client device is not capable of decoding and rendering each of the views of the first representation:

select a second, different representation of the plurality of representations that the manifest file indicates does not include packed frames; and

retrieve video data of the second, different representation.