Frame packing for asymmetric stereo video

• US 9,225,961 B2
• Filed: 02/25/2011
• Issued: 12/29/2015
• Est. Priority Date: 05/13/2010
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Chen, Ying, Karczewicz, Marta
• Primary Examiner(s): TORRENTE, RICHARD T
• Application Number:
  US13/035,448
• Publication Number:
  US 20110280316A1
• Time in Patent Office: 1,768 Days
• Field of Search: 375/240.25
• US Class Current: 1/1
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 19/523 with sub-pixel accuracy
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards
• CPC Class Codes: H04N 19/82 involving filtering within a prediction loop

Abstract

An asymmetric frame of a coded video bitstream may include a full resolution picture of a left view and a reduced resolution picture of a right view, where the left and right views form a stereo view pair for three-dimensional video playback. In one example, an apparatus includes a video encoder configured to receive a first picture of a first view of a scene having a first resolution, receive a second picture of a second view of the scene having a reduced resolution relative to the first resolution, form an asymmetric frame including the first picture and the second picture, and encode the asymmetric frame. In this manner, decoders of varying capabilities may receive the same bitstream, and the bitstream may consume less bandwidth than one or more bitstreams having full resolution pictures of a stereo view pair. The bitstream may have better quality than a bitstream having subsampled pictures.

Claims

1. A method of encoding video data, the method comprising:

receiving a first picture of a first view of a scene having a first resolution;

receiving a second picture of a second view of the scene having a reduced resolution relative to the first resolution;

forming an asymmetric frame comprising the first picture and the second picture, wherein the asymmetric frame comprises the first picture and the second picture as a single frame;

encoding the asymmetric frame, wherein encoding the asymmetric frame comprises encoding the single frame;

forming a supplemental enhancement information (SEI) message comprising an asymmetric packing indicator value that indicates that the asymmetric frame is an asymmetric frame, a frame packing arrangement type value indicating that the first picture and the second picture have one of a side-by-side arrangement and a top-bottom arrangement, and information indicating a location of the first picture relative to the second picture in the asymmetric frame;

encapsulating the asymmetric frame in a video coding layer (VCL) network abstraction layer (NAL) unit;

encapsulating the SEI message in a non-VCL NAL unit;

encapsulating the VCL NAL unit including the asymmetric frame and the non-VCL NAL unit including the SEI message in an access unit, wherein the access unit corresponds to a specific time instance; and

outputting the access unit.

2. The method of claim 1, wherein the reduced resolution of the second picture is one-half of the first resolution of the first picture, and wherein forming the SEI message comprises forming the SEI message to indicate that the second picture in the asymmetric frame comprises one-half of the first resolution of the first picture.

3. The method of claim 1, wherein forming the asymmetric frame comprises arranging the first picture and the second picture in a side-by-side arrangement, and wherein forming the SEI message comprises forming the SEI message to indicate that the asymmetric frame is formed in the side-by-side arrangement and that the first picture and the second picture have the same height.

4. The method of claim 1, wherein asymmetric packed pictures form a frame in a top-bottom arrangement, and wherein forming the SEI message comprises forming the SEI message to indicate that the asymmetric frame is formed in the top-bottom arrangement and that the first picture and the second picture have the same width.

5. The method of claim 1, wherein encapsulating the asymmetric frame comprises forming at least one of a file conforming to ISO base media file format, a file conforming to an extension of the ISO base media file format, data conforming to an MPEG-2 transport stream, header information for an MPEG-2 transport stream, or data in accordance with hypertext transfer protocol (HTTP) streaming format.

6. The method of claim 1, wherein the first resolution comprises a first spatial resolution defined by a first horizontal resolution and a first vertical resolution, wherein the reduced resolution is defined by a second horizontal resolution and a second vertical resolution, and wherein either the second horizontal resolution is equal to the first horizontal resolution or the second vertical resolution is equal to the first vertical resolution.

7. An apparatus for encoding video data, the apparatus comprising:

a memory configured to store video data and

a video encoder configured to;

receive a first picture of a first view of a scene having a first resolution, receive a second picture of a second view of the scene having a reduced resolution relative to the first resolution,

form an asymmetric frame of the video data, the asymmetric frame comprising the first picture and the second picture, wherein the asymmetric frame comprises the first picture and the second picture as a single frame,

encode the asymmetric frame, wherein to encode the asymmetric frame, the video encoder is configured to encode the single frame,

form a supplemental enhancement information (SEI) message comprising an asymmetric packing indicator value that indicates that the asymmetric frame is an asymmetric frame, a frame packing arrangement type value indicating that the first picture and the second picture have one of a side-by-side arrangement and a top-bottom arrangement, and information indicating a location of the first picture relative to the second picture in the asymmetric frame,

encapsulate the asymmetric frame in a video coding layer (VCL) network abstraction layer (NAL) unit,

encapsulate the SEI message in a non-VCL NAL unit, and

encapsulate the VCL NAL unit including the asymmetric frame and the non-VCL NAL unit including the SEI message in an access unit, wherein the access unit corresponds to a specific time instance.

8. The apparatus of claim 7, wherein the reduced resolution of the second picture is one-half of the first resolution of the first picture, and wherein the video encoder is configured to form the SEI message to indicate that the second picture in the asymmetric frame comprises one-half of the first resolution of the first picture.

9. The apparatus of claim 7, wherein the video encoder is configured to arrange the first picture and the second picture in a side-by-side arrangement in the asymmetric frame, and to form the SEI message to indicate that the asymmetric frame is formed in the side-by-side arrangement and that the first picture and the second picture have the same height.

10. The apparatus of claim 7, wherein the video encoder is configured to arrange the first picture and the second picture in a top-bottom arrangement in the asymmetric frame, and to form the SEI message to indicate that the asymmetric frame is formed in the top-bottom arrangement and that the first picture and the second picture have the same width.

11. The apparatus of claim 7, wherein the video encoder is configured to encapsulate the encoded asymmetric frame as at least one of a file conforming to ISO base media file format, a file conforming to an extension of the ISO base media file format, data conforming to an MPEG-2 transport stream, header information for an MPEG-2 transport stream, or data in accordance with hypertext transfer protocol (HTTP) streaming format.

12. The apparatus of claim 7, wherein the apparatus comprises at least one of:

an integrated circuit;

a microprocessor;

or

a wireless communication device that includes the video encoder.

13. The apparatus of claim 7, wherein the first resolution comprises a first spatial resolution defined by a first horizontal resolution and a first vertical resolution, wherein the reduced resolution is defined by a second horizontal resolution and a second vertical resolution, and wherein either the second horizontal resolution is equal to the first horizontal resolution or the second vertical resolution is equal to the first vertical resolution.

14. An apparatus for encoding video data, the apparatus comprising:

means for receiving a first picture of a first view of a scene having a first resolution;

means for receiving a second picture of a second view of the scene having a reduced resolution relative to the first resolution;

means for forming an asymmetric frame comprising the first picture and the second picture, wherein the asymmetric frame comprises the first picture and the second picture as a single frame;

means for encoding the asymmetric frame, wherein the means for encoding the asymmetric frame comprises means for encoding the single frame;

means for forming a supplemental enhancement information (SEI) message comprising an asymmetric packing indicator value that indicates that the asymmetric frame is an asymmetric frame, a frame packing arrangement type value indicating that the first picture and the second picture have one of a side-by-side arrangement and a top-bottom arrangement, and information indicating a location of the first picture relative to the second picture in the asymmetric frame;

means for encapsulating the asymmetric frame in a video coding layer (VCL) network abstraction layer (NAL) unit;

means for encapsulating the SEI message in a non-VCL NAL unit; and

means for encapsulating the VCL NAL unit including the asymmetric frame and the non-VCL NAL unit including the SEI message in an access unit, wherein the access unit corresponds to a specific time instance.

15. The apparatus of claim 14, wherein the reduced resolution of the second picture is one-half of the first resolution of the first picture, and wherein the means for forming the SEI message comprises means for forming the SEI message to indicate that the second picture in the asymmetric frame comprises one-half of the first resolution of the first picture.

16. The apparatus of claim 14, wherein the means for forming the asymmetric frame comprises means for arranging the first picture and the second picture in a side-by-side arrangement in the asymmetric frame, and wherein the means for forming the SEI message comprises means for forming the SEI message to indicate that the asymmetric frame is formed in the side-by-side arrangement and that the first picture and the second picture have the same height.

17. The apparatus of claim 14, wherein the means for forming the asymmetric frame comprises means for arranging the first picture and the second picture in a top-bottom arrangement in the asymmetric frame, and wherein the means for forming the SEI message comprises means for forming the SEI message to indicate that the asymmetric frame is formed in the top-bottom arrangement and that the first picture and the second picture have the same width.

18. The apparatus of claim 14, wherein the first resolution comprises a first spatial resolution defined by a first horizontal resolution and a first vertical resolution, wherein the reduced resolution is defined by a second horizontal resolution and a second vertical resolution, and wherein either the second horizontal resolution is equal to the first horizontal resolution or the second vertical resolution is equal to the first vertical resolution.

19. A computer program product comprising a non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause a processor of a device for encoding video data to:

receive a first picture of a first view of a scene having a first resolution;

receive a second picture of a second view of the scene having a reduced resolution relative to the first resolution;

form an asymmetric frame comprising the first picture and the second picture, wherein the asymmetric frame comprises the first picture and the second picture as a single frame;

encode the asymmetric frame, wherein the instructions that cause the processor to encode the asymmetric frame comprise instructions that cause the processor to encode the single frame;

form a supplemental enhancement information (SEI) message comprising an asymmetric packing indicator value that indicates that the asymmetric frame is an asymmetric frame, a frame packing arrangement type value indicating that the first picture and the second picture have one of a side-by-side arrangement and a top-bottom arrangement, and information indicating a location of the first picture relative to the second picture in the asymmetric frame;

encapsulate the asymmetric frame in a video coding layer (VCL) network abstraction layer (NAL) unit;

encapsulate the SEI message in a non-VCL NAL unit;

encapsulate the VCL NAL unit including the asymmetric frame and the non-VCL NAL unit including the SEI message in an access unit, wherein the access unit corresponds to a specific time instance; and

output the access unit.

20. The computer program product of claim 19, wherein the reduced resolution of the second picture is one-half of the first resolution of the first picture, wherein the instructions that cause the processor to form the SEI message comprise instructions that cause the processor to form the SEI message to indicate that the second picture in the asymmetric frame comprises one-half of the first resolution of the first picture.

21. The computer program product of claim 19, wherein the instructions that cause the processor to form the asymmetric frame comprise instructions that cause the processor to arrange the first picture and the second picture in a side-by-side arrangement, and wherein the instructions that cause the processor to form the SEI message comprise instructions that cause the processor to form the SEI message to indicate that the asymmetric frame is formed in the side-by-side arrangement and that the first picture and the second picture have the same height.

22. The computer program product of claim 19, wherein the instructions that cause the processor to form the asymmetric frame comprise instructions that cause the processor to arrange the first picture and the second picture in a top-bottom arrangement, and wherein the instructions that cause the processor to form the SEI message comprise instructions that cause the processor to form the SEI message to indicate that the asymmetric frame is formed in the top-bottom arrangement and that the first picture and the second picture have the same width.

23. The computer program product of claim 19, wherein the instructions that cause the processor to encapsulate the encoded asymmetric frame comprise instructions that cause the processor to output at least one of a file conforming to ISO base media file format, a file conforming to an extension of the ISO base media file format, data conforming to an MPEG-2 transport stream, header information for an MPEG-2 transport stream, or data in accordance with hypertext transfer protocol (HTTP) streaming format.

24. The computer program product of claim 19, wherein the first resolution comprises a first spatial resolution defined by a first horizontal resolution and a first vertical resolution, wherein the reduced resolution is defined by a second horizontal resolution and a second vertical resolution, and wherein either the second horizontal resolution is equal to the first horizontal resolution or the second vertical resolution is equal to the first vertical resolution.

25. A method of decoding video data, the method comprising:

receiving an access unit corresponding to a specific time instance, the access unit comprising an encoded asymmetric frame, encapsulated in a video coding layer (VCL) network abstraction layer (NAL) unit, the encoded asymmetric frame comprising a first resolution picture of a first view of a scene and a reduced resolution picture of a second view of the scene, the reduced resolution picture having a reduced resolution relative to a first resolution of the first resolution picture, the asymmetric frame comprising the first resolution picture and the reduced resolution picture as a single frame, the access unit further comprising a supplemental enhancement information (SEI) message, encapsulated in a non-VCL NAL unit, the SEI message comprising an asymmetric packing indicator value that indicates that the asymmetric frame is an asymmetric frame, a frame packing arrangement type value indicating that the first resolution picture and the reduced resolution picture have one of a side-by-side arrangement and a top-bottom arrangement, and information indicating a location of the first picture relative to the second picture in the asymmetric frame;

decoding the asymmetric frame based on the SEI message, wherein decoding the asymmetric frame comprises decoding the single frame;

separating the decoded asymmetric frame into the first resolution picture and the reduced resolution picture based on the SEI message;

upsampling the reduced resolution picture to produce a second picture of the second view of the scene having the first resolution of the first resolution picture; and

outputting the first resolution picture, as a first picture, and the second picture, wherein the first picture and the second picture form a stereo image pair.

26. The method of claim 25, wherein the SEI message includes information indicating that the asymmetric frame has one of a top-bottom packing arrangement and a side-by-side packing arrangement.

27. The method of claim 26, wherein, when the SEI message indicates that the asymmetric frame has the top-bottom packing arrangement, separating the decoded asymmetric frame using the SEI message includes separating the asymmetric frame into a top portion and a bottom portion, wherein the top portion corresponds to the first resolution picture, and wherein the bottom portion corresponds to the reduced resolution picture.

28. The method of claim 26, wherein when the SEI message indicates that the asymmetric frame has the side-by-side packing arrangement, separating the decoded asymmetric frame includes separating the asymmetric frame into a left portion and a right portion, wherein the left portion corresponds to the first resolution picture, and wherein the right portion corresponds to the reduced resolution picture.

29. The method of claim 25, wherein receiving the access unit comprising the asymmetric frame comprises receiving the access unit comprising the asymmetric frame via a high definition multimedia interface (HDMI) configured to receive asymmetric frames.

30. The method of claim 25, wherein the first resolution comprises a first spatial resolution defined by a first horizontal resolution and a first vertical resolution, wherein the reduced resolution is defined by a second horizontal resolution and a second vertical resolution, and wherein either the second horizontal resolution is equal to the first horizontal resolution or the second vertical resolution is equal to the first vertical resolution.

31. An apparatus for decoding video data, the apparatus comprising:

a memory configured to store video data and

a video decoder configured to;

receive an access unit of the video data, the access unit corresponding to a specific time instance, the access unit comprising an encoded asymmetric frame encapsulated in a video coding layer (VCL) network abstraction layer (NAL) unit of the video data, the encoded asymmetric frame comprising a first resolution picture of a first view of a scene and a reduced resolution picture of a second view of the scene, the reduced resolution picture having a reduced resolution relative to a first resolution of the first resolution picture, the asymmetric frame comprising the first resolution picture and the reduced resolution picture as a single frame, the access unit further comprising a supplemental enhancement information (SEI) message, encapsulated in a non-VCL NAL unit, the SEI message comprising an asymmetric packing indicator value that indicates that the asymmetric frame is an asymmetric frame, a frame packing arrangement type value indicating that the first resolution picture and the reduced resolution picture have one of a side-by-side arrangement and a top-bottom arrangement, and information indicating a location of the first resolution picture relative to the reduced resolution picture in the asymmetric frame,

decode the asymmetric frame based on the SEI message, wherein to decode the asymmetric frame, the video decoder is configured to decode the single frame, and wherein the video decoder is further configured to;

separate the decoded asymmetric frame into the first resolution picture and the reduced resolution picture based on the SEI message, and

upsample the reduced resolution picture to produce a second picture of the second view of the scene having the first resolution the first resolution picture, wherein the first resolution picture, when output as a first picture, and the second picture form a stereo image pair.

32. The apparatus of claim 31, further comprising a three-dimensional video display configured to display the first picture and the second picture at substantially the same time to display three-dimensional video data.

33. The apparatus of claim 31, wherein the SEI message includes information indicating that the asymmetric frame has one of a top-bottom packing arrangement and a side-by-side packing arrangement.

34. The apparatus of claim 33, wherein the video decoder is configured to separate the decoded asymmetric frame into a top portion and a bottom portion when the SEI message indicates that the asymmetric frame has the top-bottom packing arrangement, wherein the top portion corresponds to the first resolution picture, and wherein the bottom portion corresponds to the reduced resolution picture.

35. The apparatus of claim 33, wherein the video decoder is configured to separate the decoded asymmetric frame into a left portion and a right portion when the SEI message indicates that the asymmetric frame has the side-by-side packing arrangement, wherein the left portion corresponds to the first resolution picture, and wherein the right portion corresponds to the reduced resolution picture.

36. The apparatus of claim 31, further comprising a high definition multimedia interface (HDMI) configured to receive asymmetric frames and further configured to provide the asymmetric frames to the video decoder.

37. The apparatus of claim 31, wherein the apparatus comprises at least one of:

an integrated circuit;

a microprocessor;

or

a wireless communication device that includes the video decoder.

38. The apparatus of claim 31, wherein the first resolution comprises a first spatial resolution defined by a first horizontal resolution and a first vertical resolution, wherein the reduced resolution is defined by a second horizontal resolution and a second vertical resolution, and wherein either the second horizontal resolution is equal to the first horizontal resolution or the second vertical resolution is equal to the first vertical resolution.

39. An apparatus for decoding video data, the apparatus comprising:

means for receiving an access unit corresponding to a specific time instance, the access unit comprising an encoded asymmetric frame, encapsulated in a video coding layer (VCL) network abstraction layer (NAL) unit, the encoded asymmetric frame comprising a first resolution picture of a first view of a scene and a reduced resolution picture of a second view of the scene, the reduced resolution picture having a reduced resolution relative to a first resolution of the first resolution picture, the asymmetric frame comprising the first resolution picture and the reduced resolution picture as a single frame, the access unit further comprising a supplemental enhancement information (SEI) message, encapsulated in a non-VCL NAL unit, the SEI message comprising an asymmetric packing indicator value that indicates that the asymmetric frame is an asymmetric frame, a frame packing arrangement type value indicating that the first resolution picture and the reduced resolution picture have one of a side-by-side arrangement and a top-bottom arrangement, and information indicating a location of the first resolution picture relative to the reduced resolution picture in the asymmetric frame;

means for decoding the asymmetric frame based on the SEI message, wherein the means for decoding the asymmetric frame comprises means for decoding the single frame;

means for separating the decoded asymmetric frame into the first resolution picture and the reduced resolution picture based on the SEI message; and

means for upsampling the reduced resolution picture to produce a second picture of the second view of the scene having the first resolution of the first resolution picture, wherein the first resolution picture, when output as a first picture, and the second picture form a stereo image pair.

40. The apparatus of claim 39, wherein the SEI message includes information indicating that the asymmetric frame has one of a top-bottom packing arrangement and a side-by-side packing arrangement.

41. The apparatus of claim 40, wherein the means for separating comprises means for separating the asymmetric frame into a top portion and a bottom portion, wherein the top portion corresponds to the first resolution picture, and wherein the bottom portion corresponds to the reduced resolution picture, when the SEI message indicates that the asymmetric frame has the top-bottom packing arrangement.

42. The apparatus of claim 40, wherein the means for separating comprises means for separating the asymmetric frame into a left portion and a right portion, wherein the left portion corresponds to the first resolution picture, and wherein the right portion corresponds to the reduced resolution picture, when the SEI message indicates that the asymmetric frame has the side-by-side packing arrangement.

43. The apparatus of claim 39, wherein the means for receiving the access unit comprising the asymmetric frame comprises means for receiving the access unit comprising the asymmetric frame via a high definition multimedia interface (HDMI) configured to receive asymmetric frames.

44. The apparatus of claim 39, wherein the first resolution comprises a first spatial resolution defined by a first horizontal resolution and a first vertical resolution, wherein the reduced resolution is defined by a second horizontal resolution and a second vertical resolution, and wherein either the second horizontal resolution is equal to the first horizontal resolution or the second vertical resolution is equal to the first vertical resolution.

45. A computer program product comprising a non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause a processor of a device for decoding video data to:

receive an access unit corresponding to a specific time instance, the access unit comprising an encoded asymmetric frame, encapsulated in a video coding layer (VCL) network abstraction layer (NAL) unit, the encoded asymmetric frame comprising a first resolution picture of a first view of a scene and a reduced resolution picture of a second view of the scene, the reduced resolution picture having a reduced resolution relative to a first resolution of the first resolution picture, the asymmetric frame comprising the first resolution picture and the reduced resolution picture as a single frame, the access unit further comprising a supplemental enhancement information (SEI) message, encapsulated in a non-VCL NAL unit, the SEI message comprising an asymmetric packing indicator value that indicates that the asymmetric frame is an asymmetric frame, a frame packing arrangement type value indicating that the first resolution picture and the reduced resolution picture have one of a side-by-side arrangement and a top-bottom arrangement, and information indicating a location of the first resolution picture relative to the reduced resolution picture in the asymmetric frame;

decode the asymmetric frame based on the SEI message, wherein the instructions that cause the processor to decode the asymmetric frame comprise instructions that cause the processor to decode the single frame;

separate the decoded asymmetric frame into the first resolution picture and the reduced resolution picture based on the SEI message;

upsample the reduced resolution picture to produce a second picture of the second view of the scene having the first resolution of the first resolution picture; and

output the first resolution picture, as a first picture, and the second picture, wherein the first picture and the second picture form a stereo image pair.

46. The computer program product of claim 45, wherein the SEI message includes information indicating that the asymmetric frame has one of a top-bottom packing arrangement and a side-by-side packing arrangement.

47. The computer program product of claim 46, wherein when the SEI message indicates that the asymmetric frame has the top-bottom packing arrangement, the instructions cause the processor to separate the asymmetric frame into a top portion and a bottom portion, wherein the top portion corresponds to the first resolution picture, and wherein the bottom portion corresponds to the reduced resolution picture.

48. The computer program product of claim 46, wherein when the SEI message indicates that the asymmetric frame has the side-by-side packing arrangement, the instructions cause the processor to separate the asymmetric frame into a left portion and a right portion, wherein the left portion corresponds to the first resolution picture, and wherein the right portion corresponds to the reduced resolution picture.

49. The computer program product of claim 45, wherein the instructions that cause the processor to receive the access unit comprising the asymmetric frame comprise instructions that cause the processor to receive the access unit comprising the asymmetric frame via a high definition multimedia interface (HDMI) configured to receive asymmetric frames.

50. The computer program product of claim 45, wherein the first resolution comprises a first spatial resolution defined by a first horizontal resolution and a first vertical resolution, wherein the reduced resolution is defined by a second horizontal resolution and a second vertical resolution, and wherein either the second horizontal resolution is equal to the first horizontal resolution or the second vertical resolution is equal to the first vertical resolution.