Frame packing for video coding

• US 20110286530A1
• Filed: 01/26/2010
• Published: 11/24/2011
• Est. Priority Date: 01/26/2009
• Status: Active Grant

• Current Assignee: Interdigital Vc Holdings, Inc. (InterDigital, Inc.)
• Original Assignee: Thomson Licensing (Vantiva SA)
• Inventors: Lai, Polin, Luo, Jiancong, Yin, Peng, Tian, Dong
• Granted Patent: US 9,036,714 B2
• Time in Patent Office: Days
• US Class Current: 375/240.25
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/513 Processing of motion vectors
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards
• CPC Class Codes: H04N 21/2365 Multiplexing of several video streams
• CPC Class Codes: H04N 21/2383 Channel coding or modulation of digital bit-stream, e.g. QPSK modulation arrangements for detecting or preventing errors in the information received by adapting the channel coding H04L1/0009; analogue front ends or means for connecting modulators, demodulators or transceivers to a transmission line H04L27/0002
• CPC Class Codes: H04N 21/4347 Demultiplexing of several video streams

Abstract

Implementations are provided that relate, for example, to view tiling in video encoding and decoding. A particular implementation accesses a video picture that includes multiple pictures combined into a single picture, and accesses additional information indicating how the multiple pictures in the accessed video picture are combined. The accessed information includes spatial interleaving information and sampling information. Another implementation encodes a video picture that includes multiple pictures combined into a single picture, and generates information indicating how the multiple pictures in the accessed video picture are combined. The generated information includes spatial interleaving information and sampling information. A bitstream is formed that includes the encoded video picture and the generated information. Another implementation provides a data structure for transmitting the generated information.

Claims

1. A method comprising:

accessing a video picture that includes multiple pictures combined into a single picture, the video picture being part of a received video stream;

accessing information that is part of the received video stream, the accessed information indicating how the multiple pictures in the accessed video picture are combined, wherein the accessed information includes spatial interleaving information and sampling information, wherein the spatial interleaving information indicates spatial interleaving applied to the multiple pictures in forming the single picture and wherein the sampling information indicates one or more parameters related to an upsampling filter for restoring each of the multiple pictures to a desired resolution, the one or more parameters related to the upsampling filter including an indication of filtering direction; and

decoding the video picture to provide a decoded representation of at least one of the multiple pictures.

2. The method of claim 1, wherein the decoding is performed based on the spatial interleaving information and the sampling information.

3. The method of claim 1, wherein the spatial interleaving information indicates spatial interleaving of a type from a group including side-by-side interleaving, top-bottom interleaving, row-by-row interleaving, column-by-column interleaving, and checkerboard interleaving.

4. The method of claim 1, wherein the spatial interleaving information further includes relationship information, the relationship information indicating whether and a type of relationship that exists between the multiple pictures.

5. The method of claim 4, wherein the decoding is further based on the relationship information.

6. The method of claim 4, wherein the relationship information indicates a relationship from the group consisting of:

the multiple pictures are left and right views of an image, the multiple pictures are stereo views of an image, the multiple views are not related, the multiple views are a 2D image and its related depth map (2D+Z), the multiple views are multiple sets of a 2D+Z (MVD), the multiple views represent images in a layer depth video format (LDV), and the multiple views represent images in two sets of LDV (DES), and the multiple views are formatted according to a depth related format.

7. The method of claim 6, wherein the accessed information further includes camera parameter information, the camera parameter information including at least one of a focal length of a camera, a baseline distance, at least one camera position, a Znear parameter to indicate a minimum distance between an image in the multiple pictures and the camera, a Zfar parameter to indicate a maximum distance between the image in the multiple pictures and the camera, a 3×

3 intrinsic matrix, a 3×

3 rotation matrix, and a 3D translation vector.

8. The method of claim 7, wherein the decoding is further based on the camera parameter information.

9. The method of claim 1, wherein the accessed information further includes orientation information, the orientation information indicating an orientation of at least one of the multiple pictures.

10. The method of claim 9, wherein the decoding is further based on the orientation information.

11. The method of claim 9, wherein the orientation information is from a group consisting of:

no picture has a flipped orientation, one picture of the multiple pictures has its orientation flipped vertically, and one picture of the multiple pictures has its orientation flipped horizontally.

12. The method of claim 11, wherein the orientation information also indicates the one or more pictures of the multiple pictures to which the orientation information pertains.

13. The method of claim 1, wherein the sampling information includes information from a group consisting of sampling indication, sampling factor, sampling filter type, a number of filter coefficients, and one or more sampling filter coefficient values.

14. The method of claim 13, wherein the sampling indication indicates whether the sampling information relates to upsampling or downsampling.

15. The method of claim 13, wherein the sampling information includes a sampling factor that indicates that sampling is performed by a factor selected from a group consisting of the values 0, 2, and 4.

16. The method of claim 13, wherein the sampling filter type indicates a type of sampling filter from the group consisting of 2-dimensional symmetrical FIR filter, an interpolation filter, a bilinear filter, a cubic filter, a spline filter, a Wiener filter, a least squares filter, a 6-tap AVC filter, a 4-tap SVC filter, a custom filter including a 2D customized filter, a 1D filter, and a nonlinear filter.

17. The method of claim 13, wherein the one or more sampling filter coefficient values are from a group consisting of upsampling filter coefficient values and downsampling filter coefficient values.

18. The method of claim 13, wherein the decoding further includes deriving a set of one or more upsampling filter coefficient values from downsampling filter coefficient values included in the accessed information.

19. The method of claim 1, wherein the information is included in at least one of a slice header, a sequence parameter set, a picture parameter set, a view parameter set, a network abstraction layer unit header, or a supplemental enhancement information message (804).

20. The method of claim 1, wherein the indication of filtering direction indicates that upsampling is to be performed in the horizontal direction.

21. The method of claim 1, wherein the indication of filtering direction indicates that upsampling is to be performed in the vertical direction.

22. The method of claim 1, wherein the indication of filtering direction indicates that upsampling is to be performed in the horizontal direction and the vertical direction.

23. The method of claim 1, further comprising providing the decoded representation and the accessed information as output for further processing of the decoded representation based on the accessed information.

24. The method of claim 1, wherein the accessed information is included in a supplemental enhancement information message of a standard such as H.264/AVC.

25. The method of claim 1, further comprising providing the decoded representation for display.

26. The method of claim 1, wherein the one or more parameters related to the upsampling filter include an indication of filter parameters used in downsampling one or more of the multiple pictures to combine the multiple pictures into the single picture.

27. The method of claim 26, wherein the indication of filter parameters used in downsampling is usable in determining upsample filter parameters for use in upsampling to provide the decoded representation.

41. The method of claim 1, wherein the indication of filtering direction indicates that upsampling is to be performed in the horizontal direction.

42. The method of claim 1, wherein the indication of filtering direction indicates that upsampling is to be performed in the vertical direction.

43. The method of claim 1, wherein the indication of filtering direction indicates that upsampling is to be performed in the horizontal direction and the vertical direction.

28. An apparatus comprising

means for accessing a video picture that includes multiple pictures combined into a single picture, the video picture being part of a received video stream;

means for accessing information that is part of the received video stream, the accessed information indicating how the multiple pictures in the accessed video picture are combined, wherein the accessed information includes spatial interleaving information and sampling information, wherein the spatial interleaving information indicates spatial interleaving applied to the multiple pictures in forming the single picture and wherein the sampling information indicates one or more parameters related to an upsampling filter for restoring each of the multiple pictures to a desired resolution, the one or more parameters related to the upsampling filter including an indication of filtering direction; and

means for decoding the video picture to provide a decoded representation of at least one of the multiple pictures.

29. An apparatus, comprising a processor configured to perform at least the following:

accessing a video picture that includes multiple pictures combined into a single picture, the video picture being part of a received video stream;

accessing information that is part of the received video stream, the accessed information indicating how the multiple pictures in the accessed video picture are combined, wherein the accessed information includes spatial interleaving information and sampling information, wherein the spatial interleaving information indicates spatial interleaving applied to the multiple pictures in forming the single picture and wherein the sampling information indicates one or more parameters related to an upsampling filter for restoring each of the multiple pictures to a desired resolution, the one or more parameters related to the upsampling filter including an indication of filtering direction;

decoding the video picture to provide a decoded representation of at least one of the multiple pictures.

30. A processor readable medium having stored therein instructions for causing a processor to perform at least the following:

accessing a video picture that includes multiple pictures combined into a single picture, the video picture being part of a received video stream;

accessing information that is part of the received video stream, the accessed information indicating how the multiple pictures in the accessed video picture are combined, wherein the accessed information includes spatial interleaving information and sampling information, wherein the spatial interleaving information indicates spatial interleaving applied to the multiple pictures in forming the single picture and wherein the sampling information indicates one or more parameters related to an upsampling filter for restoring each of the multiple pictures to a desired resolution, the one or more parameters related to the upsampling filter including an indication of filtering direction;

decoding the video picture to provide a decoded representation of at least one of the multiple pictures.

31. An apparatus comprising:

a decoder configured to access a video picture that includes multiple pictures combined into a single picture, the video picture being part of a received video stream; and

configured to access information that is part of the received video stream, the accessed information indicating how the multiple pictures in the accessed video picture are combined, wherein the accessed information includes spatial interleaving information and sampling information, wherein the spatial interleaving information indicates spatial interleaving applied to the multiple pictures in forming the single picture and wherein the sampling information indicates one or more parameters related to an upsampling filter for restoring each of the multiple pictures to a desired resolution, the one or more parameters related to the upsampling filter including an indication of filtering direction; and

configured to decode the video picture to provide a decoded representation of at least one of the multiple pictures.

32. An apparatus comprising:

a demodulator for demodulating a signal that includes a received video stream; and

a decoder configured to access a video picture that includes multiple pictures combined into a single picture, the video picture being part of a received video stream; and

configured to access information that is part of the received video stream, the accessed information indicating how the multiple pictures in the accessed video picture are combined, wherein the accessed information includes spatial interleaving information and sampling information, wherein the spatial interleaving information indicates spatial interleaving applied to the multiple pictures in forming the single picture and wherein the sampling information indicates one or more parameters related to an upsampling filter for restoring each of the multiple pictures to a desired resolution, the one or more parameters related to the upsampling filter including an indication of filtering direction; and

configured to decode the video picture to provide a decoded representation of at least one of the multiple pictures.

33. (canceled)

34. (canceled)

35. A processor readable medium having stored thereon a video signal structure, the video signal structure comprising:

an encoded picture section including an encoding of a video picture, the video picture including multiple pictures combined into a single picture; and

a signaling section including an encoding of generated information indicating how the multiple pictures in the accessed video picture are combined, wherein the generated information includes spatial interleaving information and sampling information, wherein the spatial interleaving information indicates spatial interleaving applied to the multiple pictures in forming the single picture and wherein the sampling information indicates one or more parameters related to an upsampling filter for restoring each of the multiple pictures to a desired resolution, the one or more parameters related to the upsampling filter including an indication of filtering direction, the generated information providing information for use in decoding the encoded video picture.

36. A method comprising:

encoding a video picture that includes multiple pictures combined into a single picture (826);

generating information indicating how the multiple pictures in the accessed video picture are combined, wherein the generated information includes spatial interleaving information and sampling information, wherein the spatial interleaving information indicates spatial interleaving applied to the multiple pictures in forming the single picture and wherein the sampling information indicates one or more parameters related to an upsampling filter for restoring each of the multiple pictures to a desired resolution, the one or more parameters related to the upsampling filter including an indication of filtering direction; and

forming a bitstream that includes the encoded video picture and the generated information, the generated information providing information for use in processing the encoded video picture.

37. The method of claim 36, wherein the spatial interleaving information indicates spatial interleaving of a type from a group including side-by-side interleaving, top-bottom interleaving, row-by-row interleaving, column-by-column interleaving, and checkerboard interleaving.

38. The method of claim 36, wherein the spatial interleaving information further include relationship information, the relationship information indicating whether and a type of relationship that exists between the multiple pictures;

and

wherein the relationship information is also for use in processing the encoded video picture.

39. The method of claim 36, wherein the generated information further includes orientation information, the orientation information indicating an orientation of at least one of the multiple pictures;

and

wherein the orientation information is also for use in processing the encoded video picture.

40. The method of claim 39, wherein the orientation information is from a group consisting of:

no picture has a flipped orientation, one picture of the multiple pictures has its orientation flipped vertically, and one picture of the multiple pictures has its orientation flipped horizontally.

44. An apparatus comprising

means for encoding a video picture that includes multiple pictures combined into a single picture;

means for generating information indicating how the multiple pictures in the accessed video picture are combined, wherein the generated information includes spatial interleaving information and sampling information, wherein the spatial interleaving information indicates spatial interleaving applied to the multiple pictures in forming the single picture and wherein the sampling information indicates one or more parameters related to an upsampling filter for restoring each of the multiple pictures to a desired resolution, the one or more parameters related to the upsampling filter including an indication of filtering direction; and

means for forming a bitstream that includes the encoded video picture and the generated information, the generated information providing information for use in processing the encoded video picture.

45. An apparatus, comprising a processor configured to perform at least the following:

encoding a video picture that includes multiple pictures combined into a single picture (826);

generating information indicating how the multiple pictures in the accessed video picture are combined, wherein the generated information includes spatial interleaving information and sampling information, wherein the spatial interleaving information indicates spatial interleaving applied to the multiple pictures in forming the single picture and wherein the sampling information indicates one or more parameters related to an upsampling filter for restoring each of the multiple pictures to a desired resolution, the one or more parameters related to the upsampling filter including an indication of filtering direction; and

forming a bitstream that includes the encoded video picture and the generated information, the generated information providing information for use in processing the encoded video picture.

46. A processor readable medium having stored therein instructions for causing a processor to perform at least the following:

encoding a video picture that includes multiple pictures combined into a single picture (826);

generating information indicating how the multiple pictures in the accessed video picture are combined, wherein the generated information includes spatial interleaving information and sampling information, wherein the spatial interleaving information indicates spatial interleaving applied to the multiple pictures in forming the single picture and wherein the sampling information indicates one or more parameters related to an upsampling filter for restoring each of the multiple pictures to a desired resolution, the one or more parameters related to the upsampling filter including an indication of filtering direction; and

forming a bitstream that includes the encoded video picture and the generated information, the generated information providing information for use in processing the encoded video picture.

47. An apparatus comprising:

an encoder configured to encode a video picture that includes multiple pictures combined into a single picture; and

to generate information indicating how the multiple pictures in the accessed video picture are combined, wherein the generated information includes spatial interleaving information and sampling information, wherein the spatial interleaving information indicates spatial interleaving applied to the multiple pictures in forming the single picture and wherein the sampling information indicates one or more parameters related to an upsampling filter for restoring each of the multiple pictures to a desired resolution, the one or more parameters related to the upsampling filter including an indication of filtering direction; and

to form a bitstream that includes the encoded video picture and the generated information, the generated information providing information for use in processing the encoded video picture.

48. An apparatus comprising:

an encoder configured to encode a video picture that includes multiple pictures combined into a single picture; and

to generate information indicating how the multiple pictures in the accessed video picture are combined, wherein the generated information includes spatial interleaving information and sampling information, wherein the spatial interleaving information indicates spatial interleaving applied to the multiple pictures in forming the single picture and wherein the sampling information indicates one or more parameters related to an upsampling filter for restoring each of the multiple pictures to a desired resolution, the one or more parameters related to the upsampling filter including an indication of filtering direction; and

to form a bitstream that includes the encoded video picture and the generated information, the generated information providing information for use in processing the encoded video picture; and

a modulator for modulating a signal that includes the encoded video picture.