Frame packing for video coding

• US 9,420,310 B2
• Filed: 04/15/2015
• Issued: 08/16/2016
• Est. Priority Date: 01/26/2009
• Status: Active Grant

• Current Assignee: Interdigital Vc Holdings, Inc. (InterDigital, Inc.)
• Original Assignee: Thomson Licensing (Vantiva SA)
• Inventors: Lai, Polin, Tian, Dong, Luo, Jiancong, Yin, Peng
• Primary Examiner(s): Li, Tracy
• Application Number:
  US14/687,481
• Publication Number:
  US 20150222928A1
• Time in Patent Office: 489 Days
• US Class Current: 1/1
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/513 Processing of motion vectors
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards
• CPC Class Codes: H04N 21/2365 Multiplexing of several video streams
• CPC Class Codes: H04N 21/2383 Channel coding or modulation of digital bit-stream, e.g. QPSK modulation arrangements for detecting or preventing errors in the information received by adapting the channel coding H04L1/0009; analogue front ends or means for connecting modulators, demodulators or transceivers to a transmission line H04L27/0002
• CPC Class Codes: H04N 21/4347 Demultiplexing of several video streams

Abstract

Implementations are provided that relate, for example, to view tiling in video encoding and decoding. A particular implementation accesses a video picture that includes multiple pictures combined into a single picture, and accesses additional information indicating how the multiple pictures in the accessed video picture are combined. The accessed information includes spatial interleaving information and sampling information. Another implementation encodes a video picture that includes multiple pictures combined into a single picture, and generates information indicating how the multiple pictures in the accessed video picture are combined. The generated information includes spatial interleaving information and sampling information. A bitstream is formed that includes the encoded video picture and the generated information. Another implementation provides a data structure for transmitting the generated information.

Claims

1. A method, executed by a processor, comprising:

accessing a video picture that includes multiple pictures combined into a single picture, the video picture being part of a received video stream;

accessing information that is part of the received video stream, the accessed information indicating how the multiple pictures in the accessed video picture are combined,

wherein the accessed information includes spatial interleaving information and sampling information,

wherein the spatial interleaving information indicates spatial interleaving, the spatial interleaving being side-by-side interleaving, top-bottom interleaving, row-by-row interleaving, column-by-column interleaving, or checkerboard interleaving, applied to the multiple pictures in forming the single picture and wherein the sampling information indicates one or more parameters related to an upsampling filter for restoring each of the multiple pictures to another resolution, the one or more parameters related to the upsampling filter including an indication of filtering direction,

and wherein the spatial interleaving information further includes relationship information, the relationship information indicating a type of relationship that exists between the multiple pictures, the relationship information indicating that the multiple pictures are stereo views of an image, the multiple pictures are not related, the multiple pictures are a 2D image and its related depth map (2D+Z), the multiple pictures are multiple sets of a 2D+Z (MVD), the multiple pictures represent images in a layer depth video format (LDV), or the multiple pictures represent images in two sets of LDV (DES); and

decoding the video picture to provide a decoded representation of at least one of the multiple pictures.

2. The method of claim 1, wherein the decoding is further based on the relationship information.

3. The method of claim 1, wherein the accessed information further includes camera parameter information, the camera parameter information including at least one of a focal length of a camera, a baseline distance, at least one camera position, a Znear parameter to indicate a minimum distance between an image in the multiple pictures and the camera, a Zfar parameter to indicate a maximum distance between the image in the multiple pictures and the camera, a 3×

3 intrinsic matrix, a 3×

3 rotation matrix, and a 3D translation vector.

4. The method of claim 3, wherein the decoding is further based on the camera parameter information.

5. An apparatus comprising at least a memory and one or more processors, the one or more processors being configured to:

access a video picture that includes multiple pictures combined into a single picture, the video picture being part of a received video stream;

access information that is part of the received video stream, the accessed information indicating how the multiple pictures in the accessed video picture are combined,

wherein the accessed information includes spatial interleaving information and sampling information,

wherein the spatial interleaving information indicates spatial interleaving, the spatial interleaving being side-by-side interleaving, top-bottom interleaving, row-by-row interleaving, column-by-column interleaving, or checkerboard interleaving, applied to the multiple pictures in forming the single picture and wherein the sampling information indicates one or more parameters related to an upsampling filter for restoring each of the multiple pictures to another resolution, the one or more parameters related to the upsampling filter including an indication of filtering direction,

and wherein the spatial interleaving information further includes relationship information, the relationship information indicating a type of relationship that exists between the multiple pictures, the relationship information indicating that the multiple pictures are stereo views of an image, the multiple pictures are not related, the multiple pictures are a 2D image and its related depth map (2D+Z), the multiple pictures are multiple sets of a 2D+Z (MVD), the multiple pictures represent images in a layer depth video format (LDV), or the multiple pictures represent images in two sets of LDV (DES); and

decode the video picture to provide a decoded representation of at least one of the multiple pictures.

6. The apparatus of claim 5, wherein the one or more processors are configured to decode the video picture based on the relationship information.

7. The apparatus of claim 5, wherein the accessed information further includes camera parameter information, the camera parameter information including at least one of a focal length of a camera, a baseline distance, at least one camera position, a Znear parameter to indicate a minimum distance between an image in the multiple pictures and the camera, a Zfar parameter to indicate a maximum distance between the image in the multiple pictures and the camera, a 3×

3 intrinsic matrix, a 3×

3 rotation matrix, and a 3D translation vector.

8. The apparatus of claim 7, wherein the one or more processors are configured to decode the video picture based on the camera parameter information.

9. A method, executed by a processor, comprising:

encoding a video picture that includes multiple pictures combined into a single picture;

generating information indicating how the multiple pictures are combined,

wherein the generated information includes spatial interleaving information and sampling information,

wherein the spatial interleaving information indicates spatial interleaving, the spatial interleaving being side-by-side interleaving, top-bottom interleaving, row-by-row interleaving, column-by-column interleaving, or checkerboard interleaving, applied to the multiple pictures in forming the single picture and wherein the sampling information indicates one or more parameters related to an upsampling filter for restoring each of the multiple pictures to another resolution, the one or more parameters related to the upsampling filter including an indication of filtering direction,

and wherein the spatial interleaving information further includes relationship information, the relationship information indicating a type of relationship that exists between the multiple pictures, the relationship information indicating that the multiple pictures are stereo views of an image, the multiple pictures are not related, the multiple pictures are a 2D image and its related depth map (2D+Z), the multiple pictures are multiple sets of a 2D+Z (MVD), the multiple pictures represent images in a layer depth video format (LDV), or the multiple pictures represent images in two sets of LDV (DES); and

forming a bitstream that includes the encoded video picture and the generated information, the generated information providing information for use in processing the encoded video picture.

10. The method of claim 9, wherein the encoding is further based on the relationship information.

11. The method of claim 9, wherein the generated information further includes camera parameter information, the camera parameter information including at least one of a focal length of a camera, a baseline distance, at least one camera position, a Znear parameter to indicate a minimum distance between an image in the multiple pictures and the camera, a Zfar parameter to indicate a maximum distance between the image in the multiple pictures and the camera, a 3×

3 intrinsic matrix, a 3×

3 rotation matrix, and a 3D translation vector.

12. The method of claim 11, wherein the encoding is further based on the camera parameter information.

13. An apparatus comprising at least a memory and one or more processors, the one or more processors being configured to:

encode a video picture that includes multiple pictures combined into a single picture;

generate information indicating how the multiple pictures are combined,

wherein the generated information includes spatial interleaving information and sampling information,

wherein the spatial interleaving information indicates spatial interleaving, the spatial interleaving being side-by-side interleaving, top-bottom interleaving, row-by-row interleaving, column-by-column interleaving, or checkerboard interleaving, applied to the multiple pictures in forming the single picture and wherein the sampling information indicates one or more parameters related to an upsampling filter for restoring each of the multiple pictures to another resolution, the one or more parameters related to the upsampling filter including an indication of filtering direction,

and wherein the spatial interleaving information further includes relationship information, the relationship information indicating a type of relationship that exists between the multiple pictures, the relationship information indicating that the multiple pictures are stereo views of an image, the multiple pictures are not related, the multiple pictures are a 2D image and its related depth map (2D+Z), the multiple pictures are multiple sets of a 2D+Z (MVD), the multiple pictures represent images in a layer depth video format (LDV), or the multiple pictures represent images in two sets of LDV (DES); and

form a bitstream that includes the encoded video picture and the generated information, the generated information providing information for use in processing the encoded video picture.

14. The apparatus of claim 13, wherein the the one or more processors are configured to encode the video picture based on the relationship information.

15. The apparatus of claim 13, wherein the generated information further includes camera parameter information, the camera parameter information including at least one of a focal length of a camera, a baseline distance, at least one camera position, a Znear parameter to indicate a minimum distance between an image in the multiple pictures and the camera, a Zfar parameter to indicate a maximum distance between the image in the multiple pictures and the camera, a 3×

3 intrinsic matrix, a 3×

3 rotation matrix, and a 3D translation vector.

16. The apparatus of claim 15, wherein the one or more processors are configured to encode the video picture based on the camera parameter information.

17. A non-transitory processor readable medium having stored thereon a video signal structure, the video signal structure comprising:

an encoded picture section including an encoding of a video picture, the video picture including multiple pictures combined into a single picture; and

a signaling section including an encoding of generated information indicating how the multiple pictures in the accessed video picture are combined,

wherein the generated information includes spatial interleaving information and sampling information,

wherein the spatial interleaving information indicates spatial interleaving, the spatial interleaving being side-by-side interleaving, top-bottom interleaving, row-by-row interleaving, column-by-column interleaving, or checkerboard interleaving, applied to the multiple pictures in forming the single picture and wherein the sampling information indicates one or more parameters related to an upsampling filter for restoring each of the multiple pictures to another resolution, the one or more parameters related to the upsampling filter including an indication of filtering direction,

and wherein the spatial interleaving information further includes relationship information, the relationship information indicating a type of relationship that exists between the multiple pictures, the relationship information indicating that the multiple pictures are stereo views of an image, the multiple pictures are not related, the multiple pictures are a 2D image and its related depth map (2D+Z), the multiple pictures are multiple sets of a 2D+Z (MVD), the multiple pictures represent images in a layer depth video format (LDV), or the multiple pictures represent images in two sets of LDV (DES).

18. An apparatus comprising at least a memory and one or more processors, the one or more processors being configured to:

transmit an encoded picture section including an encoding of a video picture, the video picture including multiple pictures combined into a single picture; and

transmit a signaling section including an encoding of generated information indicating how the multiple pictures in the accessed video picture are combined,

wherein the generated information includes spatial interleaving information and sampling information,

wherein the spatial interleaving information indicates spatial interleaving, the spatial interleaving being side-by-side interleaving, top-bottom interleaving, row-by-row interleaving, column-by-column interleaving, or checkerboard interleaving, applied to the multiple pictures in forming the single picture and wherein the sampling information indicates one or more parameters related to an upsampling filter for restoring each of the multiple pictures to another resolution, the one or more parameters related to the upsampling filter including an indication of filtering direction,

and wherein the spatial interleaving information further includes relationship information, the relationship information indicating a type of relationship that exists between the multiple pictures, the relationship information indicating that the multiple pictures are stereo views of an image, the multiple pictures are not related, the multiple pictures are a 2D image and its related depth map (2D+Z), the multiple pictures are multiple sets of a 2D+Z (MVD), the multiple pictures represent images in a layer depth video format (LDV), or the multiple pictures represent images in two sets of LDV (DES).