HYBRID VIDEO CODING SUPPORTING INTERMEDIATE VIEW SYNTHESIS

• US 20140028793A1
• Filed: 01/11/2013
• Published: 01/30/2014
• Est. Priority Date: 07/15/2010
• Status: Active Grant

• Current Assignee: GE Video Compression, LLC (General Electric Company)
• Original Assignee: Fraunhofer Gesellschaft Zur Foerderung Der Angewandten Forsching E.V.
• Inventors: WIEGAND, Thomas, MUELLER, Karsten, MERKLE, Philipp
• Granted Patent: US 9,118,897 B2
• Time in Patent Office: Days
• US Class Current: 348/42
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 19/105 Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
• CPC Class Codes: H04N 19/109 among a plurality of temporal predictive coding modes
• CPC Class Codes: H04N 19/11 among a plurality of spatial predictive coding modes
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/177 the unit being a group of pictures [GOP]
• CPC Class Codes: H04N 19/30 using hierarchical techniques, e.g. scalability H04N19/63 takes precedence
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/513 Processing of motion vectors
• CPC Class Codes: H04N 19/521 for estimating the reliability of the determined motion vectors or motion vector field, e.g. for smoothing the motion vector field or for correcting motion vectors
• CPC Class Codes: H04N 19/567 Motion estimation based on rate distortion criteria
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/80 Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation H04N19/635, H04N19/86 take precedence
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

Hybrid video decoder supporting intermediate view synthesis of an intermediate view video from a first- and a second-view video which are predictively coded into a multi-view data signal with frames of the second-view video being spatially subdivided into sub-regions and the multi-view data signal having a prediction mode is provided, having: an extractor configured to respectively extract, from the multi-view data signal, for sub-regions of the frames of the second-view video, a disparity vector and a prediction residual; a predictive reconstructor configured to reconstruct the sub-regions of the frames of the second-view video, by generating a prediction from a reconstructed version of a portion of frames of the first-view video using the disparity vectors and a prediction residual for the respective sub-regions; and an intermediate view synthesizer configured to reconstruct first portions of the intermediate view video.

Claims

1. Hybrid video decoder supporting intermediate view synthesis of an intermediate view video from a first- and a second-view video which are predictively coded into a multi-view data signal with frames of the second-view video being spatially subdivided into sub-regions and the multi-view data signal comprising a prediction mode out of a set of possible prediction modes, associated with each of the sub-regions, wherein the set of possible prediction modes comprises at least an inter-view prediction mode and an intra-view prediction mode, the hybrid video decoder comprising:

an extractor configured to respectively extract, from the multi-view data signal, for sub-regions of the frames of the second-view video with which the inter-view prediction mode is associated, a disparity vector and a prediction residual;

a predictive reconstructor configured to reconstruct the sub-regions of the frames of the second-view video with which the inter-view prediction mode is associated, by generating a prediction from a reconstructed version of a portion of frames of the first-view video using the disparity vectors extracted from the multi-view data signals for the respective sub-regions, and the prediction residual for the respective sub-regions; and

an intermediate view synthesizer configured to reconstruct first portions of the intermediate view video using the reconstructed version of the portions of the frames of the first-view video, and the disparity vectors extracted from the multi-view data signal,

wherein the intermediate view synthesizer is configured to reconstruct fourth portions of the intermediate view video other than the first portions by temporally and/or spatially interpolating disparity vectors extracted from the multi-view data signal for the sub-regions of the frames of the second-view video with which the inter-view prediction mode is associated, to acquire disparity vectors for sub-regions with which the intra-view prediction mode is associated.

2. Hybrid video decoder according to claim 1, wherein the intermediate view synthesizer is configured to use the reconstruction of the first portions of the intermediate view video as supporting points in extra- and/or interpolating third portions of the intermediate view video.

3. Hybrid video decoder according to claim 1, wherein the intermediate view synthesizer is configured to finish a reconstruction of the intermediate-view video group-of-picture wise.

4. Hybrid video decoder according to claim 1, wherein the predictive reconstructor and the intermediate view synthesizer are configured to linearly map, dependent on the disparity vectors, sample positions of sub-regions of the frames of the second-view video with which the inter-view prediction mode is associated into the first view video, and sample the reconstructed version of the portions of the frames of the first-view video at the sample positions thus acquired to acquire the prediction for the sub-regions of the frames of the second-view video with which the inter-view prediction mode is associated, and the reconstruction of the first portions of the intermediate-view video respectively, with a reduction in the amount of the linear mapping of the intermediate view synthesizer relative to the predictive reconstructor, which depends on a spatial location of an intermediate view corresponding to the intermediate view video, relative to a first view corresponding to the first-view video, and a second view corresponding to the second-view video.

5. Hybrid video decoder according to claim 1, wherein the predictive reconstructor is configured to linearly map, depending on the disparity vectors, sample positions of the sub-regions of the frames of the second-view video with which the inter-video prediction mode is associated into the first view video, and sample the reconstructed version of the portions of the frames of the first-view video at the sample positions thus acquired, to acquire the prediction for the sub-regions of the frames of the second-view video with which the inter-view prediction mode is associated, and the intermediate view synthesizer is configured to linearly map, dependent on the disparity vector, sample positions of the sub-regions of the frames of the second-view video with which the inter-view prediction mode is associated, into a direction opposite to the disparity vectors, and sample the reconstruction of the sub-regions of the frames of the second-view video at the sample positions thus acquired, with a reduction of an amount of the linear mapping of the intermediate view synthesizer relative to the predictive reconstructor, which depends on a spatial location of an intermediate view corresponding to the intermediate view video, relative to a first view corresponding to the first-view video, and a second view corresponding to the second-view video, respectively.

6. Hybrid video decoder according to claim 1, wherein the extractor is configured to extract, from the multi-view data signal, for the sub-regions of frames of the second-view video, also reliability data, wherein the intermediate view synthesizer is configured to exclude the disparity vectors extracted from the multi-view data signal for sub-regions of frames of the second-view video for which the reliability data extracted from the multi-view data signal fails to fulfill a predetermined minimum requirement, from being used in reconstructing the first portions of the intermediate view video.

7. Hybrid video decoding method supporting intermediate view synthesis of an intermediate view video from a first- and a second-view video which are predictively coded into a multi-view data signal with frames of the second-view video being spatially subdivided into sub-regions and the multi-view data signal comprising a prediction mode out of a set of possible prediction modes, associated with each of the sub-regions, wherein the set of possible prediction modes comprises at least an inter-view prediction mode and an intra-view prediction mode, the hybrid video decoding method comprising:

respectively extracting, from the multi-view data signal, for sub-regions of the frames of the second-view video with which the inter-view prediction mode is associated, a disparity vector and a prediction residual;

predictively reconstructing the sub-regions of the frames of the second-view video with which the inter-view prediction mode is associated, by generating a prediction from a reconstructed version of a portion of frames of the first-view video using the disparity vectors extracted from the multi-view data signals for the respective sub-regions, and the prediction residual for the respective sub-regions; and

reconstructing first portions of the intermediate view video using the reconstructed version of the portions of the frames of the first-view video, and the disparity vectors extracted from the multi-view data signal

wherein the method further comprises reconstructing fourth portions of the intermediate view video other than the first portions by temporally and/or spatially interpolating disparity vectors extracted from the multi-view data signal for the sub-regions of the frames of the second-view video with which the inter-view prediction mode is associated, to acquire disparity vectors for sub-regions with which the intra-view prediction mode is associated.

8. Multi-view data signal comprising a first- and a second-view video predictively coded therein with frames of the second-view video being spatially subdivided into sub-regions and the multi-view data signal comprising a prediction mode out of a set of possible prediction modes, associated with each of the sub-regions, wherein the set of possible prediction modes comprises at least an inter-view prediction mode and an intra-view prediction mode, the multi-view data signal comprising, for sub-regions of the frames of the second-view video with which the inter-view prediction mode is associated, a disparity vector, a prediction residual and reliability data, with the reliability data being determined in dependence on a function which monotonically increases with decreasing value of a dispersion measure of the distribution of a resulting prediction error at a set of disparity vectors when plotted against a distance of the respective one of the set of disparity vectors from the disparity vector inserted into the multi-view data signal.

9. Hybrid video encoder for predictively encoding a first- and a second-view video into a multi-view data signal with frames of the second-view video being spatially subdivided into sub-regions, the hybrid video encoder being configured to

assign a prediction mode out of a set of possible prediction modes, to each of the sub-regions of the frames of the second-view video, wherein the set of possible prediction modes comprises at least an inter-view prediction mode and an intra-view prediction mode;

respectively determine, for sub-regions of the frames of the second-view video with which the inter-view prediction mode is associated, a disparity vector among disparity vectors out of a set of disparity vectors lying within a predetermined search area, which correspond to a local minimum of a respective prediction error resulting from applying the respective disparity vector to a reconstructed version of a portion of frames of the first-view video, and the prediction residual for the respective sub-regions, resulting from applying the disparity vector determined; and

respectively inserting, for sub-regions of the frames of the second-view video with which the inter-view prediction mode is associated, the disparity vector determined, the prediction residual determined, and reliability data into the multi-view data signal, with the reliability data being determined in dependence on a function which monotonically increases with decreasing value of a dispersion measure of the distribution of a resulting prediction error at the set of disparity vectors when plotted against a distance of the respective one of the set of disparity vectors from the disparity vector inserted into the multi-view data signal.

10. Hybrid video encoding method for predictively encoding a first- and a second-view video into a multi-view data signal with frames of the second-view video being spatially subdivided into sub-regions, the hybrid video encoding method comprising

assigning a prediction mode out of a set of possible prediction modes, to each of the sub-regions of the frames of the second-view video, wherein the set of possible prediction modes comprises at least an inter-view prediction mode and an intra-view prediction mode;

respectively determining, for sub-regions of the frames of the second-view video with which the inter-view prediction mode is associated, a disparity vector among disparity vectors out of a set of disparity vectors lying within a predetermined search area, which correspond to a local minimum of a respective prediction error resulting from applying the respective disparity vector to a reconstructed version of a portion of frames of the first-view video, and the prediction residual for the respective sub-regions, resulting from applying the disparity vector determined; and

respectively inserting, for sub-regions of the frames of the second-view video with which the inter-view prediction mode is associated, the disparity vector determined, the prediction residual determined, and reliability data into the multi-view data signal, with the reliability data being determined in dependence on a function which monotonically increases with decreasing value of a dispersion measure of the distribution of a resulting prediction error at the set of disparity vectors when plotted against a distance of the respective one of the set of disparity vectors from the disparity vector inserted into the multi-view data signal.

11. A non-transitory computer readable medium including a computer program comprising a program code for performing, when running on a computer, a hybrid video decoding method supporting intermediate view synthesis of an intermediate view video from a first- and a second-view video which are predictively coded into a multi-view data signal with frames of the second-view video being spatially subdivided into sub-regions and the multi-view data signal comprising a prediction mode out of a set of possible prediction modes, associated with each of the sub-regions, wherein the set of possible prediction modes comprises at least an inter-view prediction mode and an intra-view prediction mode, the hybrid video decoding method comprising:

respectively extracting, from the multi-view data signal, for sub-regions of the frames of the second-view video with which the inter-view prediction mode is associated, a disparity vector and a prediction residual;

predictively reconstructing the sub-regions of the frames of the second-view video with which the inter-view prediction mode is associated, by generating a prediction from a reconstructed version of a portion of frames of the first-view video using the disparity vectors extracted from the multi-view data signals for the respective sub-regions, and the prediction residual for the respective sub-regions; and

reconstructing first portions of the intermediate view video using the reconstructed version of the portions of the frames of the first-view video, and the disparity vectors extracted from the multi-view data signal

wherein the method further comprises reconstructing fourth portions of the intermediate view video other than the first portions by temporally and/or spatially interpolating disparity vectors extracted from the multi-view data signal for the sub-regions of the frames of the second-view video with which the inter-view prediction mode is associated, to acquire disparity vectors for sub-regions with which the intra-view prediction mode is associated.

12. A non-transitory computer readable medium including a computer program comprising a program code for performing, when running on a computer, a hybrid video encoding method for predictively encoding a first- and a second-view video into a multi-view data signal with frames of the second-view video being spatially subdivided into sub-regions, the hybrid video encoding method comprising

assigning a prediction mode out of a set of possible prediction modes, to each of the sub-regions of the frames of the second-view video, wherein the set of possible prediction modes comprises at least an inter-view prediction mode and an intra-view prediction mode;

respectively determining, for sub-regions of the frames of the second-view video with which the inter-view prediction mode is associated, a disparity vector among disparity vectors out of a set of disparity vectors lying within a predetermined search area, which correspond to a local minimum of a respective prediction error resulting from applying the respective disparity vector to a reconstructed version of a portion of frames of the first-view video, and the prediction residual for the respective sub-regions, resulting from applying the disparity vector determined; and

respectively inserting, for sub-regions of the frames of the second-view video with which the inter-view prediction mode is associated, the disparity vector determined, the prediction residual determined, and reliability data into the multi-view data signal, with the reliability data being determined in dependence on a function which monotonically increases with decreasing value of a dispersion measure of the distribution of a resulting prediction error at the set of disparity vectors when plotted against a distance of the respective one of the set of disparity vectors from the disparity vector inserted into the multi-view data signal.