Optimal disparity estimation for stereoscopic video coding

• US 5,652,616 A
• Filed: 08/06/1996
• Issued: 07/29/1997
• Est. Priority Date: 08/06/1996
• Status: Expired due to Term

• Current Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Original Assignee: General Instrument Corporation (CommScope Holding Company, Inc.)
• Inventors: Chen, Xuemin, Liu, Vincent
• Primary Examiner(s): Britton, Howard W.
• Assistant Examiner(s): Snow, Frank
• Application Number:
  US08/692,630
• Time in Patent Office: 357 Days
• Field of Search: 348/42, 348/43, 348/46, 348/47
• US Class Current: 348/43
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06T 9/004 Predictors, e.g. intraframe, interframe coding
• CPC Class Codes: H04N 13/133 Equalising the characteristics of different image components, e.g. their average brightness or colour balance
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/218 using spatial multiplexing
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

Disparity estimation between the right and left view pixel luminance values in a stereoscopic video signal is optimized by determining the minimum least-square-error between macroblocks of the right and left view pictures. Affine transform coefficients and disparity vectors which correspond to the minimum error are also determined and transmitted in the data stream for use by a decoder in reconstructing the right view picture. The scheme can be implemented either locally, at the macroblock level, or globally, at the picture level. At the macroblock level, least-square-error optimization may occur for each individual macroblock in the right view picture. In this case, affine transform coefficients are provided for each macroblock. At the picture level, the sum of the least-square-errors is minimized after the blocks of the right view picture are matched to the left view picture. In this case, only one set of affine transform coefficients are required for the entire right view picture. Or, block matching between an affinely transformed left view picture and the right view picture may be performed after minimizing the sum of the least-square-errors. The scheme is particularly useful in minimizing the effects of cross-channel luminance imbalances due to camera variations and scenes with significant changes in brightness or contrast, and is compatible with stereoscopic video systems such as the MPEG Multi-view Profile (MVP) system.

Claims

1. A method for reducing cross-channel luminance imbalance in an enhancement layer picture of a stereoscopic video signal, comprising the steps of:

providing a search window which comprises at least a portion of the pixels in said enhancement layer picture;

providing a reference window which comprises at least a portion of the pixels in a reference picture of a base layer of said stereoscopic video signal; and

determining affine transform coefficients a, b of said reference window which minimize the least-square-error between luminance values of pixels of said search window and said reference window.

2. The method of claim 1, comprising the further step of:

affinely transforming said reference window pixel data using said affine transform coefficients a, b;

differentially encoding said search window pixel data using said transformed reference window pixel data; and

transmitting said differentially encoded search window pixel data in conjunction with said stereoscopic video signal for use in reconstructing said enhancement layer picture.

3. The method of claim 1, comprising the further step of:

adjusting a size of at least one of said search window and said reference window and repeating said minimizing step until said least-square-error is within a target error range.

4. The method of claim 1, comprising the further step of:

transmitting said affine transform coefficients a, b in said stereoscopic video signal for use in reconstructing said enhancement layer picture.

5. The method of claim 1, comprising the further steps of:

providing additional search windows which comprise respective portions of the pixels in said enhancement layer picture;

providing additional corresponding reference windows which comprise respective portions of the pixels in said reference picture; and

for each of said additional search windows, determining a set of affine transform coefficients a, b which minimize the least-square-error between luminance values of pixels of said search window and the corresponding reference.

6. A method for reducing cross-channel luminance imbalance in an enhancement layer picture of a stereoscopic video signal, comprising the steps of:

providing a plurality of windows which comprise respective portions of the pixels in said enhancement layer picture;

providing a corresponding plurality of reference windows which comprise respective portions of the pixels in a reference picture of a base layer of said stereoscopic video signal; and

determining affine transform coefficients a, b which minimize the sum of the least-square-errors between luminance values of pixels of said enhancement layer picture windows and said corresponding reference windows.

7. The method of claim 6, wherein said plurality of enhancement layer picture windows are search windows, comprising the further step of:

prior to said determining step, matching said plurality of search windows to respective regions of said corresponding plurality of reference windows.

8. The method of claim 6, comprising the further steps of:

transforming said corresponding plurality of reference windows according to said affine transform coefficients a, b to provide a plurality of transformed reference windows;

matching said plurality of enhancement layer picture windows to regions of respective ones of said transformed plurality of reference windows to provide a matched plurality of enhancement layer picture windows; and

for each of said matched plurality of enhancement layer picture windows, determining a disparity vector which is indicative of a translational difference between the matched of enhancement layer picture window and the corresponding transformed reference window.

9. The method of claim 8, comprising the further step of:

transmitting the disparity vectors in said stereoscopic video signal for use in reconstructing said enhancement layer picture.

10. A method for decoding a stereoscopic video signal with a reduced cross-channel luminance imbalance in an enhancement layer picture thereof, comprising the steps of:

retrieving affine transform coefficients a, b from said stereoscopic video signal;

said affine transform coefficients being determined by minimizing the least-square-error between luminance values of pixels of a search window and a reference window, wherein said search window comprises at least a portion of the pixels in said enhancement layer picture, and said reference window comprises at least a portion of the pixels in a reference picture of a base layer of said stereoscopic video signal; and

using said affine transform coefficients to recover said search window pixel data.

11. The method of claim 10, wherein said search window pixel data is carried in said stereoscopic video signal as differentially encoded data, comprising the further steps of:

retrieving said reference window pixel data from said stereoscopic video signal;

using said reference window pixel data to provide reference window pixel prediction data;

affinely transforming said reference window pixel prediction data in accordance with said affine transform coefficients; and

summing said affinely transformed reference window pixel prediction data with said differentially encoded data to recover said search window pixel data.

12. An encoder for reducing cross-channel luminance imbalance in an enhancement layer picture of a stereoscopic video signal, comprising:

means for providing a search window which comprises at least a portion of the pixels in said enhancement layer picture;

means for providing a reference window which comprises at least a portion of the pixels in a reference picture of a base layer of said stereoscopic video signal; and

a disparity estimator for determining affine transform coefficients a, b of said reference window which minimize the least-square-error between luminance values of pixels of said search window and said reference window.

13. The apparatus of claim 12, further comprising:

a predictor operatively associated with said disparity estimator for use in affinely transforming said reference window pixel data using said affine transform coefficients a, b;

an adder for use in differentially encoding said search window pixel data using said transformed reference window pixel data; and

a transmitter for transmitting said differentially encoded search window pixel data in conjunction with said stereoscopic video signal for use in reconstructing said enhancement layer picture.

14. The apparatus of claim 13, further comprising:

a motion compensator for processing said search window pixel data to provide motion compensated search window pixel data; and

a switch for selecting one of said motion compensated data and said differentially encoded data for use in reconstructing said search window pixel data at a decoder.

15. An encoder for reducing cross-channel luminance imbalance in an enhancement layer picture of a stereoscopic video signal, comprising:

means for providing a plurality of windows which comprise respective portions of the pixels in said enhancement layer picture;

means for providing a corresponding plurality of reference windows which comprise respective portions of the pixels in a reference picture of a base layer of said stereoscopic video signal; and

a disparity estimator for determining affine transform coefficients a, b which minimize the sum of the least-square-errors between luminance values of pixels of said enhancement layer picture windows and said corresponding reference windows.

16. A decoder apparatus for decoding a stereoscopic video signal with a reduced cross-channel luminance imbalance in an enhancement layer picture thereof, comprising:

means for retrieving affine transform coefficients a, b from said stereoscopic video signal;

said affine transform coefficients being determined by minimizing the least-square-error between luminance values of pixels of a search window and a reference window, wherein said search window comprises at least a portion of the pixels in said enhancement layer picture, and said reference window comprises at least a portion of the pixels in a reference picture of a base layer of said stereoscopic video signal; and

processing means for using said affine transform coefficients to recover said search window pixel data.

17. The apparatus of claim 16, wherein said search window comprises a macroblock of said enhancement layer picture.

18. The apparatus of claim 17, wherein said search window pixel data is carried in said stereoscopic video signal as differentially encoded data, further comprising:

means for retrieving said reference window pixel data from said stereoscopic video signal;

prediction means for using said reference window pixel data to provide reference window pixel prediction data;

transform means for affinely transforming said reference window pixel prediction data in accordance with said affine transform coefficients; and

an adder for summing said affinely transformed reference window pixel prediction data with said differentially encoded data to recover said search window pixel data.