Multiview video decomposition and encoding

• US 20060132610A1
• Filed: 12/17/2004
• Published: 06/22/2006
• Est. Priority Date: 12/17/2004
• Status: Active Grant

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Vetro, Anthony, Martinian, Emin, Xin, Jun
• Granted Patent: US 7,468,745 B2
• Time in Patent Office: Days
• US Class Current: 348/207.990
• CPC Class Codes: H04N 19/103 Selection of coding mode or of prediction mode
• CPC Class Codes: H04N 19/13 Adaptive entropy coding, e.g. adaptive variable length coding [AVLC] or context adaptive binary arithmetic coding [CABAC]
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/615 using motion compensated temporal filtering [MCTF]
• CPC Class Codes: H04N 19/63 using sub-band based transform, e.g. wavelets

Abstract

A method decomposes multiview video acquired of a scene by multiple cameras. Each multiview video includes a sequence of frames, and each camera provides a different view of the scene. A prediction mode is selected from a temporal prediction mode, a spatial prediction mode, and a view interpolation prediction mode. The multiview videos are then decomposed into low band frames, high band frames, and side information according to the selected prediction mode.

Claims

1. A method for decomposing multiview videos, comprising:

acquiring a plurality of videos of a scene by a plurality of cameras, each video including a plurality of frames and each camera providing a different view of the scene;

selecting a prediction mode in which the prediction mode is selected from a temporal prediction mode, a spatial prediction mode, and a view interpolation prediction mode; and

decomposing, according to the selected prediction mode, the plurality of frames of the videos into a plurality of low band frames, a plurality of high band frames, and side information.

2. The method of claim 1, in which the high band frames encode prediction errors based on the low band frames as reference frames and the selected prediction mode.

3. The method of claim 1, in which the low band frames and the high band frames form a checkerboard pattern, the checkerboard pattern alternating the low band frames and high band frames in a spatial dimension for a single instant in time, and the checkerboard pattern alternating the low band frames and the high band frames in a temporal dimension for a particular different view.

4. The method of claim 2, in which a lifting-based wavelet transform is performed on the low band frames to determine the high band frames that encodes the prediction errors.

5. The method of claim 1, in which the side information corresponding to the temporal prediction mode is motion information, the side information corresponding to the spatial prediction mode is disparity information, and the side information corresponding to the view interpolation prediction mode is motion, disparity and depth information.

6. The method of claim 3, further comprising:

generating each high band frame based on predictions from four nearest low-band frames in the space dimension and the time dimensions.

7. The method of claim 5, in which the motion information is obtained by a motion estimation search on low band frames, and the disparity information is obtained by a disparity estimation search on low band frames.

8. The method of claim 5, in which the motion information is used to apply a motion compensated temporal filtering to the plurality of frames, and the disparity information is used to apply a disparity compensated inter-view filtering to the plurality of frames.

9. The method of claim 8, in which each frame includes a plurality of blocks of pixels, and the motion compensated temporal filtering and the disparity compensated inter-view filtering is performed adaptively to each block of each frame according to the selected prediction mode.

10. The method of claim 9, further comprising:

minimizing a cost function

$m^{*}=\underset{m}{\arg <br><br>\text{\hspace{1em}<br><br>}<br><br>\min }<br><br>\text{\hspace{1em}<br><br>}<br><br>J<br><br>\left(m\right),$

where J(m)=D(m)+λ

R(m), and D is distortion, λ

is a weighting parameter, R is rate, m indicates the set of candidate modes, and m* indicates an optimal prediction mode that has been selected based on the cost function, to adaptively select the prediction mode.

11. The method of claim 8, in which the motion compensated temporal filtering is first applied independently to each video to obtain motion filtered frames, and then applying the disparity compensated inter-view filtering to the filtered frames to provide a temporal scalability for the decomposed videos.

12. The method of claim 8, in which the disparity compensated inter-view filtering is first applied to all frames for a single instant in time to obtain filtered frames, and then applying the motion compensated temporal filtering to the filtered frames to provide view scalability for the decomposed videos.

13. The method of claim 1, in which the decomposing is performed with a resulting set of low band frames from a previous decomposition stage.

14. The method of claim 1 in which the low band frames and the high band frames are transformed, quantized and entropy encoded to produce encoded frames, and further comprising:

encoding the side information to produce encoded side information; and

multiplexing the encoded frames and encoded side information to produce a bitstream.

15. The method of claim 1, further comprising:

selecting a novel view and an instant of time;

interpolating the low band frames neighboring the novel view and at the instant of time to form an interpolated frame; and

generating the high band frame as a prediction error between a current frame and the interpolated frame.

16. The method of claim 15, in which the interpolating is based on a depth value.

17. The method of claim 16, in which the depth value is adaptively selected for each block of the current frame.

18. The method of claim 14, further comprising:

decoding the bitstream to reconstruct the plurality of videos.

19. The method of claim 18, in which a number of views is reduced for the plurality of videos.

20. The method of claim 18, in which the reconstructed videos have a reduced temporal rate.

21. A system for decomposing multiview videos, comprising:

a plurality of cameras configured to acquire a plurality of videos of a scene, each video including a plurality of frames and each camera providing a different view of the scene;

means for selecting a prediction mode, in which the prediction mode is selected from a temporal prediction, a spatial prediction, and a view interpolation prediction; and

means for decomposing, according to the selected prediction mode, the plurality of frames of the videos into a plurality of low band frames, a plurality of high band frames, and side information, and in which for the temporal prediction the side information corresponds to motion information, for the spatial prediction the side corresponds to disparity information, and for the view interpolation prediction the side information corresponds to motion, disparity and depth information.