Method and system for processing multiview videos for view synthesis using skip and direct modes

• US 7,671,894 B2
• Filed: 01/09/2007
• Issued: 03/02/2010
• Est. Priority Date: 12/17/2004
• Status: Expired due to Fees

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Vetro, Anthony, Yea, Sehoon
• Primary Examiner(s): Ometz; David L
• Assistant Examiner(s): Trehan; Akshay
• Application Number:
  US11/621,400
• Publication Number:
  US 20070109409A1
• Time in Patent Office: 1,148 Days
• Field of Search: 348/207.99, 348/218.1, 37524001-24029
• US Class Current: 348/218.1
• CPC Class Codes: H04N 19/122 Selection of transform size, e.g. 8x8 or 2x4x8 DCT; Selection of sub-band transforms of varying structure or type
• CPC Class Codes: H04N 19/13 Adaptive entropy coding, e.g. adaptive variable length coding [AVLC] or context adaptive binary arithmetic coding [CABAC]
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/147 according to rate distortion criteria rate-distortion as a criterion for motion estimation H04N19/567
• CPC Class Codes: H04N 19/159 Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/174 the region being a slice, e.g. a line of blocks or a group of blocks
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/423 characterised by memory arrangements H04N19/433 takes precedence
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/52 by predictive encoding
• CPC Class Codes: H04N 19/54 using feature points or meshes
• CPC Class Codes: H04N 19/573 Motion compensation with multiple frame prediction using two or more reference frames in a given prediction direction
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 19/593 involving spatial prediction techniques
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/615 using motion compensated temporal filtering [MCTF]
• CPC Class Codes: H04N 19/63 : using sub-band based transform, e.g. wavelets
• CPC Class Codes: H04N 19/635 : characterised by filter definition or implementation details
• CPC Class Codes: H04N 19/70 : characterised by syntax aspects related to video coding, e.g. related to compression standards
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources

Abstract

A method processes a multiview videos of a scene, in which each video is acquired by a corresponding camera arranged at a particular pose, and in which a view of each camera overlaps with the view of at least one other camera. Side information for synthesizing a particular view of the multiview video is obtained in either an encoder or decoder. A synthesized multiview video is synthesized from the multiview videos and the side information. A reference picture list is maintained for each current frame of each of the multiview videos, the reference picture indexes temporal reference pictures and spatial reference pictures of the acquired multiview videos and the synthesized reference pictures of the synthesized multiview video. Each current frame of the multiview videos is predicted according to reference pictures indexed by the associated reference picture list with a skip mode and a direct mode, whereby the side information is inferred from the synthesized reference picture.

Claims

1. A method for processing a plurality of multiview videos of a scene, in which each video is acquired by a corresponding camera arranged at a particular pose, and in which a view of each camera overlaps with the view of at least one other camera, comprising the steps of:

obtaining side information for synthesizing a particular view of the multiview videos;

synthesizing a synthesized reference picture from at least one input video selected from the plurality of multiview videos and the side information, wherein the synthesized reference picture corresponds to a single pose different than the input video;

maintaining a reference picture list for each current frame of each of the plurality of multiview videos, wherein the reference picture list indexes temporal reference pictures and spatial reference pictures of the plurality of multiview videos and the synthesized reference picture and wherein the temporal reference pictures are associated with different time instants and the spatial reference pictures are associated with a same time instant; and

predicting each current frame corresponding to the single pose of the synthesized reference picture according to reference pictures indexed by the associated reference picture list, wherein the predicting uses a synthetic skip mode and a synthetic direct mode based on the synthesized reference picture, and wherein the side information is inferred from an earliest synthesized reference picture in the reference picture list.

2. The method of claim 1 in which the synthetic skip mode is with respect to the corresponding synthesized reference picture.

3. The method of claim 2 in which the synthetic skip mode is signaled by adding an additional bit to a mb_skip_flag.

4. The method of claim 3 in which the additional bit is added when the mb_skip_flag is equal to 1 to distinguish a conventional skip mode from the synthetic skip mode, where the additional bit equals 1 to signal the synthetic skip mode, and equals 0 to signal the conventional skip mode.

5. The method of claim 3 in which the additional bit is added when the mb_skip_flag is equal to 0 to distinguish a conventional non-skip mode from the synthetic skip mode, where the additional bit equals 1 to signal the synthetic skip mode, and equals 0 to signal the conventional non-skip mode.

6. The method of claim 2 in which the synthetic skip mode is signaled at a slice layer to collectively signal the synthetic skip mode for all macroblocks in the slice.

7. The method of claim 2 in which the synthetic skip mode is signaled for P-slices, where a first view synthesis reference picture in the reference picture list is chosen as the reference picture.

8. The method of claim 2 in which the synthetic skip mode is signaled for B-slices, where an earliest view synthesis reference picture in the reference picture list is chosen as the reference picture.

9. The method of claim 1, in which the side information includes depth values and correction vectors.

10. The method of claim 2 in which a depth value is derived for the synthetic skip mode and is a median of available neighboring depth values.

11. The method of claim 2 in which a correction vector is derived for the synthetic skip mode and is a median of available neighboring correction vectors.

12. The method of claim 1 in which the synthetic direct mode is with respect to the corresponding synthesized reference picture.

13. The method of claim 12 in which the synthetic direct mode is signaled as an additional macroblock mode.

14. The method of claim 12 in which the synthetic direct mode is signaled with an additional flag.

15. The method of claim 12 in which the synthetic direct mode is signaled for B-slices, where earliest view synthesis reference pictures in the reference picture lists are chosen as the references pictures.

16. The method of claim 12 in which a depth value is derived for the synthetic direct mode is a median of available neighboring depth values.

17. The method of claim 12 in which a correction vector is derived for the synthetic direct mode is a median of available neighboring correction vectors.

18. The method of claim 1, in which the side information is obtained in an encoder.

19. The method of claim 1, in which the side information is obtained in a decoder.

20. A system for processing a plurality of multiview videos of a scene, comprising:

a plurality of cameras, each camera configured to acquire a multiview video of a scene, each camera arranged at a particular pose, and in which a view of each camera overlaps with the view of at least one other camera;

means for obtaining side information for synthesizing a particular view of the multiview videos;

means for synthesizing a synthesized reference picture from at least one input video selected from the plurality of multiview videos and the side information, wherein the synthesized reference picture corresponds to a single pose different than the input video;

a memory buffer configured to maintain a reference picture list for each current frame of each of the plurality of multiview videos, wherein the reference picture list indexes temporal reference pictures and spatial reference pictures of the plurality of multiview videos and the synthesized reference picture; and

means for predicting each current frame corresponding to the single pose of the synthesized reference picture according to reference pictures indexed by the associated reference picture list, wherein the predicting uses a synthetic skip mode and a synthetic direct mode, and wherein the side information is inferred from an earliest synthesized reference picture in the reference picture list.