Method and system for processing multiview videos for view synthesis using side information

• US 7,728,878 B2
• Filed: 07/12/2006
• Issued: 06/01/2010
• Est. Priority Date: 12/17/2004
• Status: Expired due to Fees

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Oh, Jong Dae, Ince, Serdar, Vetro, Anthony, Martinian, Emin, Yea, Sehoon
• Primary Examiner(s): Ometz; David L
• Assistant Examiner(s): Trehan; Akshay
• Application Number:
  US11/485,092
• Publication Number:
  US 20070030356A1
• Time in Patent Office: 1,420 Days
• Field of Search: 348/207.99, 348/218.1, 37524001-24029
• US Class Current: 348/218.1
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 19/103 Selection of coding mode or of prediction mode
• CPC Class Codes: H04N 19/105 Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
• CPC Class Codes: H04N 19/13 Adaptive entropy coding, e.g. adaptive variable length coding [AVLC] or context adaptive binary arithmetic coding [CABAC]
• CPC Class Codes: H04N 19/147 according to rate distortion criteria rate-distortion as a criterion for motion estimation H04N19/567
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/573 Motion compensation with multiple frame prediction using two or more reference frames in a given prediction direction
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/615 using motion compensated temporal filtering [MCTF]
• CPC Class Codes: H04N 19/63 using sub-band based transform, e.g. wavelets

Abstract

A method processes a multiview videos of a scene, in which each video is acquired by a corresponding camera arranged at a particular pose, and in which a view of each camera overlaps with the view of at least one other camera. Side information for synthesizing a particular view of the multiview video is obtained in either an encoder or decoder. A synthesized multiview video is synthesized from the of multiview videos and the side information. A reference picture list is maintained for each current frame of each of the multiview videos, the reference picture indexes temporal reference pictures and spatial reference pictures of the acquired multiview videos and the synthesized reference pictures of the synthesized multiview video. Each current frame of the multiview videos is predicted according to reference pictures indexed by the associated reference picture list.

Claims

1. A method for processing a plurality of multiview videos of a scene, in which each video is acquired by a corresponding camera arranged at a particular pose, and in which a view of each camera overlaps with the view of at least one other camera, comprising the steps of:

obtaining side information for synthesizing a particular view of the multiview video, wherein the side information includes depth values;

synthesizing, using the side information, a synthesized multiview video from an input video, wherein the input video includes at least one of the multiview videos, and the synthesized multiview video corresponds to single pose different than the input video;

maintaining a reference picture list for a picture associated with a particular time instant and a particular one of the plurality of multiview videos, wherein the reference picture list indexes temporal reference pictures and spatial reference pictures of the plurality of acquired multiview videos and the synthesized reference pictures of the synthesized multiview video, and wherein where the temporal reference pictures are associated with a time instant that is different than the particular time instant and the spatial reference pictures are associated with the same particular time instant, and wherein each reference picture is assigned a unique picture order count; and

predicting each current frame of the plurality of multiview videos according to reference pictures indexed by the associated reference picture list.

2. The method of claim 1, in which the side information additionally includes correction vectors.

3. The method of claim 2, in which the side information is associated with each block partition of each macroblock in each frame.

4. The method of claim 2, in which the correction vector has sub-pixel accuracy with respect to the current frame.

5. The method of claim 2, in which luminance and chrominance samples of the spatial reference pictures are used to determine the associated depth values for each block partition in each frame.

6. The method of claim 5, in which the displacement vector has sub-pixel accuracy.

7. The method of claim 1, further comprising:

warping a single one of the acquired plurality of multiview videos to synthesize the synthesized multiview video.

8. The method of claim 1, further comprising:

interpolating two or more of the acquired plurality of multiview videos to synthesize the synthesized multiview video.

9. The method of claim 1, in which the side information is obtained in an encoder.

10. The method of claim 1, in which the side information is obtained in a decoder.

11. The method of claim 10, in which the side information is obtained in the decoder through a context-based binary arithmetic decoder.

12. The method of claim 1, in which the synthesizing is performed by applying a geometric transformation to pixels in a macroblock partition according to camera parameters and the associated side information, and rounding to a nearest sub-pel position in the spatial reference pictures.

13. The method of claim 1, in which each frame of the synthesized multiview video includes a plurality of synthesized macroblocks that are generated based on the spatial reference pictures and the side information associated with each macroblock partition.

14. The method of claim 13, in which a reconstructed macroblock is generated by adding the synthesized macroblock and a decoded residual macroblock.

15. The method of claim 13, in which the synthesized macroblock is associated with a displacement vector.

16. The method of claim 1, in which the predicting using the synthesized multiview video as a reference is performed for V-pictures.

17. The method of claim 1, in which a synthesized frame of the synthesized multiview video is generated by warping a corresponding frame of one of the acquired plurality of multiview videos.

18. The method of claim 1, in which the synthesizing uses a model of the scene.

19. The method of claim 1, in which the synthesizing of each uses each frame of the synthesized multiview video uses a depth map, an intrinsic matrix, a rotation matrix, and a translation vector.

20. A system for processing a plurality of multiview videos of a scene, comprising:

a plurality of cameras, each camera configured to acquire a multiview video of a scene, each camera arranged at a particular pose, and in which a view of each camera overlaps with the view of at least one other camera;

means for obtaining side information for synthesizing a particular view of the multiview video, wherein the side information includes depth values;

means for synthesizing a synthesized multiview video from an input video using the side information, wherein the input video includes at least one of the multiview videos, and the synthesized multiview video corresponds to single pose different than the input video;

a memory for maintaining a reference picture list for a picture associated with a particular time instant and a particular one of the plurality of multiview videos, wherein the reference picture list indexes temporal reference pictures and spatial reference pictures of the plurality of acquired multiview videos and the synthesized reference pictures of the synthesized multiview video, and wherein where the temporal reference pictures are associated with a time instant that is different than the particular time instant and the spatial reference pictures are associated with the same particular time instant, and wherein each reference picture is assigned a unique picture order count; and

means for predicting each current frame of the plurality of multiview videos according to reference pictures indexed by the associated reference picture list.