Apparatus and Method for Producting Multi-View Contents

• US 20070296721A1
• Filed: 07/26/2005
• Published: 12/27/2007
• Est. Priority Date: 11/08/2004
• Status: Abandoned Application

• Current Assignee: Electronics and Telecommunications Research Institute
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: Um, Gi-Mun, Ahn, Chung-Hyun, Lee, Soo-In, Chang, Eun-Young, Kim, Daehee
• Application Number:
  US11/718,796
• Publication Number:
  US 20070296721A1
• Time in Patent Office: Days
• US Class Current: 345/427
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/133 Equalising the characteristics of different image components, e.g. their average brightness or colour balance
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 13/359 Switching between monoscopic and stereoscopic modes
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2213/003 Aspects relating to the "2D+depth" image format
• CPC Class Codes: H04N 2213/005 Aspects relating to the "3D+depth" image format

Abstract

Provided are a contents generating apparatus that can support functions of moving object substitution, depth-based object insertion, background image substitution, and view offering upon a user request and provide realistic image by applying lighting information applied to a real image to computer graphics object when a real image is composited with computer graphics object, and a contents generating method thereof. The apparatus includes: a preprocessing block, a camera calibration block, a scene model generating block, an object extracting/tracing block, a real image/computer graphics object compositing block, an image generating block, and the user interface block. The present invention can provide diverse production methods such as testing for the optimal camera viewpoint and scenic structure before contents are actually authored and compositing two different scenes taken in different places into one scene based on a concept of a three-dimensional virtual studio in the respect of a contents producer.

Claims

1. An apparatus for generating multi-view contents, comprising:

a preprocessing block for performing correction on and removing noise from depth/disparity map data and multi-view images which are inputted from outside to thereby produce corrected multi-view images;

a camera calibration block for calculating camera parameters based on basic camera information and the corrected multi-view images outputted from the preprocessing block, and performing epipolar rectification to thereby produce rectified multi-view images;

a scene model generating block for generating a scene model by using the camera parameters and the epipolar-rectified multi-view images, which are outputted from the camera calibration block, and a depth/disparity map which is outputted from the preprocessing block;

an object extracting/tracing block for extracting an object binary mask, an object motion vector, and a position of an object central point by using the rectified multi-view images outputted from the preprocessing block, the camera parameters outputted from the camera calibration block, and target object setting information outputted from the user interface block;

a real image/computer graphics object compositing block for extracting lighting information of a background image, which is a real image, applying the extracted lighting information when a pre-produced computer graphic is inserted into the real image, and compositing the pre-produced computer graphics model and the real image;

an image generating block for generating stereoscopic images, virtual multi-view images, and intermediate-view images by using the camera parameters outputted from the camera calibration block, the user selected viewpoint information outputted from a user interface block, and the virtual multi-view images corresponding to the user selected viewpoint information; and

the user interface block for converting requirements from a user into internal data and transmitting the internal data to the preprocessing block, the camera calibration block, the scene modeling block, the object extracting/tracing block, the real image/computer graphics object compositing block, and the image generating block.

2. The apparatus as recited in claim 1, wherein the preprocessing block includes:

a size corrector for correcting the multi-view images to have the same size, when the sizes of the multi-view images are different;

a color corrector for correcting the multi-view images to have the same colors based on a color correction algorithm, when the colors of the multi-view images are different; and

a depth/disparity preprocessor for removing noise from the depth/disparity data through filtering.

3. The apparatus as recited in claim 1, wherein the camera calibration block includes:

a parameter calculator for extracting the camera parameters based on the basic camera information and the corrected multi-view images outputted from the preprocessing block; and

an epipolar rectifier for performing epipolar rectification of the multi-view images outputted from the preprocessing block based on the camera parameters outputted from the parameter calculator.

4. The apparatus as recited in claim 1, wherein the scene model generating block includes:

a disparity map extractor for generating a plurality of disparity maps by using the camera parameters outputted from the camera calibration block and the epipolar-rectified multi-view images;

an integrator for generating a scene model by integrating a disparity map outputted from the disparity map extractor and a depth/disparity map outputted from the preprocessing block;

an object depth mask generator for generating an object mask having depth information by using the object binary mask information outputted from the object extracting/tracing block and the scene model outputted from the integrator; and

a three-dimensional point cloud generator for generating a three-dimensional point cloud of a scene/object and a mesh model by using the camera parameters outputted from the camera calibration block.

5. The apparatus as recited in claim 1, wherein the object extracting/tracing means includes:

an object extractor for extracting an object binary mask by using at least one among the multi-view images outputted from the preprocessing block, the preprocessed depth/disparity map outputted from the preprocessing block, and the scene model outputted from the scene model generating block, and the target object setting information outputted from the user interface block;

an object motion vector extractor for extracting a central point of the object binary mask outputted from the object extractor, and calculating and storing image coordinates of the central point for every frame; and

a three-dimensional coordinates converter for converting image coordinates of the object motion vector outputted from the object motion vector extractor into three-dimensional world coordinates by using at least one between the depth/disparity map outputted from the preprocessing block and a scene model outputted from the scene model generator, and the camera parameters outputted from the camera calibration block.

6. The apparatus as recited in claim 1, wherein the real image/computer graphics object compositing block includes:

a lighting information extractor for extracting lighting information of the background image, which is the real image, based on a plurality of images with different light exposure levels and light exposure values thereof;

a computer graphic renderer for rendering computer graphics object according to a viewpoint based on viewpoint information outputted from the user interface block; and

an image compositor for inserting a computer graphics object model into the real image based on a depth key according to a computer graphic insertion position transmitted from the user interface block.

7. The apparatus as recited in claim 1, wherein the image generating block includes:

a stereoscopic image generator for generating stereoscopic images, virtual multi-view images by using the multi-view images outputted from the preprocessing block, at least one between the preprocessed depth/disparity map and the scene model outputted from the scene model generating block, and the camera parameters from the camera calibration block; and

an intermediate-view image generator for generating intermediate-view images by using the multi-view images outputted from the preprocessing block, at least one among the preprocessed depth/disparity map outputted from the preprocessing block, the scene model outputted from the scene model generating block, and a plurality of disparity maps outputted from the scene model generating block, the user selected viewpoint information outputted from the user interface block.

8. A method for generating multi-view contents, comprising the steps of:

a) performing correction on and removing noise from depth/disparity map data and multi-view images which are inputted from outside to thereby produce a corrected multi-view images;

b) calculating camera parameters based on basic camera information and the corrected multi-view images and performing epipolar rectification to thereby produce epipolar-rectified multi-view images;

c) generating a scene model by using the camera parameters and the epipolar-rectified multi-view images, which are outputted from the step b), and the preprocessed depth/disparity maps which are outputted from the step a);

d) extracting an object binary mask, an object motion vector, and a position of an object central point by using target object setting information, the corrected multi-view images, and the camera parameters;

e) extracting lighting information of a background image, which is a real image, applying the lighting information extracted when a pre-produced computer graphic is inserted into the real image, and compositing the pre-produced computer graphic and the real image; and

f) generating stereoscopic images, virtual multi-view images, and intermediate-view images by using user selected viewpoint information, the multi-view images corresponding to the user selected viewpoint information, and the camera parameters.

9. The method as recited in claim 8, wherein the step a) includes the steps of:

a1) correcting the multi-view images to have the same size, when the sizes of the multi-view images are different;

a2) correcting the multi-view images to have the same colors based on a color correction algorithm, when the colors of the multi-view images are different; and

a3) removing noise from the depth/disparity data through filtering.

10. The method as recited in claim 8, wherein the step b) includes the steps of:

b1) extracting the camera parameters based on the basic camera information and the corrected multi-view images; and

b2) performing epipolar rectification on the multi-view images based on the camera parameters to thereby produce epipolar-rectified multi-view images.

11. The method as recited in claim 8, wherein the step c) includes the steps of:

c1) generating a plurality of disparity maps by using the camera parameters and the epipolar-rectified multi-view images;

c2) generating a scene model by integrating a disparity map outputted from the step c1) and the preprocessed depth/disparity map outputted from the step a);

c3) generating an object mask having depth information by using the object binary mask information outputted from the step d) and the scene model generated in the step c2); and

c4) generating a three-dimensional point cloud of a scene/object and a mesh model by using the camera parameters outputted from the step b).

12. The method as recited in claim 8, wherein the step d) includes the steps of:

d1) extracting an object binary mask by using at least one among the corrected multi-view images outputted from the step a), the preprocessed depth/disparity map, and the scene model generated in the step c), and target object setting information inputted from a user;

d2) extracting a central point of the object binary mask extracted in the step d1), and calculating and storing image coordinates of the central point for every frame; and

d3) converting image coordinates of the object motion vector outputted from the step d2) into three-dimensional world coordinates by using at least one between the depth/disparity map preprocessed in the step a) and the scene model generated in the step c), and the camera parameters calculated in the step b).

13. The method as recited in claim 8, wherein the step e) includes the steps of:

e1) extracting lighting information of the background image, which is the real image, based on a plurality of images with different light exposure levels and light exposure values thereof;

e2) rendering computer graphics object according to a viewpoint based on viewpoint information transmitted from the user; and

e3) inserting a computer graphics object model into the real image based on a depth key according to a computer graphic insertion position transmitted from the user interface block.

14. The method as recited in claim 8, wherein the step f) includes the steps of:

f1) generating stereoscopic images and virtual multi-view images by using at least among the multi-view images preprocessed in the step a), the preprocessed depth/disparity map and the scene model generated in the step c), the camera parameters calculated in the step b), and user selected viewpoint information; and

f2) generating intermediate-view images by using at least one among the multi-view images preprocessed in the step a), the preprocessed depth/disparity map, the scene models generated in the step c), a plurality of disparity maps generated in the step c), the camera parameters, and the user selected viewpoint information.