Apparatus for synthesizing multiview image using two images of stereo camera and depth map

• US 20030095119A1
• Filed: 11/15/2002
• Published: 05/22/2003
• Est. Priority Date: 11/17/2001
• Status: Active Grant

• Current Assignee: Pohang University of Science and Technology Foundation
• Original Assignee: Pohang University of Science and Technology Foundation
• Inventors: Oh, Yuns, Jeong, Hong
• Granted Patent: US 6,903,735 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A multiview image synthesizing apparatus using a depth map of a stereo camera includes a multiview image synthesizing unit for converting an image signal input from first and second cameras to a digital signal and outputting a value of a depth map at a position of a virtual camera located at an arbitrary position between the first and second cameras, a multiview image re-projection unit for calculating a mapping pixel index of a multiview re-projection image for the virtual camera set by a user, by receiving the digital signal transmitted from the multiview image synthesizing unit, and outputting the calculated mapping pixel index, and a pixel selection unit for selecting a predetermined pixel to display an overlapping pixel of the digital signal input through the multiview image synthesizing unit, by using an output signal output from the multiview image re-projection unit. Thus, an image of a virtual camera set by a user is produced in real time by using a stereo image sequence using the serial processing step of a depth map and the recurrence algorithm.

Claims

1. A multiview image synthesizing apparatus using a depth map of a stereo camera comprising:

a multiview image synthesizing unit for converting an image signal input from first and second cameras to a digital signal and outputting a value of a depth map at a position of a virtual camera located in a position between the first and second cameras;

a multiview image re-projection unit for calculating a mapping pixel index of a multiview re-projection image for the virtual camera set by a user, by receiving the digital signal transmitted from the multiview image synthesizing unit, and outputting the calculated mapping pixel index; and

a pixel selection unit for selecting a predetermined pixel to display an overlapping pixel of the digital signal input through the multiview image synthesizing unit, by using a signal output from the multiview image re-projection unit.

2. The apparatus as claimed in claim 1, wherein the digital signal converted by the multiview image synthesizing unit is a depth map that contains three-dimensional reconstruction data.

3. The apparatus as claimed in claim 1, wherein the multiview image synthesizing unit comprises:

a storage portion for storing the digital image pixels of the digital signal converted from the image signal output from the first and second cameras; and

a processing portion for outputting data of disparity by using the digital signal stored in the storage portion.

4. The apparatus as claimed in claim 1, wherein the storage portion of the multiview image re-projection unit obtains and stores an updated index value by adding a value combined according to a predetermined equation to an index of a re-projection image pixel in the previous step by using two integer numbers to represent the position of a virtual camera on a virtual line between the first and second cameras.

5. The apparatus as claimed in claim 1, wherein a depth map of the image signals output from the first and second cameras is used to obtain an image by the virtual camera located at an arbitrary position between the two images obtained from the first and second cameras.

6. The apparatus as claimed in claim 1, wherein the multiview image re-projection unit comprises:

a first control portion for outputting a predetermined control signal by receiving the digital signal transmitted from the multiview image synthesizing unit;

a coding portion for converting the second input signal of the user to a positive or negative number according to a predetermined control signal of the first control portion;

a first zero portion for converting the second input signal of the user coded by the coding portion to “

0”

according to a predetermined control signal of the first control portion;

a shift register portion for converting the first input signal of the user to “

½

”

according to a predetermined control signal of the first control portion;

a second zero portion for converting a value output from the shift register portion to “

0”

according to a predetermined control signal of the first control portion;

a first sum portion for adding a signal output from the second zero portion of the first input signal to a signal output from the first zero portion of the second input signal and outputting the added value as a difference value of a mapped re-projection image index;

a second sum portion for adding the difference value of an index output from the first sum portion to an index of a re-projection image and outputting an updated index value; and

a register portion for storing the updated index value output from the second sum portion.

7. The apparatus as claimed in claim 1, wherein the multiview image re-projection unit uses a predetermined number (N) of left image pixels per scan line, a predetermined number (N) of right image pixels, a center-referenced depth map containing 2×

predetermined number (N)+1 pixels, and an arbitrary virtual camera reference image containing (a user'"'"'s first input signal I x predetermined number (N)+1) pixels.

8. The apparatus as claimed in claim 1, wherein the pixel selection unit comprises:

a first pixel buffer portion for storing a predetermined number (N) of scan line pixels input from the fist camera;

a first pixel pointer portion for storing an index of the first pixel buffer portion where a left image pixel to be mapped is stored;

a second pixel buffer portion for storing a predetermined number (N) scan line pixels input from the second camera;

a second pixel pointer portion for storing an index of the second pixel buffer portion where a right image pixel to be mapped is stored;

a second control portion for increasing an index of each of the first and second pixel pointer portions with respect to a difference value (Δ

d_i) of disparity which is input; and

an output portion for outputting a pixel of an image of an arbitrary virtual camera.

9. The apparatus as claimed in claim 8, wherein the output portion generates a pixel of a re-projection image by using the difference value (Δ

d_i) of disparity and a pixel that pixel pointers at the left and right side indicate.

10. The apparatus as claimed in claim 8, wherein the first pixel buffer portion and the second pixel buffer portion are initialized when a pixel of a scan line is processed by the processing portion of the multiview image synthesizing unit.