Apparatus and method for converting two-dimensional image sequence into three-dimensional image using conversion of motion disparity into horizontal disparity and post-processing method during generation of three-dimensional image

• US 6,314,211 B1
• Filed: 12/29/1998
• Issued: 11/06/2001
• Est. Priority Date: 12/30/1997
• Status: Expired due to Term

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Kim, Do-kyoon, Kim, Man-bae, Song, Mun-sup
• Primary Examiner(s): COUSO, YON JUNG
• Application Number:
  US09/222,713
• Time in Patent Office: 1,043 Days
• Field of Search: 382/285, 382/294, 382/278, 382/154, 382/236, 382/284, 348/42, 345/419, 345/425
• US Class Current: 382/285
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 10/10 Image acquisition document image scanning and transmission H04N1/00; control of digital cameras H04N23/60
• CPC Class Codes: G06V 2201/12 Acquisition of 3D measurements of objects
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/167 Synchronising or controlling image signals
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/221 using the relative movement between cameras and objects
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 13/286 having separate monoscopic and stereoscopic modes
• CPC Class Codes: H04N 13/341 using temporal multiplexing
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0085 Motion estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0092 Image segmentation from stereoscopic image signals

Abstract

An apparatus and a method for converting a two-dimensional image sequence into a three-dimensional image using a conversion of a motion disparity into a horizontal disparity and a post-processing method during the generation of a three-dimensional image are provided. The apparatus for converting a two-dimensional image sequence into a three-dimensional image according to the present invention includes a block motion measuring portion for measuring a motion vector for each block of a current image divided into blocks having a predetermined size using a previous image frame, a horizontal disparity generating portion for obtaining a horizontal disparity from the motion vector of each block according to the motion characteristic of the current image, an image generating portion for moving each block in a horizontal direction according to each horizontal disparity and generating a composite image corresponding to the current image, and an outputting portion for displaying a three-dimensional image formed of the current image and the composite image.

Claims

1. An apparatus for converting a two-dimensional image sequence into a three-dimensional image, comprising:

a block motion measuring portion for measuring a motion vector for each block of a current image divided into blocks having a predetermined size using a previous image frame;

a horizontal disparity generating portion for obtaining a horizontal disparity from the motion vector of each block according to the motion characteristic of the current image;

an image generating portion for moving each block in a horizontal direction according to each horizontal disparity and generating a composite image; and

an outputting portion for displaying a three-dimensional image comprised of the current image and the composite image.

2. The apparatus of claim 1, wherein, when a block having the same center position as an input current block and which is larger than the current block is called a second block, the block motion measuring portion detects a block similar to the second block in a search region on the previous frame and measures a difference of the center positions between the similar block and the current block as the motion of the current block.

3. The apparatus of claim 2, wherein the search region is at least twice as large as the current block.

4. The apparatus of claim 1, wherein the horizontal disparity generating portion comprises:

a horizontal disparity converting portion for converting the motion vector of each block measured by the block motion measuring portion into a horizontal disparity;

a horizontal disparity controlling portion for controlling the horizontal disparity converted by the horizontal disparity converting portion in response to a predetermined control signal; and

a motion characteristic determining portion for determining the motion characteristic of the current image from the motion vector of each block and generating the control signal according to the determination result.

5. The apparatus of claim 4, wherein the horizontal disparity converting portion calculates the magnitude of the motion vector and considers the calculated value as the horizontal disparity.

6. The apparatus of claim 4, wherein the horizontal disparity controlling portion outputs the horizontal disparity converted by the horizontal disparity converting portion as it is or subtracts the horizontal disparity from a maximum horizontal disparity in response to the control signal and outputs the subtraction result as the horizontal disparity of the respective blocks,

wherein the maximum horizontal disparity is determined by the block motion measuring portion.

7. The apparatus of claim 4, wherein the motion characteristic determining portion analyses the motion vector of each block, compares the magnitude of the entire motion of the current image with a predetermined reference value, and generates the control signal when the entire motion is larger than the reference value,

wherein the predetermined reference value represents the magnitude of the entire motion when a camera which captures the current image moves and a main subject in the current image moves.

8. The apparatus of claim 1, wherein the image generating portion comprises:

a block horizontal moving portion for moving the positions of all the blocks in the current image in a right horizontal direction by a horizontal disparity of each block; and

a block composing portion for composing all the blocks moved by the block horizontal moving portion and generating the composite image.

9. The apparatus of claim 8, wherein the image generating portion further comprises a block interpolating portion for interpolating regions not filled by all the blocks in the composite image generated by the block composing portion.

10. The apparatus of claim 1, wherein the outputting portion comprises a left/right image determining portion for outputting either the current image or the composite image as a left image signal and the other as a right image signal,

wherein the left image signal is displayed on the left eye and the right image signal is displayed on the right eye.

11. The apparatus of claim 1, further comprising:

an image block portion for dividing the current image into blocks having the predetermined size and outputting the blocks to the block motion measuring portion; and

a previous frame storing portion for storing the previous image frame and outputting the previous image frame to the block motion measuring portion.

12. The apparatus of claim 1, further comprising a motion normalizing portion for normalizing the motion measured in each block by the block motion measuring portion to have a correlation with the motions of peripheral blocks.

13. The apparatus of claim 12, wherein the motion normalizing portion replaces a weighted mean value between the magnitude of the motion of the current block measured by the block motion measuring portion and the magnitudes of the motions of the blocks adjacent to the current block by the magnitude of the motion of the current block.

14. The apparatus of claim 13, wherein a weighted value with respect to each adjacent block is set proportional to the similarity to the current block with respect to the magnitude of the motion and a distance from the current block.

15. The apparatus of claim 13, wherein a weighted value with respect to each adjacent block is set proportional to the similarity to the current block with respect to the magnitude of the motion and a distance from the current block and wherein the weighted value with respect to blocks vertically adjacent to the current block is set to be larger than blocks horizontally adjacent to the current block.

16. A method for converting a two-dimensional image sequence into a three-dimensional image, comprising the steps of:

(a) dividing an input current image into blocks having a predetermined size;

(b) measuring a motion vector for each block of the current image using a previous image frame;

(c) obtaining a horizontal disparity from the motion vector of each block according to the motion characteristic of the current image;

(d) generating a composite image by moving each block in a horizontal direction according to the each horizontal disparity; and

(e) displaying a three-dimensional image formed of the current image and the composite image.

17. The method of claim 16, wherein the step (b) comprises the steps of:

(b1) extracting a second block having the same center position as a current block the block motion of which is to be estimated and larger than the current block from the current image;

(b2) extracting a search region from the previous frame;

(b3) detecting a block similar to the second block in the search region; and

(b4) calculating a difference of center positions between the similar block and the current block,

wherein the difference is estimated to be the motion of the current block.

18. The method of claim 17, wherein the search region is at least twice as large as the current block.

19. The method of claim 16, wherein the step (c) comprises the steps of:

(c1) converting the motion vector of each block into a horizontal disparity;

(c2) determining the motion characteristic of the current image; and

(c3) controlling the horizontal disparity according to the motion characteristic.

20. The method of claim 19, wherein, in the step (c1), the magnitude of the motion vector is calculated and the calculated value is obtained as the horizontal disparity.

21. The method of claim 19, wherein, in the step (c2), the motion vector of each block is analyzed, the magnitude of the entire motion of the current image is compared with a predetermined reference value, and the comparison result is considered as the motion characteristic,

wherein the predetermined reference value represents the magnitude of the entire motion when a camera which captures the current image moves and a main subject in the current image moves.

22. The method of claim 21, wherein the step (c3) comprises the steps of:

outputting the horizontal disparity obtained in the step (c1) as it is when the entire motion of the current image is no more than a predetermined reference value; and

subtracting the horizontal disparity from a maximum horizontal disparity when the entire motion of the current image is larger than the predetermined reference value and outputting the subtraction result as the horizontal disparity of each block,

wherein the maximum horizontal disparity is determined in the step (b).

23. The method of claim 16, wherein the step (d) comprises the steps of:

(d1) moving the positions of all the blocks in the current image in a right horizontal direction by the horizontal disparity of each block; and

(d2) composing all the moved blocks and generating the composite image.

24. The method of claim 23, wherein the step (d) further comprises the step of interpolating regions not filled by all the blocks in the composite image.

25. The method of claim 16, wherein, in the step (e), either the current image or the composite image is displayed on the left eye and the other is displayed on the right eye.

26. The method of claim 17, wherein after the step (b) further comprises the steps of:

(b5) setting a weighted value with respect to each block adjacent to a current block to be normalized proportional to a similarity to the current block with respect to the magnitude of the motion and a distance from the current block;

(b6) obtaining a weighted mean value between the current block and the adjacent blocks; and

(b7) replacing the weighted mean value by the motion vector of the current block.

27. The method of claim 26, wherein, in the steps (b5) through (b7), the motion of the current block is normalized using the equation, $\begin{array}{c}t=\end{array}$



∑

i=0i=n

 



n-s

ti



-(t-ti)2



-((a-ai)2+(b-bi)2)/ 



∑

i=0i=n

 



n-s



-(t-ti)2



-((a-ai)2+(b-bi)2)

wherein when the set of the magnitudes of the motions of the blocks excluding predetermined s blocks from the n×

n (n>

0) adjacent blocks aligned in the order of the magnitude of the motion is represented to be {t₁, t₂, . . . , t<sub>n n−

s</sub>}, t, t_i, and (a_i,b_i) respectively represent the magnitude of the motion of the current block, the magnitude of an arbitrary block which belongs to the set, and the position of the block having t_i in the image.

28. The method of claim 26, wherein, in the steps (b5) through (b7), the motion of the current block is normalized using the equation, $\begin{array}{c}t=\end{array}$



∑

i=0i=n

 



n-s

ti(t-ti)2

 



((a-ai)2+(b-bi)2)/ 



∑

i=0i=n

 



n-s

1(t-ti)2

 



((a-ai)2+(b-bi)2)

wherein, when the set of the magnitudes of the motions of the blocks excluding predetermined s blocks from the n×

n (n>

0) adjacent blocks aligned in the order of the magnitude of the motion is represented to be {t₁, t₂, . . . , t<sub>n n−

s</sub>}, t, t_i, and (a_i,b_i) respectively represent the magnitude of the motion of the current block, the magnitude of an arbitrary block which belongs to the set, and the position of the block having t_i in the image.

29. The method of claim 17, wherein after the step (b) further comprises the steps of:

(b5) setting a weighted value with respect to each block adjacent to a current block to be normalized proportional to a similarity to the current block with respect to the magnitude of the motion and a distance from the current block;

(b6) setting a larger weighted value to blocks vertically adjacent to the current block than to blocks horizontally adjacent to the current block;

(b7) obtaining a weighted mean value between the current block and the adjacent blocks; and

(b8) replacing the weighted mean value by the motion vector of the current block.

31. The method of claim 17, wherein after the step (b) further comprises the steps of:

(b5) setting a weighted value with respect to each block adjacent to a current block to be normalized proportional to a similarity to the current block with respect to the magnitude of the motion and a distance from the current block;

(b6) obtaining a weighted mean value between the current block and the adjacent blocks; and

(b7) replacing the weighted mean value by the motion vector of the current block.

32. The method of claim 17, wherein after the step (b) further comprises the steps of:

(b5) setting a weighted value with respect to each block adjacent to a current block to be normalized proportional to a similarity to the current block with respect to the magnitude of the motion and a distance from the current block;

(b6) setting a larger weighted value to blocks vertically adjacent to the current block than to blocks horizontally adjacent to the current block;

(b7) obtaining a weighted mean value between the current block and the adjacent blocks; and

(b8) replacing the weighted mean value by the motion vector of the current block.

30. An apparatus for converting a two-dimensional image sequence into a three-dimensional image, comprising:

a block motion measuring portion for measuring a motion vector of each block of a current image divided into blocks having a predetermined size using a previous image frame;

a horizontal disparity generating portion for obtaining a horizontal disparity from the motion vector of each block according to the motion characteristic of the current image;

an image generating portion for moving each block in a horizontal direction according to each horizontal disparity and generating a composite image;

a post-processor for determining whether adjacent blocks belong to the same subject using the brightness characteristic of each block in the current image and post-processing the composite image according to the result; and

an outputting portion for displaying a three dimensional image comprised of the current image and the post-processed composite image.