Methods and systems for 2D/3D image conversion and optimization

• US 20030128871A1
• Filed: 01/08/2003
• Published: 07/10/2003
• Est. Priority Date: 04/01/2000
• Status: Active Grant

• Current Assignee: Phoenix 3D, Inc.
• Original Assignee: NewSight Corp. (Hong Kong Winalite Group, Inc.)
• Inventors: Naske, Rolf-Dieter, Schwartz, Steven M., Hopewell, William
• Granted Patent: US 7,254,265 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/211 using temporal multiplexing
• CPC Class Codes: H04N 13/221 using the relative movement between cameras and objects
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 13/359 Switching between monoscopic and stereoscopic modes

Abstract

A method that improves compensation for hard cuts and/or vertical motion effects in the conversion of 2D images to 3D images, displays 2D images to one eye and computed images to the other eye. Generating the computed images includes using an approximation variable that represents some relationship between the 2D images, such as the speed of motion across two images. Compensating for the hard cut or vertical motion effect includes temporarily setting the approximation variable to a value that makes the left and right eye images more similar (if identical, displaying a 2D image). The approximation variable can be pre-set to a value based on the speed of motion in a scene to be displayed. A method of displaying images in either 2D or 3D may include converting 3D images to 2D images by displaying images computed from images intended for either the left eye or the right eye.

Claims

5. Apparatus for generating 3D images represented by 3D image data that includes at least two channels of data for respective left and right eye views from a sequence of 2D images represented by 2D image data, comprising:

i) assigning the 2D image data to a first channel;

ii) determining a measure of similarity (δ

_k) between sets of 2D image data representative of sequential 2D images;

iii) comparing the measure of similarity (δ

_k) to predetermined threshold values (δ

₀<

δ

₁<

δ

₂) where

(1) if δ

₀<

δ

_k<

δ

₂ and δ

_k−

δ

<sub>k−

1</sub>≦

−

δ

₁ and an approximation variable (α

) is less than or equal to Ks, the approximation variable is set to α

;

=α

+s,

(2) if δ

₀<

δ

_k<

δ

₂ and δ

_k−

δ

<sub>k−

1</sub>≧

δ

₁ and α

≧

s, the approximation variable (α

) is set to α

;

=α

−

s, and

(3) if δ

₀≧

δ

_k, the approximation variable (α

) is set to α

;

=0, where the approximation variable (α

) determines the width of a stereo base, and s denotes a step width;

iv) calculating 3D image data for a second channel from the 2D image data using the approximation variable (α

); and

v) assigning the first channel and the second channel to respective left and right eye views for display.

6. A system for the generation of 3D images represented by 3D image data that includes at least two channels of data for respective left and right eye views from a sequence of 2D images represented by 2D image data, comprising:

i) means for determining a measure of similarity (δ

_k) between sets of 2D image data representative of sequential 2D images;

ii) comparing the measure of similarity (δ

_k) to predetermined threshold values (δ

₀<

δ

₁<

δ

₂) where

(1) if δ

₀<

δ

_k<

δ

₂ and δ

_k−

δ

<sub>k−

1</sub>≦

−

δ

₁ and an approximation variable (α

) is less than or equal to Ks, the approximation variable is set to α

;

=α

+s,

(2) if δ

₀<

δ

_k<

δ

₂ and δ

_k−

δ

<sub>k−

1</sub>≧

δ

₁ and α

≧

s, the approximation variable (α

) is set to α

;

=α

−

s, and

(3) if δ

₀≧

δ

_k, the approximation variable (α

) is set to α

;

=0, where the approximation variable (α

) determines the width of a stereo base, and s denotes a step width;

iii) an image generator for calculating 3D image data for a second channel from the 2D image data using the approximation variable (α

).

7. A system as set forth in claim 6, further comprising means for displaying 3D images from the 3D image data.

8. A system as set forth in claim 6, further comprising a phase analyzer for determining the predominant direction of motion in the 2D image data.

9. A system as set forth in claim 6, further comprising a phase selector for assigning a current image and a calculated image to a left and right viewing channel, respectively, in dependence on the predominant direction of motion in sets of 2D image data representative of sequential 2D images.

10. A method for generating 3D images, comprising:

providing a first sequence of images for display,

computing a second sequence of images for display from one or more images of the first sequence using an approximation variable representative of a relationship between at least two images of the first sequence,

allowing the approximation variable to change in response to a relationship between at least two images of the first sequence,

checking for occurrence of a hard cut between images of the first sequence, and

in response to occurrence of a hard cut, setting the approximation variable to a value that provides a low degree of parallax between the respective images of the fist and second sequences of images.

11. The method of claim 10, further comprising displaying sequentially in alternating relation respective images of the first sequence and respective images of the second sequence to provide 3D images when there is no occurrence of hard cut.

12. The method of claim 11, further comprising detecting a hard cut as two successive substantially different images of the first sequence.

13. The method of claim 11, further comprising examining successive images of the first sequence to determine the extent of similarity thereof, and detecting occurrence of a hard cut when the difference between such successive images exceeds a threshold.

14. Apparatus for generating 3D images, comprising

means for providing a first sequence of images for display,

means for computing a second sequence of images for display from one or more images of the first sequence using an approximation variable representative of a relationship between at least two images of the first sequence and allowing the approximation variable to change in response to a relationship between at least two images of the first sequence,

means for checking for occurrence of a hard cut between images of the first sequence, and

means operative in response to occurrence of a hard cut for setting the approximation variable to a value that provides a low degree of parallax between the respective images of the fist and second sequences of images.

15. A method of compensating 3D image data in response to hard cut detection, comprising

preparing 3D image data for display from a first sequence of images and a second sequence of images computed from at least two respective images of the first sequence by using an approximation variable that changes based on a relationship between at least two images of the first sequence,

in response to occurrence of a hard cut in images of the first sequence, setting the approximation variable to a value that provides a low degree of parallax between the respective images of the fist and second sequences of images.

16. The method of claim 15, further comprising promptly after such setting of the approximation variable, allowing the approximation variable to change based on relationship between at least two images of the first sequence.

17. The method of claim 15, further comprising promptly after such setting of the approximation variable, resetting the approximation variable to a predetermined value.

18. The method of claim 17, wherein resetting the approximation variable includes resetting the approximation variable to a predetermined value to optimize the viewing experience.

19. The method of claim 17, further comprising after such resetting of the approximation variable, allowing the approximation variable to change based on relationship between at least two images of the first sequence.

20. The method of claim 17, further comprising regulating the amount of each change of the approximation variable in the computation of successive images of the second sequence.

21. The method of claim 20, wherein regulating continues until the approximation variable returns to the value of the approximation variable just prior to the setting step.

22. The method of claim 20, wherein regulating governs the amount of each change to provide a smooth visual appearance of images.

23. The method of claim 22, wherein setting includes setting the approximation variable to zero.

24. The method of claim 22, further comprising prior to the setting step storing the value of the approximation variable.

25. The method of claim 16, further comprising regulating the amount of each change of the approximation variable in the computation of successive images of the second sequence until the approximation variable reaches a target value.

26. Apparatus for compensating 3D image data in response to hard cut detection, comprising

means for preparing 3D image data for display from a first sequence of images and a second sequence of images computed from at least two respective images of the first sequence by using an approximation variable that changes based on a relationship between at least two images of the first sequence,

means responsive to occurrence of a hard cut in images of the first sequence for setting the approximation variable to a value that provides a low degree of parallax between the respective images of the fist and second sequences of images.

27. A method for generating 3D images, comprising

preparing 3D image data from a first sequence of images and a second image sequence of images computed from at least two images of the first sequence using an approximation variable that changes based on a relationship between at least two images of the first sequence, and

reducing the value of the approximation variable upon the occurrence of vertical disparity between at least two images of the first sequence.

1. A method for the generation of 3D images represented by 3D image data that includes at least two channels of data for respective left and right eye views from a sequence of 2D images represented by 2D image data, comprising the steps of:

i) assigning the 2D image data to a first channel;

ii) determining a measure of similarity (δ

_k) between sets of 2D image data representative of sequential 2D images;

iii) comparing the measure of similarity (δ

_k) to predetermined threshold values (δ

₀<

δ

₁<

δ

₂) where

(1) if δ

₀<

δ

_k<

δ

₂ and δ

_k−

δ

<sub>k−

1</sub>≦

−

δ

₁ and an approximation variable (α

) is less than or equal to Ks, the approximation variable is set to α

;

=α

+s,

(2) if δ

₀<

δ

_k<

δ

₂ and δ

_k−

δ

<sub>k−

1</sub>≧

δ

₁ and α

≧

s, the approximation variable (α

) is set to α

;

=α

−

s, and

(3) if δ

₀≧

δ

_k, the approximation variable (α

) is set to α

;

=0, where the approximation variable (α

) determines the width of a stereo base, and s denotes a step width;

iv) calculating 3D image data for a second channel from the 2D image data using the approximation variable (α

); and

v) assigning the first channel and the second channel to respective left and right eye views for display.

2. A method as set forth in claim 1, further comprising displaying 3D images from the 3D image data in the first channel and the second channel.

3. A method as set forth in claim 1, wherein the step of assigning the first channel and the second channel to respective left and right eye views includes determining the predominant direction of motion in the 2D image data.

4. A method as set forth in claim 1, further comprising repeating the steps of determining, comparing and calculating.

28. The method of claim 27, wherein reducing includes reducing the approximation variable upon the occurrence of vertical disparity over a number of images of the first sequence, wherein the number of images exceeds two.

29. The method of claim 27, wherein reducing includes reducing the approximation variable upon the vertical disparity exceeding a predetermined threshold.

30. The method of claim 27, wherein reducing the approximation variable includes reducing the approximation variable to zero.

31. The method of claim 27, further comprising promptly after such reducing of the approximation variable, allowing the approximation variable to change based on relationship between at least two images of the first sequence.

32. The method of claim 31, further comprising regulating the amount of each change of the approximation variable in the computation of successive images of the second sequence until the approximation variable returns to the value of the approximation variable just prior to the reducing step.

33. The method of claim 30, wherein setting includes setting the approximation variable to zero.

34. The method of claim 33, further comprising prior to the setting step storing the value of the approximation variable.

35. The method of claim 27, wherein the amount of vertical disparity is used to determine the value of the reduction of the approximation variable.

34-1. The method of claim 31, further comprising regulating the amount of each change of the approximation variable in the computation of successive images of the second sequence until the approximation variable reaches a target value.

36. Apparatus for generating 3D images, comprising

means for preparing 3D image data from a first sequence of images and a second image sequence of images computed from at least two images of the first sequence using an approximation variable that changes based on a relationship between at least two images of the first sequence, and

means for reducing the value of the approximation variable upon the occurrence of vertical disparity between at least two images of the first sequence.

37. A method of converting 3D image data for a 3D image that includes image data representative of respective left and right eye views to 2D image data for a 2D image, comprising

i) using the 3D image data representative of one of the left and right eye views to provide first image data representative of a 2D image, and

ii) computing computed image data representative of a subsequent 2D image from the first image data.

38. The method of claim 37, further comprising discarding the image data representative of the other eye view.

39. The method of claim 37, further comprising sequentially interleaving computed image data with respective first image data thereby to obtain sequential image data for 2D image display.

40. The method of claim 37, wherein computing includes pixel by pixel averaging of the image data for alternate lines of image data.

41. The method of claim 37, wherein computing includes pixel by pixel interpolating the image data for alternate lines of image data.

42. The method of claim 41, interpolating includes taking an average of the image data of respective pixels in one line of image data and the image data of respective pixels in a succeeding line of image data to obtain image data representing respective pixels on the intermediate line between the one line and the succeeding line.

43. The method of claim 37, wherein the data representative of the left and right eye views is interlaced data, whereby respective eye views are provided by image data on alternate lines, and computing includes averaging the image data on alternate lines to obtain image data for the lines between respective pairs of alternate lines.

44. The method of claim 37, further comprising displaying images representative of the first image data and the computed image data.

45. A device for converting 3D image data, which includes image data representative of respective left and right eye views of a 3D image, to 2D image data, comprising

i) an output providing the image data that is representative of one of the respective left and right eye views of the 3D image data as 2D output image data for a 2D image, and

ii) a computation device to compute from the image data representative of one of the respective left and right eye views computed image data that is representative of a subsequent 2D image.

46. The device of claim 45, further comprising a display for displaying 2D images using the 2D output image data and the computed image data.

47. The device of claim 45, further comprising a data storage device to store image data in interlaced format, the output device providing image data from alternate lines of data stored in the data storage device, and the computation device being operative to compute computed image data from respective sequential pairs of alternate lines of data.

48. A method of providing images for display, comprising

i) receiving image data in either of 2D format or 3D format, and

ii) selectively outputting in either 2D format or 3D format output image data for display, wherein selectively outputting 2D format output image data from 3D image data includes outputting at least some output image data computed from one eye view of the 3D format image data.

49. The method of claim 48, further comprising displaying images representative of such output image data.

50. The method of claim 48, further comprising selecting whether the received image data is in 2D format or 3D format.

51. The method of claim 50, further comprising selecting whether the output image data is in 2D format or 3D format regardless of the 2D or 3D format of the input image data.

52. A system for use in displaying images, comprising

i) an input for receiving image data in either of 2D or 3D format,

ii) a converter to convert 2D image data to 3D image data and to convert 3D image data to 2D image data, and

iii) a selector operable to select the format of output image data, including output image data in response to 2D input image data,

output image data in response to 3D input image data, output image data converted by the converter from 3D input image data, and output image data converted by the converter from 2D input image data.

53. The system of claim 52, further comprising a computation device to compute from said image data representative of said one of said respective left and right eye views computed image data that is representative of a subsequent 2D image.

54. A method for generating 3D images, comprising

preparing 3D image data from a first sequence of images and a second sequence of images computed from at least two images of the first sequence using an approximation variable that changes based on a relationship between at least two images of the first sequence, and

presetting the initial value of the approximation variable according to a property of the anticipated motion across at least two images of the first sequence.

55. The method of claim 54, wherein presetting includes presetting according to anticipated speed of motion.

56. The method of claim 54, wherein presetting includes presetting according to the quantity of motion.

57. The method of claim 54, further comprising selecting the type of motion to be present from a selection.

58. The method of claim 54, further comprising presetting other parameters that affect the computation of the second sequence of images based on the anticipated motion.

59. The method of claim 54, wherein presetting includes presetting at least one of parallax and warp.

60. The method of claim 54, further comprising setting the preset initial value of the approximation variable as a target value of the approximation variable.

61. The method of claim 60, further comprising following resolution of the occurrence of either a hard cut or vertical motion beyond prescribed limits, allowing the approximation variable to increase in at least one of the following ways:

i) immediately to the target value,

ii) immediately to a value that existed just prior to the decreasing step,

iii) in stepwise fashion, and

iv) to a low parallax value.

62. The method of claim 54, wherein presetting includes presetting at least one of the approximation value, parallax and warp parameters based on the following scene types,

i) Music video

ii) Action

iii) Talk Show

iv) Wildlife.

63. The method of claim 62, wherein presetting includes presetting according to the following table:

i) for Music Video, approximation variable to 1.1, parallax to 4, and warp to lenticular,

ii) for Action, approximation variable to 2.1, parallax to 5, and warp to assymetric,

iii) for Talk Show, approximation variable to 4.3, parallax to 6, and warp to assymetric,

iv) for Wildlife, approximation variable to 6, parallax to 4, and warp to tilt.