Stereoscopic image processing apparatus and method, stereoscopic vision parameter setting apparatus and method, and computer program storage medium information processing method and apparatus

• US 20030179198A1
• Filed: 02/25/2000
• Published: 09/25/2003
• Est. Priority Date: 07/08/1999
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Uchiyama, Shinji
• Granted Patent: US 6,677,939 B2
• Time in Patent Office: Days
• US Class Current: 345/427
• CPC Class Codes: H04N 13/305 using lenticular lenses, e.g. arrangements of cylindrical lenses
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A stereoscopic vision parameter is determined on the basis of the 3D space data to be displayed and the disparity distance at which the left- and right-eye images look displaced from each other in a predetermined 2D rendering space. In determining a parameter, the disparity distance between left and right image data of an object point in the 3D space is set in advance, which appears when the image data are rendered in the 2D rendering space, and the position of a displaced point which is separated from the coordinates of the object point in the 2D rendering space in the base line length direction of the observer by the disparity distance is computed. The coordinates of this displaced point in a 3D space are computed. In addition, the base line length is determined on the basis of the distance between the 3D coordinates of the displaced point and the object point.

Claims

1. A stereoscopic image processing apparatus for displaying an arbitrary object so as to obtain stereoscopic vision of said object, characterized by comprising:

parameter determining means for determining a stereoscopic vision parameter on the basis of 3D space data to be displayed and displacement information representing a displacement amount between the left- and right-eye images in a 2D rendering space; and

output means for generating and outputting the left- and right-eye images for stereoscopic vision output by using the determined stereoscopic vision parameter.

2. The apparatus according to claim 1, characterized in that said parameter determining means comprises:

displacement distance setting means for presetting a displacement distance between left and right image data of a predetermined object point of the 3D space, which appears when the image data are rendered in the 2D rendering space;

displacement computing means for computing a position of a displaced point separated from coordinates of the object point in the 2D rendering space by the displacement distance in a base line length direction of an observer;

coordinate transformation means for computing coordinates of the displaced point in a 3D space; and

base line length determining means for determining a base line length on the basis of a distance between 3D coordinates of the displaced point and the object point.

3. The apparatus according to claim 1, characterized in that said apparatus further comprises moving means for moving a viewpoint position of the observer, and

an initial value of a stereoscopic vision parameter which is determined by said parameter determining means is used for stereoscopic vision output regardless of movement of the viewpoint position.

4. The apparatus according to claim 1, characterized in that said parameter determining means is restarted to recalculate a value of a stereoscopic vision parameter at an arbitrary time point.

5. The apparatus according to claim 2, characterized in that a plurality of search positions in the rendering space are set to have a discrete distribution so as to quickly search for the predetermined object point that satisfies a predetermined condition in the rendering space.

6. The apparatus according to claim 1, characterized in that the stereoscopic vision parameter includes at least one of a base line length and vergence angle.

7. The apparatus according to claim 3, characterized in that said moving means comprises a sensor for detecting a moving amount of an actual viewpoint position of the observer.

8. The apparatus according to claim 3, characterized in that said moving means comprises a sensor for detecting a moving amount of a virtual viewpoint position of the observer.

9. The apparatus according to claim 3, characterized in that if the moving amount of the viewpoint position of the observer exceeds a predetermined value, said parameter determining means is restarted to recalculate a corresponding value of a stereoscopic vision parameter.

10. The apparatus according to claim 3, characterized in that said parameter determining means is restarted in response to a request from the observer to recalculate a corresponding value of a stereoscopic vision parameter at the viewpoint position of the observer.

11. The apparatus according to claim 1, characterized by further comprising user interface means for manually setting a value of a stereoscopic vision parameter.

12. The apparatus according to claim 11, characterized by further comprising means for arbitrating between said user interface means and said stereoscopic vision parameter setting means.

13. The apparatus according to claim 1, characterized in that said parameter determining means obtains a spread of an input virtual space in a depth direction, and determines a vergence angle on the basis of a direction of a substantially a midpoint position in the spread in the virtual space in the depth direction which is viewed from left and right viewpoint positions of the observer.

14. The apparatus according to claim 2, characterized in that the predetermined object point is set at a nearest point on the object which is substantially nearest to the viewpoint position of the observer in a depth direction of a virtual space.

15. The apparatus according to claim 1, characterized in that the displacement distance is determined in advance on the basis of a pixel size of a display apparatus on which a stereoscopic image is displayed and a distance from a display screen of said display apparatus to the observer.

16. The apparatus according to claim 1, characterized in that said apparatus comprises a memory for storing predetermined displacement distance values in advance in accordance with combinations of a plurality of types of display apparatuses that can be connected to said stereoscopic image processing apparatus and distances from display screens to the observer, and

means for detecting a distance from the display screen to the observer, and

said parameter setting means reads out a displacement distance value matching a distance from the connected display apparatus to the observer from said memory.

17. The apparatus according to claim 14, characterized in that the nearest point is determined to be a pixel having a minimum depth value which is selected from image data obtained by perspective transformation of 3D space data from the viewpoint of the observer'"'"'s left- or right-eye.

18. The apparatus according to claim 2, characterized in that said displacement computing means comprises:

means for performing perspective transformation of 3D image data from the viewpoint of the observer'"'"'s left- or right-eye;

nearest point determining means for determining, as the nearest point, a pixel, of the image data subjected to perspective transformation by said means, which has a minimum depth value;

means for computing a position of a displaced point, in the 2D rendering space, which is separated from coordinates of the pixel having the minimum depth value by the displacement distance in the base line length direction of the observer; and

means for obtaining coordinates of the displaced point in a 3D space by inverse perspective transformation of image at a position of the displaced point in the 2D rendering space from another viewpoint.

19. The apparatus according to claim 17, characterized by further comprising a z buffer memory for storing a depth value.

20. The apparatus according to claim 17, characterized in that a distribution pattern of a plurality of search points in the 2D rendering space is set with a density of search points increasing toward a center so as to search for the nearest point in the 2D rendering space.

21. The apparatus according to claim 17, characterized in that a plurality of distribution patterns of a plurality of search points in the 2D rendering space are set to search for the nearest point in the 2D rendering space, and

said apparatus comprises means for detecting a viewpoint position/posture of the observer and selecting a distribution pattern matching the position/posture.

24. A stereoscopic image display apparatus comprising said stereoscopic image processing apparatus defined in claim 1.

22. A stereoscopic vision parameter setting apparatus for determining a stereoscopic vision parameter for stereoscopically displaying an arbitrary object, characterized by comprising:

parameter determining means for determining a stereoscopic vision parameter on the basis of 3D space data to be displayed and displacement information representing a displacement distance between the left- and right-eye images in a 2D rendering space,

wherein the determined stereoscopic vision parameter is stored or output outside to be used for stereoscopic vision afterward.

23. The apparatus according to claim 22, characterized by further comprising:

a memory for storing a determined value of a stereoscopic vision parameter; and

means for updating the value of the stereoscopic vision parameter and storing the update result in said memory.

25. A stereoscopic image processing method of displaying an arbitrary object so as to obtain stereoscopic vision of said arbitrary object, characterized by comprising:

the parameter determining step of determining a stereoscopic vision parameter on the basis of 3D space data to be displayed and a displacement distance at which the left- and right-eye images look displaced from each other in a predetermined 2D rendering space; and

the output step of generating and outputting the left- and right-eye images for stereoscopic vision output by using the determined stereoscopic vision parameter.

26. The method according to claim 25, characterized in that the parameter determining step comprises:

the displacement distance setting step of presetting a displacement distance between left and right image data of a predetermined object point of the 3D space, which appears when the image data are rendered in the 2D rendering space;

the displacement computing step of computing a position of a displaced point separated from coordinates of the object point in the 2D rendering space by the displacement distance in a base line length direction of an observer;

the coordinate transformation step of computing coordinates of the displaced point in a 3D space; and

the base line length determining step of determining a base line length on the basis of a distance between 3D coordinates of the displaced point and the object point.

27. The method according to claim 25, characterized in that said method further comprises the moving step of moving a viewpoint position of the observer, and

an initial value of a stereoscopic vision parameter which is determined in the parameter determining step is used for stereoscopic vision output regardless of movement of the viewpoint position.

28. The method according to claim 25, characterized in that the parameter determining step is restarted to recalculate a value of a stereoscopic vision parameter at an arbitrary time point.

29. The method according to claim 26, characterized in that a plurality of search positions in the rendering space are set to have a discrete distribution so as to quickly search for the predetermined object point that satisfies a predetermined condition in the rendering space.

30. The method according to claim 25, characterized in that the stereoscopic vision parameter includes at least one of a base line length and vergence angle.

31. The method according to claim 27, characterized in that the moving step comprises a sensor for detecting a moving amount of an actual viewpoint position of the observer.

32. The method according to claim 27, characterized in that the moving step comprises a sensor for detecting a moving amount of a virtual viewpoint position of the observer.

33. The method according to claim 27, characterized in that if the moving amount of the viewpoint position of the observer exceeds a predetermined value, the parameter determining step is restarted to recalculate a corresponding value of a stereoscopic vision parameter.

34. The method according to claim 27, characterized in that the parameter determining step is restarted in response to a request from the observer to recalculate a corresponding value of a stereoscopic vision parameter at the viewpoint position of the observer.

35. The method according to claim 25, characterized by further comprising the user interface step of manually setting a value of a stereoscopic vision parameter.

36. The method according to claim 35, characterized by further comprising the step of arbitrating between the user interface step and the stereoscopic vision parameter setting step.

37. The method according to claim 25, characterized in that the parameter determining step comprises obtaining a spread of an input virtual space in a depth direction, and determining a vergence angle on the basis of a direction of a substantially a midpoint position in the spread in the virtual space in the depth direction which is viewed from left and right viewpoint positions of the observer.

38. The method according to claim 26, characterized in that the predetermined object point is set at a nearest point on the object which is substantially nearest to the viewpoint position of the observer in a depth direction of a virtual space.

39. The method according to claim 25, characterized in that the displacement distance is determined in advance on the basis of a pixel size of a display apparatus on which a stereoscopic image is displayed and a distance from a display screen of said display apparatus to the observer.

40. The method according to claim 25, characterized in that the method comprises a memory for storing predetermined displacement distance values in advance in accordance with combinations of a plurality of types of display apparatuses that can be connected to the stereoscopic image processing method and distances from display screens to the observer, and

the step of detecting a distance from the display screen to the observer, and

the parameter setting step comprises reading out a displacement distance value matching a distance from the connected display apparatus to the observer from said memory.

41. The method according to claim 38, characterized in that the nearest point is determined to be a pixel having a minimum depth value which is selected from image data obtained by perspective transformation of 3D space data from the viewpoint of the '"'"'s left- or right-eye.

42. The method according to claim 26, characterized in that the displacement computing step comprises:

the step of performing perspective transformation of 3D space data from the viewpoint of the observer'"'"'s left- or right-eye;

the nearest point determining step of determining, as the nearest point, a pixel, of the image data subjected to perspective transformation in the step, which has a minimum depth value;

the step of computing a position of a displaced point, in the 2D rendering space, which is separated from coordinates of the pixel having the minimum depth value by the displacement distance in the base line length direction of the observer; and

the step of obtaining coordinates of the displaced point in a 3D space by inverse perspective transformation of one of the left- and right-eye images from another viewpoint at a position of the displaced point in the 2D rendering space.

43. The method according to claim 41, characterized by further comprising a z buffer memory for storing a depth value.

44. The method according to claim 41, characterized in that a distribution pattern of a plurality of search points in the 2D rendering space is set with a density of search points increasing toward a center so as to search for the nearest point in the 2D rendering space.

45. The method according to claim 41, characterized in that a plurality of distribution patterns of a plurality of search points in the 2D rendering space are set to search for the nearest point in the 2D rendering space, and

the method comprises the step of detecting a viewpoint position/posture of the observer and selecting a distribution pattern matching the position/posture.

48. A computer program storage medium storing a stereoscopic image processing program which is executed on a computer to display an arbitrary object so as to obtain stereoscopic vision of said object,

said medium storing a program code for implementing the stereoscopic image processing method defined in claim 25.

46. A stereoscopic vision parameter setting method of determining a stereoscopic vision parameter for stereoscopically displaying an arbitrary object, characterized by comprising:

the parameter determining step of determining a stereoscopic vision parameter on the basis of 3D space data to be displayed and a displacement distance at which the left- and right-eye images look displaced from each other in a predetermined 2D rendering space,

wherein the determined stereoscopic vision parameter is stored or output outside to be used for stereoscopic vision afterward.

47. The method according to claim 46, characterized by further comprising:

a memory for storing a determined value of a stereoscopic vision parameter; and

the step of updating the value of the stereoscopic vision parameter and storing the update result in said memory.

49. A computer program storage medium storing a parameter setting program which is executed on a computer to determine a stereoscopic vision parameter for stereoscopically displaying an arbitrary object,

said medium storing a program code for implementing the stereoscopic image processing method defined in claim 46.