Stereoscopic image pickup and display apparatus

• US 5,801,760 A
• Filed: 03/31/1997
• Issued: 09/01/1998
• Est. Priority Date: 08/26/1993
• Status: Expired due to Fees

• Current Assignee: Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Original Assignee: Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Inventors: Uomori, Kenya
• Primary Examiner(s): LEE, RICHARD J
• Application Number:
  US08/832,603
• Time in Patent Office: 519 Days
• Field of Search: 348/42, 348/45-48, 348/51-54, 359/466
• US Class Current: 348/47
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/167 Synchronising or controlling image signals
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 13/337 using polarisation multiplexing
• CPC Class Codes: H04N 13/346 using prisms or semi-transparent mirrors
• CPC Class Codes: H04N 13/363 using image projection screens volumetric display H04N13/388
• CPC Class Codes: H04N 13/376 for tracking left-right translational head movements, i.e. lateral movements
• CPC Class Codes: H04N 13/38 for tracking vertical translational head movements
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

Binocular parallax of a captured image is detected, and image pickup cameras are controlled so that their optic axes converge at an optimum point to ensure that the parallax of the captured image comes within the binocular fusing range of a viewer. Further, an optimum fixating point is calculated that enables the viewer to perceive the stereoscopic depth of the object over the widest possible range, and control is performed so that the fixating point is reproduced at a surface of a stereoscopic image display or at a designated distance from the surface. This suppresses the unnatural feel when the viewer views stereoscopic images.

Claims

1. A stereoscopic image pickup apparatus comprising:

an image pickup device for capturing pairs of images of an object from a plurality of viewpoints, each of said pairs of images referred to as a left image and a corresponding right image;

a converging angle moving mechanism for varying a converging angle of said image pickup device;

a signal processor for generating image data for said images captured by said image pickup device;

a parallax calculator for calculating a parallax of said object for individual pairs of images by directly using said image data from said signal processor relating only to a respective pair of images, free of the use of a pointer, and using a block-matching technique;

a parallax processor for detecting the smallest value of parallax, representing a parallax of a nearest object to said image pickup device, calculated by said parallax calculator;

a depth display position calculator for calculating from said smallest value of parallax detected by said parallax processor a depthwise position of said nearest object to be reproduced at a nearest point when a captured image is displayed by a stereoscopic image display device;

a fusing range verifier for verifying whether said depthwise position of said nearest object calculated by said depth display position calculator is within a fusional range of a viewer when the viewer views a reproduced image; and

a converging angle controller for controlling said converging angle moving mechanism, when said fusing range verifier determines that said depthwise position of said nearest object is not within the fusional range of the viewer, in order to change said converging angle so that said depthwise position of said nearest object is near a boundary of the fusional range of the viewer, thereby minimizing three-dimensional distortion of the image display device while maximizing display area that a viewer can fuse.

2. A stereoscopic image pickup apparatus according to claim 1, wherein

the parallax processor detects the largest value of parallax, representing a parallax of a farthest object from said image pickup device, calculated by said parallax calculator;

the depth display position calculator calculates from said smallest value of parallax detected by said parallax processor the depthwise position of said farthest object to be reproduced at a farthest point when the captured image is displayed by a stereoscopic image display device;

said fusing range verifier verifies whether said farthest point is within a fusional range of the viewer when the viewer views the reproduced image; and

said converging angle controller controls said converging angle moving mechanism, when said fusing range verifier determines that said depthwise position of said farthest object is not within the fusional range of the viewer, in order to change said converging angle so that said depthwise position of said farthest object is near a boundary of the fusional range of the viewer, thereby minimizing three-dimensional distortion of the image display device while maximizing display area that a viewer can fuse.

3. A stereoscopic image display apparatus comprising:

a parallax calculator for calculating and outputting a binocular parallax or three-dimensional position of an object directly from only left and right images of the object captured from at least two different angles, free of the use of a pointer, and using block-matching technique between plural viewpoint images, the images captured from the at least two different angles designated the right image and the left image;

a fixating point calculator, responsive to the output of the parallax calculator, for calculating a depthwise center position of the object; and

a parallax controller, responsive to the fixating point calculator, for moving the right and left images in opposite directions horizontally so that said depthwise center position indicated by said fixating point calculator is reproduced near a boundary of a viewer'"'"'s fusional limit such that the moving distance of the right image and the left image is minimized, thereby reducing the three-dimensional distortion of the displayed image.

4. A stereoscopic image display apparatus according to claim 3, wherein

said fixating point calculator calculates the depthwise position to be controlled by said parallax controller, by taking an average or a weighted average with greater weight given to the center of the screen, over an entire screen or over a small area within an image in the output of said parallax calculator.

5. A stereoscopic image display apparatus according to claim 3, wherein

said fixating point calculator calculates the depthwise position to be controlled by said parallax controller, by calculating the largest value or smallest value in an entire screen, or in a small area within an image, in the output of said parallax calculator.

6. A stereoscopic image display apparatus according to claim 3, wherein

said fixating point calculator includes a line-of sight detector for detecting a line of vision of a viewer;

a fixating point evaluator for detecting from the line of vision detected by said line-of-sight detector a fixating point in a displayed image on which the viewer'"'"'s fixating is focused; and

a control amount calculator for calculating an average value for binocular parallax at a required display image position from parallax values calculated by said parallax calculator and from the fixating point detected by said fixating point evaluator.

7. A stereoscopic image display apparatus according to claim 6, wherein

said fixating point evaluator extracts data only for a period in which a moving speed of the viewer'"'"'s line of vision is slow, and holds and low-pass filters the data to detect the viewer'"'"'s fixating point.

8. A stereoscopic image display apparatus according to claim 6, wherein

said control amount calculator calculates the depthwise position to be controlled by the parallax controller, by averaging the output of said parallax calculator over a prescribed range centering around the viewer'"'"'s fixating point in the displayed image indicated by the output of said fixating point evaluator.

9. A stereoscopic image display apparatus according to claim 6, wherein said parallax controller moves two images in opposite direction horizontally so that the output of said control amount calculator is reproduced near the boundary of a viewer'"'"'s fusional limit, and so that moving distance of right and left images becomes smallest for making three dimension distortion of displayed image smaller.

10. A stereoscopic image pickup and display apparatus comprising:

an image pickup device for capturing pairs of images of an object from a plurality of viewpoints, each of said pairs of images referred to as a left image and a corresponding right image;

a converging angle moving mechanism for varying a converging angle of said image pickup device;

a signal processor for generating image data for said images captured by said image pickup device;

a parallax calculator for calculating a parallax of said object for individual pairs of images by directly using said image data from said signal processor relating only to a respective pair of images, free of the use of a pointer;

a parallax processor for detecting the smallest value of parallax, representing a parallax of a nearest object to said image pickup device, calculated by said parallax calculator;

a depth display position calculator for calculating from said smallest value of parallax detected by said parallax processor a depthwise position of said nearest object to be reproduced at a nearest point when a captured image is displayed by a stereoscopic image display device;

a fusing range verifier for verifying whether said depthwise position of said nearest object calculated by said depth display position calculator is within a fusional range of a viewer when the viewer views a reproduced image; and

a converging angle controller for controlling said converging angle moving mechanism, when said fusing range verifier determines that said depthwise position of said nearest object is not within the fusional range of the viewer, in order to change said converging angle so that said depthwise position of said nearest object is near a boundary of the fusional range of the viewer, thereby minimizing three-dimensional distortion of the image display device while maximizing display area that a viewer can fuse;

a fixating point calculator for calculating a depthwise center position of the object on the basis of an output of said parallax calculator; and

a parallax controller for performing control so that said depthwise center position is reproduced near a boundary of a viewer'"'"'s fusional limit, and so that a moving distance of a right image and a left image becomes smallest, thereby reducing the three-dimensional distortion of the displayed image.