Method and apparatus for aligning stereo images

• US 6,191,809 B1
• Filed: 01/15/1998
• Issued: 02/20/2001
• Est. Priority Date: 01/15/1998
• Status: Expired due to Term

• Current Assignee: Viking Systems, Inc.
• Original Assignee: Vista Point Technologies Incorporated (Flex, Ltd.)
• Inventors: Hori, Koichiro, Thaler, Herbert A.
• Primary Examiner(s): Britton, Howard
• Assistant Examiner(s): DIEP, NHON THANH
• Application Number:
  US09/007,469
• Time in Patent Office: 1,132 Days
• Field of Search: 348/42, 348/45, 348/46, 348/47, 348/48, 348/65, 348/66, 348/79, 348/80, 600/109, 600/111
• US Class Current: 348/45
• CPC Class Codes: A61B 1/00193 adapted for stereoscopic vision
• CPC Class Codes: G02B 23/2415 Stereoscopic endoscopes
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/167 Synchronising or controlling image signals
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 13/337 using polarisation multiplexing
• CPC Class Codes: H04N 13/341 using temporal multiplexing
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays

Abstract

A method and apparatus are provided for correcting for optical misalignment of the dual (left and right) images produced by a stereo electronic endoscope, or of the corresponding dual images produced by coupling electronic imaging devices to each of two monocular teaching ports of a stereo optical endoscope. The stereo image alignment technique comprises electronically capturing the video image data of the dual images, and subsequently processing that data electronically to correct for optical alignment errors. The method involves digitizing the electronic data and digitally performing the equivalent of vertical image shift, and/or image size change, and/or image rotation as required to correct for any visual image misalignment.

Claims

1. A stereo endoscope or microscope comprising:

an objective optical system for viewing an object and forming and transmitting first and second images of said object having a parallax between said first and second images;

first and second imaging means coupled to said objective optical system for picking up said first and second images respectively and producing first and second output signals representative of said first and second images respectively;

signal processing means for converting said first and second output signals to first and second video signals respectively;

image-correcting means responsive to said first and second video signals for (a) comparing said first and second video signals to determine misalignment of said images relative to one another, and (b) modifying one or both of said video signals to correct for misalignment of said first and second images relative to one another; and

an electronic display means responsive to the first and second video signal output of said image-correcting means for reproducing said first and second images and displaying said reproduced images either alternately or concurrently for viewing as a 3-D image.

2. The stereo endoscope or microscope according to claim 1 wherein said first and second video signals are analog video signals representative of said first and second images on a frame-by-frame basis, and further wherein said image-correcting means comprises:

means for converting said first and second analog video signals to first and second digital data on a frame by frame basis;

first and second memory units for receiving and storing said first and second digital data;

write control means for controlling the writing of said first and second digital data into said first and second memory units;

readout control means for controlling the reading out of said first and second digital data from said first and second memory units;

means coupled to said first and second memory units for driving said electronic display means in response to said first and second digital data read out from said memory units;

image analysis means coupled to said first and second memory units for comparing said first and second digital data as read out from said memory units to measure any relative differences in the (a) sizes, and/or (b) horizontal and vertical positions, and/or (c) rotational positions of said first and second images and for producing at least one correctional signal according to said relative differences; and

means for applying said at least one correctional signal to said readout control means as a readout command so as to adjust the readout of digital data from at least one of said first and second memory units in a manner that eliminates said differences in the images represented by subsequent first and second digital data fed to said first and second memory units.

3. The stereo endoscope or microscope according to claim 2 wherein each of said first and second memory units is a dual port memory unit having first and second ports, with data being read into said memory units via said first ports and data being read out of said memory units via said second ports.

4. A stereo endoscope or microscope apparatus comprising:

a pair of optical systems for transmitting an image of an object as first and second images having a parallax between said first and second images;

a pair of imaging means for picking up said first and second images as transmitted by said optical systems and producing first and second output signals representative of said first and second images respectively;

a display means for displaying said two images either alternately or concurrently for viewing as a 3-D image;

image-correcting means responsive to said first and second output signals for correcting for misalignment of said first and second images relative to one another so that said images coincide with one another as displayed by said display means, said image-correcting means comprising first and second signal processing channels for processing said first and second output signals and producing first and second analog video signals therefrom, means for converting said first and second analog video signals to first and second digital data on a frame by frame basis;

first and second memory units coupled to said first and second signal processing channels for receiving and storing said first and second digital data;

write control means for controlling the writing of said first and second digital data into said first and second memory units;

readout control means for controlling the reading out of said first and second digital data from said memory units;

means coupled to said first and second memory units for driving a stereo display device in response to said first and second digital data read out from said memory units;

image analysis means coupled to said first and second memory units for comparing said first and second digital data as read out from said memory units to measure any relative differences in the sizes, and/or horizontal and vertical positions, and/or rotational positions of said first and second images and for producing first and second correctional signals according to said relative differences; and

means for applying said first and second correctional signals to said readout control means as readout commands so as to adjust the readout of digital data from at least one of said first and second memory units in a manner that eliminates said differences in images represented by subsequent first and second digital data fed to said first and second memory units.

5. A stereo microscope comprising:

first and second optical channels for viewing an object and forming and transmitting first and second images of said object having a parallax between said first and second images;

viewing means coupled to said first and second optical channels for displaying said first and second images for viewing as a 3-D image;

first and second imaging means coupled to said first and second optical channels respectively for picking up said first and second images respectively and producing first and second output signals representative of said first and second images respectively;

means for converting said first and second output signals to first and second digital video signals respectively;

image-correcting means responsive to said first and second digital video signals for (a) comparing said digital video signals to determine any misalignment of said images relative to one another, and (b) modifying said first and second video signals to correct for misalignment of said first and second images relative to one another; and

an electronic display means responsive to said modified first and second video signals for reproducing said first and second images and displaying said reproduced images either alternately or concurrently for viewing as a 3-D image.

6. A microscope according to claim 5 wherein said image-correcting means comprises first and second signal processing channels for processing said first and second output signals and producing first and second analog video signals therefrom, means for converting said first and second analog video signals to first and second digital data on a frame by frame basis;

first and second memory units coupled to said first and second signal processing channels for receiving and storing said first and second digital data;

write control means for controlling the writing of said first and second digital data into said first and second memory units;

readout control means for controlling the reading out of said first and second digital data from said memory units;

means coupled to said first and second memory units for driving said electrical display means in response to said first and second digital data read out from said memory units;

image analysis means coupled to said first and second memory units for comparing said first digital data as read out from said first memory unit to said second digital data as read out from said second memory unit so as to measure any relative differences in the sizes, and/or horizontal and vertical positions, and/or rotational positions of said first and second images and for producing first and second correctional signals according to said relative differences; and

means for applying said first and second correctional signals to said readout control means as readout commands so as to adjust the readout of digital data from at least one of said first and second memory units in a manner that eliminates said differences in images represented by subsequent first and second digital data fed to said first and second memory units, whereby said reproduced images coincide with one another and are presented for viewing by said display means as a 3-D image.

7. The microscope according to claim 6 wherein each of said first and second memory units has first and second ports, with data being read into said memory units via said first ports and data being read out of said memory units via said second ports.

15. Apparatus according to claim 6 wherein said modified first and second video signals constitute the said first and second digital data read out of said first and second memory units.

8. An apparatus for correcting for misalignment of the separate left and right images of a stereo electronic camera having a first and second optical means for transmitting an image of an object as first and second images having a parallax between said first and second images, and first and second imaging means for picking up said first and second images as transmitted by said optical systems and producing first and second output signals representative of said first and second images respectively said apparatus comprising:

first and second signal processing channels comprising means for generating first and second analog video signals in response to said first and second output signals respectively, said first and second video signals being representative of said first and second images on a frame by frame basis;

an electronic display means;

display operating means responsive to said first and second video signals for causing said display means to reproduce said first and second images on a frame by frame basis and display said reproduced first and second images either alternately or concurrently for viewing as a 3-D image;

image-correcting means for correcting for misalignment of said first and second images relative to one another so that said reproduced first and second images appear coincident with one another as a 3-D image as displayed by said display means, said image-correcting means comprising first and second means for converting said first and second video signals to first and second digital data representative of said first and second images respectively on a frame by frame basis, first and second memory units for receiving and storing said first and second digital data;

write control means for controlling the writing of said first and second digital data into said first and second memory units;

readout control means for controlling the reading out of said first and second digital data from said memory units;

image analysis means coupled to said first and second memory units for comparing said first and second digital data as read out from said memory units to measure any relative differences in the sizes, horizontal and vertical positions, and rotational positions of said first and second images and for producing first and second correctional signals according to said relative differences; and

means for applying said first and second correctional signals to said readout control means as readout commands so as to adjust the readout of digital data from at least one of said first and second memory units in a manner that eliminates any of said differences in images represented by subsequent first and second digital data fed to said first and second memory units respectively.

16. Apparatus according to claim 8 wherein each of said first and second memory units is a dual port memory unit having first and second ports, with data being read into said memory units via said first ports and data being read out of said memory units via said second ports.

9. A method of correcting for misalignment of the separate left and right images of a stereo electronic camera having a first and second optical means for transmitting an image of an object as first and second images having a parallax between them, and first and second imaging means for picking up said first and second images as transmitted by said optical systems and producing first and second output signals representative of said first and second images respectively, said method comprising the steps of:

processing said first and second output signals so as to produce first and second analog video signals therefrom on a frame by frame basis;

converting said first and second analog video signals to first and second digital data on a frame by frame basis;

writing said first and second digital data to first and second memory units on a frame by frame basis;

reading out said first and second digital data from said first and second memory units on a frame by frame basis;

analyzing and comparing said read first and second data on a frame by frame basis to measure any relative differences in the sizes, horizontal and vertical positions and rotational positions of said first and second images and producing in response to said read first and second data first and second correctional signals according to said relative differences;

applying said first and second correctional signals as readout commands to said first and second memory units respectively so that the readout of digital data from at least one of said first and second memory units representing subsequent successive frames of said first and second images is adjusted in a manner that eliminates said differences in said successive frames of said first and second images.

10. The method of claim 9 further including the step of generating a visual display of said first and second images on a frame by frame basis in response to said first and second digital data read out from said memory units.

11. The method of claim 10 wherein said first and second digital data read out from said memory units also is converted to third and fourth analog video signals respectively, and said visual display is generated in response to said third and fourth analog video signals.

12. The method of claim 9 wherein said stereo camera is part of an endoscope.

13. The method of claim 9 wherein said stereo camera is part of a stereo microscope.

14. The method of claim 9 wherein each of said first and second memory units has first and second ports, with data being read into said memory units via said first ports and data being read out of said memory units via said second ports.

17. An apparatus for correcting for misalignment of the separate left and right images of a stereo electronic camera having a first and second optical means for transmitting an image of an object as first and second images with a parallax between said first and second images, and first and second imaging means for picking up said first and second images as transmitted by said optical systems and producing first and second output signals representative of said first and second images respectively, said apparatus comprising:

first and second signal processing channels comprising means for generating first and second analog video signals in response to said first and second output signals respectively, said first and second video signals being representative of said first and second images on a frame by frame basis;

analog-to digital converter means for converting said first and second analog video signals into first and second digital data representative of said first and second images respectively on a frame by frame basis;

image-correcting means for correcting for misalignment of said first and second images relative to one another so that when said first and second images are reproduced by electronic display means they will appear coincident with one another for viewing as a 3-D image, said image-correcting means comprising first and second memory units for receiving and storing said first and second digital data, write control means for controlling the writing of said first and second digital data into said first and second memory units, readout control means for controlling the reading out of said first and second digital data from said memory units, image analysis means coupled to said first and second memory units for comparing said first and second digital data as read out from said memory units to measure any relative differences in at least the (a) sizes or (b) horizontal and vertical positions or (c) rotational positions of said first and second images and for producing first and second correctional signals according to said relative differences, and means for applying said first and second correctional signals to said readout control means as readout commands so as to adjust the readout of digital data from at least one of said first and second memory units in a manner that eliminates any of said differences in images represented by subsequent first and second digital data fed to said first and second memory units respectively;

digital-to-analog converter means;

means for applying said first and second digital data as read out of said first and second memories to said digital-to-analog converter means for producing first and second video output signals;

an electronic display means; and

display operating means responsive to said first and second video output signals for causing said display means to reproduce said first and second images on a frame by frame basis and display said reproduced first and second images either alternately or concurrently for viewing as a 3-D image.

18. Apparatus according to claim 17 wherein each of said first and second memory units is a dual port memory unit having first and second ports, with data being read into said memory units via said first ports and data being read out of said memory units via said second ports.

19. A method of correcting for misalignment of the separate left and right images of a stereo electronic camera having a first and second optical means for viewing an object and transmitting first and second images of said object with a parallax between said first and second images, and first and second imaging means for picking up said first and second images as transmitted by said optical systems and producing first and second output signals representative of said first and second images respectively, said method comprising the steps set forth in FIGS. 9A and 9B of the drawings.