Stereoscopic visualization camera and platform

• US 10,299,880 B2
• Filed: 11/15/2017
• Issued: 05/28/2019
• Est. Priority Date: 04/24/2017
• Status: Active Grant

• Current Assignee: Alcon Incorporated
• Original Assignee: TrueVision Systems, Inc.
• Inventors: Ramirez Luna, Maximiliano, Weissman, Michael, Riederer, Thomas Paul, Polchin, George Charles, Tripathi, Ashok Burton
• Primary Examiner(s): Chio, Tat C
• Application Number:
  US15/814,127
• Publication Number:
  US 20180303574A1
• Time in Patent Office: 559 Days
• CPC Class Codes: A61B 90/20 Surgical microscopes characterised by non-optical aspects
• CPC Class Codes: A61B 90/37 Surgical systems with images on a monitor during operation
• CPC Class Codes: G03B 15/14 for taking photographs during medical operations
• CPC Class Codes: G03B 2205/0046 Movement of one or more optical elements for zooming for optical zoom lenses in general G02B7/10
• CPC Class Codes: G03B 35/08 by simultaneous recording
• CPC Class Codes: G03B 35/10 having single camera with stereoscopic-base-defining system
• CPC Class Codes: G03B 5/00 Adjustment of optical system relative to image or object surface other than for focusing
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 13/15 for colour aspects of image signals
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/183 On-screen display [OSD] information, e.g. subtitles or menus
• CPC Class Codes: H04N 13/225 using parallax barriers
• CPC Class Codes: H04N 13/236 using varifocal lenses or mirrors
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 13/337 using polarisation multiplexing
• CPC Class Codes: H04N 23/10 for generating image signals from different wavelengths
• CPC Class Codes: H04N 23/50 : Constructional details
• CPC Class Codes: H04N 23/67 : Focus control based on electronic image sensor signals
• CPC Class Codes: H04N 23/69 : Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 23/843 : Demosaicing, e.g. interpolating colour pixel values

Abstract

A stereoscopic visualization camera and platform are disclosed. An example stereoscopic visualization camera includes a first plurality of lenses positioned along a first optical path and a first image sensor to record a first image stream of a target site from light in the first optical path. The stereoscopic visualization camera also includes a second plurality of lenses positioned along a second optical path, parallel to the first optical path, and a second image sensor to record a second image stream of the target site from light in the second optical path. The stereoscopic visualization camera also includes a processor configured to reduce spurious parallax between the first and second images streams by selecting pixel sets of pixel grids of the first and second image sensors such that zoom repeat points (“ZRP”) are located at a center of the respective pixel grids.

Claims

1. A stereoscopic imaging apparatus configured to reduce spurious parallax between first and second images streams recorded in parallel of a target site, the apparatus comprising:

first optical elements positioned along a first optical path and including (i) a first plurality of lenses including a first zoom lens configured to be moveable along the first optical path in a z-direction, and (ii) a first image sensor to record the first image stream of the target site from light in the first optical path;

second optical elements positioned along a second optical path parallel to the first optical path, and including (i) a second plurality of lenses including a second zoom lens configured to be moveable along the second optical path in a z-direction, and (ii) a second image sensor to record the second image stream of the target site from light in the second optical path; and

a processor configured to;

locate a position of a first zoom repeat point (“

ZRP”

) by;

causing the first zoom lens to move along the z-direction during a recording of the first image stream,

locating a first portion of area that does not move in an x-direction or a y-direction within the images of the first image stream, and

determining a first distance between an origin point within at least one of the images of the first image stream and the first portion of the area as the position of the first ZRP,

determine a first pixel set of a first pixel grid of the first image sensor using the first distance such that the first ZRP is located at a center of the first pixel set,

determine a second pixel set of a second pixel grid of the second image sensor that includes an image that is aligned with an image from the first pixel set of the first image sensor,

locate a position of a second ZRP by;

causing the second lens to move along the z-direction during a recording of the second image stream,

locating a second portion of area that does not move in the x-direction or the y-direction within the images of the second image stream, and

determining a second distance between a center of the second pixel set and the second portion of the area as the position of the second ZRP, and

adjust one of the second plurality of lenses or the second image sensor in at least one of the x-direction, the y-direction, and a tilt-direction to cause the second ZRP to be aligned with the center of the second pixel set based on the determined second distance.

2. The stereoscopic imaging apparatus of claim 1, wherein the one of the second plurality of lenses is the second zoom lens.

3. The stereoscopic imaging apparatus of claim 1, wherein the processor is configured to store, to a look-up-table as a calibration point, first coordinates of the first pixel set with respect to the first pixel grid, second coordinates of the second pixel set with respect to the second pixel grid, a location of the one of the second plurality of lenses adjusted in the at least one of the x-direction, the y-direction, and the tilt direction, and a magnification level of the first and second zoom lenses.

4. The stereoscopic imaging apparatus of claim 3, wherein the processor is configured to use the look-up-table during use of the stereoscopic imaging apparatus to select the first pixel set for the first image sensor and the second pixel set for the second image sensor when the first and second zoom lenses are moved to the magnification level.

5. The stereoscopic imaging apparatus of claim 3, wherein the processor is configured to:

locate the first ZRP and the second ZRP at different magnification levels; and

store to the look-up-table as calibration points for the different magnification levels, first coordinates of the first pixel set with respect to the first pixel grid for each magnification level, second coordinates of the second pixel set with respect to the second pixel grid for each magnification level, a location of the one of the second plurality of lenses adjusted in the at least one of the x-direction, the y-direction, and the tilt direction for each magnification level, and the magnification level of the first and second zoom lenses.

6. The stereoscopic imaging apparatus of claim 1, wherein the processor is configured to record, as a movement amount, an amount and a direction the one of the second plurality of lenses was adjusted in the x-direction, the y-direction, and the tilt direction to cause the second ZRP to be aligned with the second origin point.

7. The stereoscopic imaging apparatus of claim 1, wherein the processor is configured to cause the first zoom lens to move, during the recording of the first image stream, from a first magnification level to a second magnification level that is twice the first magnification, and

wherein the processor is configured to cause the second zoom lens to move, during the recording of the second image stream, from the first magnification level to the second magnification level.

8. The stereoscopic imaging apparatus of claim 1, wherein the first distance includes coordinates of the first ZRP with respect to the origin point.

9. The stereoscopic imaging apparatus of claim 1, wherein the target site includes at least one of a template with graphical features, a surgical site of a patient, and a grid having visually distinguishable features.

10. The stereoscopic imaging apparatus of claim 9, wherein the template includes at least one of a bullseye, a wave pattern, and a spirograph pattern.

11. The stereoscopic imaging apparatus of claim 1, wherein the processor is configured to:

create a synthesized target site image from an image recorded at the end of movement of the first lens at a second magnification level that is scaled based on an amount of the movement of the first lens along the z-direction; and

determine the position of the first ZRP based on an amount of movement in the x-direction and the y-direction to align the synthesized target site image to a target site image recorded at a first magnification level before the movement of the first lens along the z-direction.

12. A stereoscopic imaging apparatus configured to reduce spurious parallax between first and second image streams recorded in parallel of a target site, the apparatus comprising:

first optical elements positioned along a first optical path and including (i) a first plurality of lenses including a first zoom lens configured to be moveable along the first optical path in a z-direction, and (ii) a first image sensor to record the first image stream of the target site from light in the first optical path;

second optical elements positioned along a second optical path that is parallel to the first optical path, the second optical elements including (i) a second plurality of lenses including a second zoom lens configured to be moveable along the second optical path in the z-direction, and (ii) a second image sensor to record the second image stream of the target site from light in the second optical path; and

a processor configured to;

locate a position of a first zoom repeat point (“

ZRP”

) in the first image stream,

determine a first pixel set of a first pixel grid of the first image sensor such that the first ZRP is located at a center of the first pixel set, and

determine a second pixel set of a second pixel grid of the second image sensor such that an image from the second pixel set is visually aligned with an image from the first pixel set.

13. The stereoscopic imaging apparatus of claim 12, further comprising:

a main objective assembly configured to set a working distance to a focal plane on the target site and receive light from the target site,

wherein the first plurality of lenses includes a first front lens configured to define the first optical path from the light received by the main objective assembly, and a first final optical element configured to focus the first optical path on the first image sensor, and

wherein the second plurality of lenses includes a second front lens configured to define the second optical path from the light received by the main objective assembly, and a second final optical element configured to focus the second optical path on the first image sensor.

14. The stereoscopic imaging apparatus of claim 13, wherein the processor is further configured to, before determining the second pixel set, adjust one of the second plurality of lenses or the second image sensor in at least one of an x-direction, a y-direction, and a tilt-direction to cause the second ZRP to be aligned with an origin point within the second image stream.

15. The stereoscopic imaging apparatus of claim 14, wherein at least one of the second front lens and the second final optical element is the one of the second plurality of lenses that is adjusted in at least one of the x-direction, the y-direction, and the tilt-direction.

16. The stereoscopic imaging apparatus of claim 14, further comprising a flexure connected to the one of the second plurality of lenses or the second image sensor that is adjusted in at least one of the x-direction, the y-direction, and the tilt-direction, the flexure mechanically connected to an actuator configured to cause a beam of the flexure to move for adjusting the one of the second plurality of lenses or the second image sensor.

21. The stereoscopic imaging apparatus of claim 12, wherein the processor is configured to:

synchronize first images recorded from the first pixel set and second images recorded from the second pixel set;

render the first images and the second images for stereoscopic display; and

transmit the rendered first and second images to at least one display monitor.

22. The stereoscopic imaging apparatus of claim 12, wherein the rendered first and second images provide at least one of:

a three-dimensional stereoscopic view of the target site; and

a view where the first image is aligned and overlaid the second image.

17. The stereoscopic imaging apparatus of 13, further comprising a visible light source, a near-infrared (NIR) light source, and a near-ultraviolet (NUV) light source.

18. The stereoscopic imaging apparatus of 17, wherein the visible light source and the NIR light source are placed between the main objective assembly and a deflecting element, and

wherein the NUV light source is behind the deflecting element such that NUV light transmitted from the NUV light source contacts the deflecting element before a portion of the NUV light reaches the main objective assembly for propagation to the target site.

19. The stereoscopic imaging apparatus of claim 18, further comprising two low-pass excitation filters configured to respectively block a portion of light wavelengths from the NIR and NUV light sources that interfere with an emission wavelengths from fluorescence.

20. The stereoscopic imaging apparatus of claim 18, further comprising:

an infrared-cut filter configured to be used in conjunction with the visible light source to enable visible light to pass through to the first and second image sensors;

a near-ultraviolet-cut filter configured to be used in conjunction with the NUV light source to block light of a certain near-ultraviolet wavelength from passing through to the first and second image sensors; and

a near-infrared bandpass filter configured to be used in conjunction with the NIR light source and the visible light source to enable light of a certain near-infrared wavelength to pass through to the first and second image sensors.

23. The stereoscopic imaging apparatus of 12, further comprising a housing configured to enclose the first optical elements, the second optical elements, and the processor,

wherein the housing has a width between 20 and 22 cm, a length between 23 and 27 cm, and a height between 13 and 17 cm.

24. The stereoscopic imaging apparatus of 12, wherein the first optical path is separated from the second optical path by an interpupillary distance.

25. The stereoscopic imaging apparatus of 12, wherein the first zoom lens is configured to change a size of the first image stream while being fixed in position along the z-direction and the second zoom lens is configured to change a size of the second image stream while being fixed in position along the z-direction.