Image acquisition with depth enhancement

• US 6,563,105 B2
• Filed: 08/23/2001
• Issued: 05/13/2003
• Est. Priority Date: 06/08/1999
• Status: Expired due to Term

• Current Assignee: University of Washington
• Original Assignee: University of Washington
• Inventors: Seibel, Eric J., Furness, Thomas A. III, Smithwick, Quinn Y. J.
• Primary Examiner(s): LE, QUE TAN
• Application Number:
  US09/938,467
• Publication Number:
  US 20020139920A1
• Time in Patent Office: 628 Days
• Field of Search: 250/208.1, 250/234, 250/235, 250/306, 250/307, 250/227.11, 250/214.VT, 359/205, 359/207, 359/213, 359/199, 359/202, 378/4, 378/12, 378/21, 378/36, 382/203, 382/204, 382/260
• US Class Current: 250/208.1
• CPC Class Codes: A61B 1/0008 characterised by distal tip features
• CPC Class Codes: A61B 1/00165 with light-conductive means, e.g. fibre optics A61B1/07 takes precedence
• CPC Class Codes: A61B 1/00172 with means for scanning
• CPC Class Codes: A61B 1/00193 adapted for stereoscopic vision
• CPC Class Codes: A61B 1/0051 with controlled bending of insertion part
• CPC Class Codes: A61B 1/043 for fluorescence imaging
• CPC Class Codes: A61B 1/05 characterised by the image sensor, e.g. camera, being in the distal end portion
• CPC Class Codes: A61B 1/0638 providing two or more wavelengths
• CPC Class Codes: A61B 1/07 using light-conductive means, e.g. optical fibres
• CPC Class Codes: A61B 2562/028 Microscale sensors, e.g. electromechanical sensors [MEMS]
• CPC Class Codes: A61B 3/12 for looking at the eye fundus, e.g. ophthalmoscopes A61B3/13 takes precedence
• CPC Class Codes: A61B 5/0062 Arrangements for scanning
• CPC Class Codes: A61B 5/0064 Body surface scanning
• CPC Class Codes: A61B 5/0068 Confocal scanning
• CPC Class Codes: A61B 5/0071 by measuring fluorescence emission
• CPC Class Codes: A61B 5/0086 using infrared radiation
• CPC Class Codes: A61B 5/7239 using differentiation including higher order derivatives
• CPC Class Codes: G01S 17/88 Lidar systems specially adapted for specific applications
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 7/481 Constructional features, e.g. arrangements of optical elements
• CPC Class Codes: G01S 7/4814 : of transmitters alone
• CPC Class Codes: G01S 7/4817 : relating to scanning
• CPC Class Codes: G01S 7/4818 : using optical fibres
• CPC Class Codes: G02B 26/10 : Scanning systems
• CPC Class Codes: G02B 26/103 : having movable or deformable optical fibres, light guides or waveguides as scanning elements light guides per se G02B6/00
• CPC Class Codes: G06T 2207/10068 : Endoscopic image
• CPC Class Codes: G06T 2207/30004 : Biomedical image processing
• CPC Class Codes: G06T 7/586 : from multiple light sources, e.g. photometric stereo
• CPC Class Codes: H04N 13/239 : using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/254 : in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 23/55 : Optical parts specially adapted for electronic image sensors; Mounting thereof
• CPC Class Codes: H04N 23/555 : for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes
• CPC Class Codes: H04N 23/56 : provided with illuminating means
• CPC Class Codes: H04N 25/48 : Increasing resolution by shifting the sensor relative to the scene

Abstract

A minimally invasive, medical, image acquisition system outputs a light beam or pulse which illuminates a precise spot size. A plurality of photon detector detect returning photons from the object, including the spot. Pixel resolution is determined by the area of the illumination spot (and thus the lens configuration), rather than an area sensed by the detector. Depth enhancement is determined by correlating images detected by the respective detectors, or alternatively by a range finding method based on phase difference, time of flight, frequency or interferometry.

Claims

1. A method for generating an image of a target surface, comprising the steps of:

outputting a beam of light along a scan path relative to an optical axis;

focusing the beam onto a target surface being scanned, wherein at a given time the beam is focused to impinge on a spot of the target, the spot being an illuminated spot, the illuminated spot varying with time as the beam is scanned onto the target surface;

detecting light returning from the target surface responsive to the output beam of light with a plurality of detectors, each one of the plurality of detectors having a respective location relative to the optical axis, wherein the returning light is detected concurrently at the plurality of detectors;

acquiring a plurality of concurrent images with the plurality of detectors;

wherein the plurality of concurrent images have a common apparent viewpoint based upon a common light beam being used to generate the returning light, the common light beam being said output beam of light;

wherein each one image of the plurality of concurrent images has a differing apparent lighting direction, the apparent lighting direction for a given one image of the plurality of images being based upon the respective location of a corresponding one or more detectors of the plurality of detectors used to acquire said given one image; and

processing the plurality of concurrent images to derive an output image of the target surface, including relative depth information based upon the differing apparent lighting direction from each one image of the plurality of concurrent images.

2. The method of claim 1, wherein the step of detecting comprises detecting light returning from an area of the target surface which is greater than an area defined the illuminated spot.

3. The method of claim 1, in which the step of processing comprises generating an orientation map of target surface normals based upon the first image and second image.

4. The method of claim 3, in which the step of processing further comprises the step of creating an orientation histogram from the orientation map and comparing the orientation histogram to one or more prototype orientation histograms to classify the target surface.

5. The method of claim 3, in which the step of processing further comprises the step of creating a depth map from the orientation map.

6. The method of claim 3, in which the step of processing further comprises the step of creating a 3-dimensional mesh from the depth map.

7. The method of claim 6, in which the step of processing comprises rendering the 3-dimensional mesh with virtual lighting and viewpoint to enhance either one or both of depth perception and surface topography.

8. The method of claim 6, in which the step of processing comprises computing two stereographic images from the 3-dimensional mesh in which a viewer'"'"'s binocular disparity is calculated.

9. The method of claim 1, in which the step of processing comprises rendering the output image to exhibit enhanced surface topography in relation to each one of the plurality of concurrent images.

10. The method of claim 1, in which the step of processing comprises rendering the output image to exhibit enhanced lighting perspective relative to each one of the plurality of concurrent images.

11. The method of claim 1, in which the step of processing comprises rendering two stereographic images in which a viewer'"'"'s binocular disparity is calculated, the two stereographic images being output images to a viewer'"'"'s eyes.

12. The method of claim 1, wherein the output beam of light comprises visible light and ultraviolet light, wherein the first detector detects returning visible light and the second detector detects returning ultraviolet light.

13. The method of claim 1, wherein the output beam of light comprises visible light and infrared light, wherein a first detector of the plurality of detectors detects returning visible light and a second detector of the plurality of detectors detects returning infrared light.

14. The method of claim 1, in which the beam of light is an output beam of light emitting from a resonant waveguide, and further comprising, prior to the step of outputting the steps of:

generating a first beam of light of a first color, a second beam of light of a second color and a third beam of light of a third color; and

combining the first beam, the second beam and the third beam before entering the resonant waveguide, the combined first beam, second beam and third beam forming the output beam.

15. The method of claim 1, in which the output beam of light is a sequence of light pulses, and wherein the step of detecting is synchronized with the sequence of light pulses, the detected returning light at said given time corresponding to a given light pulse and said acquired pixel.

16. The method of claim 1 wherein the step of detecting returning light comprises detecting reflected light.

17. The method of claim 1 wherein the step of detecting returning light comprises detecting fluorescent light emitted from the target surface responsive to the output beam.

18. The method of claim 1 wherein the step of detecting returning light comprises detecting phosphorescent light emitted from the target surface responsive to the output beam.

19. A system for acquiring an image of a target surface, comprising:

a light source which emits light;

an actuator which deflects the emitted light along a scan path relative to an optical axis;

a plurality of photon detectors for detecting light returning from the target surface responsive to the emmitted light, each one of the plurality of photon detectors having a distinct location relative to the optical axis, wherein a plurality of concurrent images are acquired with the plurality of detectors, the plurality of concurrent images have a common apparent viewpoint based upon a common light beam being used to generate the returning light, the common light beam being said output beam of light, each one image of the plurality of concurrent images having a differing apparent lighting direction, the apparent lighting direction for a given one image of the plurality of images being based upon the respective location of a corresponding one or more detectors of the plurality of detectors used to acquire said given one image; and

a processor which processing the plurality of concurrent images to derive an output image of the target surface, including depth information based upon the differing apparent lighting direction from each one image of the plurality of concurrent images.

20. The system of claim 19, further comprising:

a flexible, optical waveguide which receives the emitted light and directs the light toward a target surface, wherein at a given time the light impinges on a spot of the target surface, the spot being an illuminated spot; and

wherein the actuator deflects the waveguide into a resonant motion, the directed light tracing a scan path along the target surface.

21. The system of claim 19, further comprising:

a mirror which receives the emitted light and directs the light toward a target surface, wherein at a given time the light impinges on a spot of the target surface, the spot being an illuminated spot; and

wherein the actuator deflects the mirror, the directed light tracing a scan path along the target surface.

22. The system of claim 19, further comprising:

a cantilever assembly having a light source at a distal cantilever tip that emits light out along an axis of the cantilever toward a target surface, wherein at a given time the light impinges on a spot of the target surface, the spot being an illuminated spot; and

wherein the actuator deflects the cantilever assembly, the directed light tracing a scan path along the target surface.

23. The system of claim 19, in which each one of the plurality of detectors has an active viewing area of the target surface which exceeds size of the illuminated spot;

and further comprising a correlator which correlates sampling time of the detector with the light as the light traces a scan path, wherein pixels are acquired of an image of a portion of the target surface, wherein resolution of each one pixel of the acquired pixels corresponds to the size of the illuminated spot; and

wherein a first one of the concurrent images is acquired from the first set of detectors.

24. The system of claim 19, wherein the output beam of light is a combination of a first color light beam, a second color light beam and a third color light beam, and wherein the output image is a color image;

wherein the plurality of detectors comprises a first set of detectors, the first set of detectors including a first color first detector for detecting returning light of the first color light beam, a second color first detector for detecting returning light of the second color light beam, and a third color first detector for detecting returning light of the third color light beam, and wherein a first one of the plurality of concurrent images is acquired from the first set of detectors.

25. The system of claim 24, wherein the plurality of detectors comprises a second set of detectors, the second set of detectors including a first color second detector for detecting returning light of the first color light beam, a second color second detector for detecting returning light of the second color light beam, and a third color second detector for detecting returning light of the third color light beam, and wherein a second one of the plurality of concurrent images is acquired from the second set of detectors.

26. The system of claim 24, wherein the plurality of detectors further comprises a second detector, the second detector being a multiwavelength detector for detecting brightness of the returning light from the first color light beam, second color light beam and third color light beam, wherein a second one of the plurality of concurrent images is acquired from the second detector.

27. The system of claim 26, wherein the plurality of detectors further comprises a third detector, and wherein a third one of the plurality of concurrent images is acquired from the third detector.

28. The system of claim 19, wherein the plurality of detectors comprises a first detector and a second detector, the first detector used for acquiring a first one of the concurrent images, the second detector used for acquiring a second one of the plurality of concurrent images.

29. The system of claim 28, wherein the plurality of detectors further comprises a third detector for acquiring a third one of the plurality of concurrent images.

30. The system of claim 19, wherein the processor generates an orientation map of target surface normals based upon the plurality of concurrent images.

31. The system of claim 30, wherein the processor creates an orientation histogram from the orientation map and compares the orientation histogram to one or more prototype orientation histograms to classify the target surface.

32. The system of claim 30, wherein the processor creates a depth map from the orientation map.

33. The system of claim 32, wherein the processor creates a 3-dimensional mesh from the depth map.

34. The system of claim 33, wherein the processor renders the 3-dimensional mesh with virtual lighting and viewpoint to enhance either one or both of depth perception and surface topography.

35. The system of claim 34, wherein the processor computes two stereographic images from the 3-dimensional mesh in which a viewer'"'"'s binocular disparity is calculated.