Methods and systems for laser based real-time structured light depth extraction

• US 7,385,708 B2
• Filed: 06/09/2003
• Issued: 06/10/2008
• Est. Priority Date: 06/07/2002
• Status: Expired due to Fees

• Current Assignee: University of North Carolina At Chapel Hill (University of North Carolina System)
• Original Assignee: University of North Carolina At Chapel Hill (University of North Carolina System)
• Inventors: Keller, Kurtis P., Ackerman, Jeremy D.
• Primary Examiner(s): Toatley, Jr.; Gregory J.
• Assistant Examiner(s): Valentin; Juan D
• Application Number:
  US10/515,305
• Publication Number:
  US 20050219552A1
• Time in Patent Office: 1,828 Days
• Field of Search: 356/603, 356/604, 356/610
• US Class Current: 356/603
• CPC Class Codes: A61B 1/042 characterised by a proximal camera, e.g. a CCD camera
• CPC Class Codes: A61B 1/0605 for spatially modulated illumination
• CPC Class Codes: A61B 5/1076 for measuring dimensions inside body cavities, e.g. using catheters A61B3/1005 takes precedence
• CPC Class Codes: G01B 11/2536 using several gratings with variable grating pitch, projected on the object with the same angle of incidence
• CPC Class Codes: G02B 23/2484 Arrangements in relation to a camera or imaging device processing or control of video signals generated by an endoscope H04N23/60, H04N23/70
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects

Abstract

Laser-based methods and systems for real-time structured light depth extraction are disclosed. A laser light source (100) produces a collimated beam of laser light. A pattern generator (102) generates structured light patterns including a plurality of pixels. The beam of laser light emanating from the laser light source (100) interacts with the patterns to project the patterns onto the object of interest (118). The patterns are reflected from the object of interest (118) and detected using a high-speed, low-resolution detector (106). A broadband light source (111) illuminates the object with broadband/light, and a separate high-resolution, low-speed detector (108) detects broadband light reflected from the object (118). A real-time structured light depth extraction engine/controller (110) based on the transmitted and reflected patterns and the reflected broadband light.

Claims

1. A laser-based real-time structured light depth extraction system comprising:

(a) a laser light source for producing a collimated beam of laser light;

(b) a pattern generator being optically coupled to the laser light source for generating a plurality of different structured light patterns, each structured light pattern including a plurality of pixels, wherein each pattern interacts with the collimated beam of laser light to simultaneously project the plurality of pixels onto an object of interest;

(c) a detector being optically coupled to the object and synchronized with the pattern generator for receiving patterns reflected from the object and for generating signals based on the reflected patterns; and

(d) a real-time structured light depth extraction engine/controller coupled to the detector and the pattern generator for controlling the projection and detection of the patterns, for calculating, in real-time, depth values for regions of the object based on the signals received from the detector, and for generating and displaying the image of the object based on the calculated depth values, wherein the projection, the calculating, and the generation are repeated continually to update display of the image in real time,

wherein the three dimensional image of the object comprises a synthetic image and wherein the real time structured light depth extraction engine/controller combines the synthetic image with a real image of a scene and displays the combined image.

2. The system of claim 1 wherein the laser light source comprises a narrowband laser light source.

3. The system of claim 2 wherein the narrowband laser light source comprises a green laser light source.

4. The system of claim 1 wherein the laser light source comprises a laser diode.

5. The system of claim 1 wherein the laser light source is optically coupled to the pattern generator without using a collimator.

6. The system of claim 1 wherein the pattern generator comprises a transmissive pattern generator for selectively passing portions of the collimated beam from the laser to generate the patterns.

7. The system of claim 1 wherein the pattern generator comprises a reflective display for selectively reflecting portions of the collimated beam from the laser to generate the patterns.

8. The system of claim 1 wherein the pattern generator comprises a micro-electromechanical system for interacting with the collimated beam of light to generate the patterns.

9. The system of claim 1 comprising a broadband light source for illuminating the object with broadband light.

10. The system of claim 9 wherein the detector comprises a single camera for detecting the reflected patterns and for detecting broadband light reflected from the object.

11. The system of claim 9 wherein the detector comprises a high-speed, low-resolution camera for detecting the reflected patterns and wherein the system further comprises a low-speed, high-resolution camera for detecting broadband light reflected from the object.

12. The system of claim 11 wherein the cameras and the light sources are operated synchronously with each other such that the laser light source is off when the broadband light source is on for separating depth extraction and broadband illumination functions.

13. The system of claim 11 wherein the cameras and the light sources are operated asynchronously with each other such that the laser light source and the broadband light source are simultaneously on for simultaneous depth extraction and broadband illumination of the object.

14. The system of claim 13 comprising a bandpass filter optically coupled to the broadband light source for reducing power in the frequency band of the laser light source.

15. The system of claim 13 comprising a bandpass filter optically coupled to the high-resolution, low-speed camera for reducing power in the frequency band of the laser light source.

16. The system of claim 11 wherein the high-speed, low-resolution camera is tuned to the bandwidth of the laser light source.

17. The system of claim 1 wherein the real-time structured light depth extraction engine/controller is adapted to control the pattern generator and the detector to project and detect the patterns at a rate of at least about 30 frames per second.

18. The system of claim 1 wherein the real-time structured light depth extraction engine/controller is adapted to control the level of depth detail obtained from the object by changing the patterns generated by the pattern generator.

19. The system of claim 1 comprising an endoscope optically coupled to the pattern generator and the detector for projecting the patterns into the body of a patient and for communicating the reflected patterns from the body of the patient to the detector.

20. The system of claim 19 comprising a beam expander optically coupled to the laser light source and to the pattern generator for expanding light generated from the laser to cover a pattern generated by the pattern generator and a beam compressor optically coupled to the pattern generator and the endoscope for compressing the patterns projected from the pattern generator to fit within an optical path of the endoscope.

21. The system of claim 19 wherein the endoscope comprises a laparoscope.

22. The system of claim 1 wherein the display of the image is updated at a rate of at least about 30 frames per second.

23. A real-time structured light depth extraction system comprising:

(a) a self-illuminating display for generating a plurality of different structured light patterns, each structured light pattern including a plurality of pixels, and for simultaneously projecting the plurality of pixels onto an object of interest;

(b) a detector being optically coupled to the self-illuminating display for receiving patterns reflected from the object and for generating signals based on the reflected patterns; and

(c) a real-time structured light depth extraction engine/controller coupled to the detector and the self-illuminating display for controlling the projection and detection of the patterns and for calculating, in real-time, depth values for regions of the object based on the signals received from the detector.

24. The system of claim 23 wherein the self-illuminating display comprises an organic light emitting (OLE) display.

25. The system of claim 23 comprising a broadband light source for broadband illumination of the object.

26. The system of claim 25 wherein the detector comprises a single camera for detecting the reflected patterns and for detecting broadband light reflected from the object.

27. The system of claim 25 wherein the detector comprises a high-speed, low-resolution camera for detecting the reflected patterns and wherein the system further comprises a low-speed, high-resolution camera for detecting broadband light reflected from the object.

28. The system of claim 27 wherein the cameras, the self-illuminating display, and the broadband light source are operated synchronously with each other such that the self-illuminating display is off when the broadband light source is on for separating depth extraction and broadband illumination functions.

29. The system of claim 27 wherein the cameras, the self-illuminating display, and the broadband light source are operated asynchronously with each other such that the self-illuminating display and the broadband light source are simultaneously on for simultaneous depth extraction and broadband illumination of the object.

30. The system of claim 29 comprising a bandpass filter optically coupled to the broadband light source for reducing power in the frequency band of the self-illuminating display.

31. The system of claim 29 comprising a bandpass filter optically coupled to the high-resolution, low-speed camera for reducing power in the frequency band of the patterns projected by the self-illuminating display.

32. The system of claim 23 wherein the high-speed, low-resolution camera is tuned to the frequency band of the patterns projected by the self-illuminating display.

33. The system of claim 23 wherein the real-time structured light depth extraction engine/controller is adapted to control the pattern generator and the detector to project and detect the patterns at a rate of at least about 30 frames per second.

34. The system of claim 23 wherein the real-time structured light depth extraction engine/controller is adapted to control the level of depth detail obtained from the object by changing the patterns generated by the pattern generator.

35. The system of claim 23 comprising an endoscope optically coupled to the pattern generator and the detector for projecting the patterns into the body of a patient and for communicating the reflected patterns from the body of the patient to the detector.

36. The system of claim 35 comprising a beam compressor optically coupled to the self-illuminating display and the endoscope for compressing the patterns projected by the display to fit within an optical path of the endoscope.

37. The system of claim 35 wherein the endoscope comprises a laparoscope.

38. A system for real-time structured light depth extraction in an endoscopic surgical environment:

(a) a laser light source for producing a collimated beam of laser light;

(b) a pattern generator optically coupled to the laser light source for presenting a plurality of different structured light patterns, each structured light pattern including a plurality of pixels, when each pattern interacts with the collimated beam of laser light to simultaneously project the plurality of pixels onto an object of interest;

(c) beam expansion optics being optically coupled to the laser light source and the pattern generator for expanding the collimated beam of laser light to illuminate an area of the pattern generator corresponding to the patterns;

(d) an endoscope optically coupled to the pattern generator for communicating the patterns into a patient'"'"'s body to illuminate an object and for receiving patterns reflected from the object;

(e) beam compression optics being optically coupled to the pattern generator and the detector for reducing a beam width of the projected patterns to a size that fits within an aperture of the endoscope;

(f) a structured light depth extraction camera optically coupled to the object via the endoscope and synchronized with the pattern generator for detecting the reflected patterns and generating signals based on the reflected patterns;

(g) a broadband light source optically coupled to the object via the endoscope for broadband illumination of the object;

(h) a color camera optically coupled to the object via the endoscope for receiving broadband light reflected from the object and for generating signals based on the reflected broadband light; and

(i) a real-time structured light depth extraction engine/controller coupled to the cameras and the pattern generator for controlling the projection and detection of patterns for calculating, in real-time, depth and color values for regions of the object based on the signals received from the cameras, and for generating and displaying the image of the object based on the calculated depth values, wherein the projecting of the plurality of pixels onto the object, the calculating of the depth and color values, and the generation of the image of the object are repeated continually to update display of the image in real time,

wherein the three dimensional image of the object comprises a synthetic image and wherein the real time structured light depth extraction engine/controller combines the synthetic image with a real image of a scene and displays the combined image.

39. The system of claim 38 wherein the display of the image is updated at a rate of at least about 30 frames per second.

40. A method for laser-based real-time structured light depth extraction comprising:

(a) projecting a beam of laser light from a laser onto a pattern generator;

(b) generating patterns on the pattern generator, each pattern including a plurality of pixels;

(c) altering the beam of laser light using the patterns such that the patterns are projected onto object of interest;

(d) detecting patterns reflected from the object of interest;

(e) calculating depth information relating to the object of interest in real-time based on the reflected patterns;

(f) generating and displaying three-dimensional image of the object based on the calculated depth information; and

(g) wherein steps (a)-(f) are continually repeated to update display of the three-dimensional image of the object in real time,

wherein the three dimensional image of the object comprises a synthetic image and wherein the method further comprises combining the synthetic image with a real image of a scene and displaying the combined image.

41. The method of claim 40 wherein projecting a beam of laser light onto a pattern generator includes projecting a beam of narrowband laser light onto the pattern generator.

42. The method of claim 41 wherein projecting a beam of narrowband laser light includes projecting a beam of green laser light onto the pattern generator.

43. The method of claim 40 wherein generating patterns includes generating patterns on a transmissive display.

44. The method of claim 40 wherein generating patterns includes generating reflective patterns using a reflective display.

45. The method of claim 40 wherein generating patterns includes generating patterns using a micro-electromechanical system.

46. The method of claim 40 comprising illuminating the object with a broadband light source.

47. The method of claim 46 wherein detecting patterns reflected from the object includes detecting patterns reflected from the object using a high-speed, high-resolution camera and wherein the method further comprises detecting broadband light reflected from the object using the high-speed, high-resolution camera.

48. The method of claim 46 wherein detecting patterns reflected from the object includes detecting the patterns using a high-speed, low-resolution camera and wherein the method further comprises detecting broadband light reflected from the object using a high-resolution, low-speed camera.

49. The method of claim 48 wherein the laser light projecting, the broadband illuminating, the reflected pattern detecting, and the broadband light detecting steps occur asynchronously with respect to each other.

50. The method of claim 48 wherein the laser light projecting, the broadband illuminating, the reflected pattern detecting, and the broadband light detecting steps occur synchronously with respect to each other.

51. The method of claim 40 wherein projecting the patterns onto an object of interest includes projecting the patterns through an endoscope onto an object within a patient'"'"'s body.

52. The method of claim 51 wherein projecting the patterns through an endoscope includes the projecting the patterns through a laparoscope.

53. The method of claim 51 comprising expanding the beam of laser light to cover a predetermined area of the pattern generator and compressing the patterns reflected from the pattern generator to fit within a diameter of the endoscope.

54. The method of claim 40 wherein calculating depth information in real-time includes rendering depth values at a rate of at least about 30 frames per second.

55. The method of claim 40 wherein the projecting and detecting steps are performed with a photonic efficiency of at least about thirty percent.

56. The method of claim 40 comprising dynamically changing the patterns to increase or decrease the resolution of the depth information.

57. The method of claim 40 wherein the display of the three dimensional image of the object is updated at a rate of at least about 30 frames per second.

58. A method for real-time structured light depth extraction comprising:

(a) projecting patterns from a self-illuminating display onto an object of interest;

(b) detecting patterns reflected from the object of interest;

(c) calculating depth information relating to the object of interest in real-time based on the reflected patterns; and

(d) generating a three-dimensional image of the object based on the calculated depth information.

59. The method of claim 58 wherein projecting patterns onto an object of interest using a self-illuminating display includes projecting patterns from an organic light emitting (OLE) display.

60. The method of claim 58 wherein projecting patterns onto an object of interest using a self-illuminating display includes projecting patterns from the display through an endoscope onto an object inside of a patient'"'"'s body.

61. The method of claim 58 wherein detecting reflected patterns includes detecting the reflected patterns using a high-speed, low-resolution camera.

62. The method of claim 58 comprising illuminating the object using a broadband light source and detecting broadband light reflected from the object using a high-resolution, low-speed camera.

63. The method of claim 62 comprising combining the depth information with the reflected broadband light to obtain color depth images of the object of interest.

64. The method of claim 58 comprising dynamically changing the patterns to increase or decrease the resolution of the depth information.