Methods and systems for real-time structured light depth extraction and endoscope using real-time structured light depth extraction

• US 6,503,195 B1
• Filed: 05/24/1999
• Issued: 01/07/2003
• Est. Priority Date: 05/24/1999
• Status: Expired due to Fees

• Current Assignee: University of South Carolina
• Original Assignee: University of North Carolina At Chapel Hill (University of North Carolina System)
• Inventors: Keller, Kurtis P., Rosenthal, Michael H., Fuchs, Henry, State, Andrei, Ackerman, Jeremy D.
• Primary Examiner(s): MULCAHY, JOHN M
• Application Number:
  US09/317,932
• Time in Patent Office: 1,324 Days
• Field of Search: 600/109, 600/111, 600/117, 600/160, 600/166, 600/181, 348/45, 348/65
• US Class Current: 600/160
• CPC Class Codes: A61B 1/00096 Optical elements
• CPC Class Codes: A61B 1/00177 for 90 degrees side-viewing
• CPC Class Codes: A61B 1/00193 adapted for stereoscopic vision
• CPC Class Codes: A61B 1/00194 adapted for three-dimensional imaging
• CPC Class Codes: A61B 1/042 characterised by a proximal camera, e.g. a CCD camera
• CPC Class Codes: A61B 1/0605 for spatially modulated illumination
• CPC Class Codes: A61B 1/3132 for laparoscopy
• CPC Class Codes: G01B 11/25 by projecting a pattern, e.g. one or more lines, moiré fringes on the object G01B11/255 takes precedence ; image analysis for depth or shape recovery G06T7/50
• CPC Class Codes: G02B 23/2415 Stereoscopic endoscopes
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 23/555 for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes
• CPC Class Codes: H04N 5/7491 of head mounted projectors

Abstract

A real-time structured light depth extraction system includes a projector for projecting structured light patterns onto an object of interest. A camera positioned off-axis from the projector samples light reflected from the object synchronously with the projection of structured light patterns and outputs digital signals indicative of the reflected light. An image processor/controller receives the digital signals from the camera and processes the digital signals to extract depth information of the object in real time.

Claims

1. A real-time structured light depth extraction system for producing real-time images in an endoscopic surgical environment comprising:

(a) a projector for projecting structured light patterns onto an object inside of a patient'"'"'s body, each structured light pattern including a plurality of pixels being simultaneously projected onto the object;

(b) at least one endoscope optically coupled to the projector for communicating the structured light patterns from the projector to a first region inside of the patient'"'"'s body and for communicating reflected structured light patterns from the first region to a second region outside of the patient'"'"'s body;

(c) a camera optically coupled to the endoscope for sampling the reflected light patterns and outputting digital signals indicative of the reflected light patterns; and

(d) an image processor/controller coupled to the camera to receive the digital signals and extract depth information of the object in real time.

2. The real-time structured light depth extraction system of claim 1 wherein the endoscope comprises a single endoscope having a shared optical path for communicating the structured light patterns from the projector to the first region and for communicating the reflected light patterns from the first region to the camera.

3. The real-time structured light depth extraction system of claim 2 wherein the single endoscope comprises a laparoscope.

4. A real-time structured light depth extraction system for producing real-time images in an endoscopic surgical environment comprising:

(a) a projector for projecting structured light patterns onto an object inside of a patient'"'"'s body;

(b) at least one endoscope optically coupled to the projector for communicating the structured light patterns from the projector to a first region inside of the patient'"'"'s body and for communicating reflected structured light patterns from the first region to a second region outside of the patient'"'"'s body;

(c) a camera optically coupled to the endoscope for sampling the reflected light patterns and outputting digital signals indicative of the reflected light patterns; and

(d) an image processor/controller coupled to the camera to receive the digital signals and extract depth information of the object in real time, wherein the endoscope comprises a first endoscope for communicating the structured light patterns from the projector to the first region and a second endoscope for communicating the reflected light patterns from the first region to the camera.

5. The real-time structured light depth extraction system of claim 4 wherein the first and second endoscopes comprise first and second laparoscopes.

6. An augmented reality visualization system for endoscopic surgery comprising:

(a) at least one endoscope for viewing objects inside of a patient'"'"'s body;

(b) a real-time structured light depth extraction system coupled to the endoscope for projecting structured light patterns into the patient'"'"'s body, receiving light reflected from the objects, determining depth information relating to the objects in real time, and producing three-dimensional images of the objects;

(c) a graphics generator for generating synthetic images;

(d) an image merger for merging the three-dimensional images and the synthetic images to produce merged images having correct occlusion relationships; and

(e) a display for displaying the merged images to a viewer.

7. The augmented reality visualization system of claim 6 wherein the display comprises a head-mounted display.

8. The augmented reality visualization system of claim 6 wherein the display comprises an optical-see-through (OST) head-mounted display.

9. The augmented reality visualization system of claim 6 wherein the display comprises a video-see-through (VST) head-mounted display including first and second video cameras for producing first and second video streams indicative of the viewer'"'"'s surroundings.

10. The augmented reality visualization system of claim 9 comprising a first tracker for tracking position and orientation of the endoscope and outputting a first tracking signal based on the position and orientation of the endoscope and a second tracker for tracking position and orientation of the viewer'"'"'s head and outputting a second tracking signal based on the position and orientation of the viewer'"'"'s head, wherein the image merger merges the first and second video streams, the three-dimensional images, and the synthetic images to produce the merged images based on the depth information and the first and second tracking signals.

11. The augmented reality visualization system of claim 6 wherein the real-time structured light depth extraction system comprises:

(a) a projector optically coupled to the endoscope for projecting, from a first position, structured light patterns into the patient'"'"'s body;

(b) a camera optically coupled to the endoscope for receiving, from a second position, light reflected from the objects inside of the patient'"'"'s body synchronously with projection of the structured light patterns; and

(c) a depth extractor for calculating depth information relating to the objects inside of the patient'"'"'s body based on the structured light patterns, the light received by the camera, and the first and second positions.

12. The augmented reality visualization system of claim 6 wherein the endoscope comprises a single endoscope having a shared optical path for transmitting structured light patterns into the patient'"'"'s body and receiving light reflected from the objects inside of the patient'"'"'s body.

13. The augmented reality visualization system of claim 11 wherein the single endoscope comprises a single laparoscope.

14. The augmented reality visualization system of claim 6 wherein the endoscope comprises a first endoscope for communicating structured light patterns into the patient'"'"'s body and a second endoscope for receiving light reflected from the objects inside of the patient'"'"'s body.

15. The augmented reality visualization system of claim 14 wherein the first and second endoscopes comprise first and second laparoscopes.