SYSTEM AND METHOD OF USING HIGH-SPEED, HIGH-RESOLUTION DEPTH EXTRACTION TO PROVIDE THREE-DIMENSIONAL IMAGERY FOR ENDOSCOPY

• US 20110057930A1
• Filed: 11/10/2010
• Published: 03/10/2011
• Est. Priority Date: 07/26/2006
• Status: Abandoned Application

• Current Assignee: Inneroptic Technology, Inc.
• Original Assignee: Inneroptic Technology, Inc.
• Inventors: Keller, Kurtis P., State, Andrei, Green, Caroline K., Razzaque, Sharif A.
• Application Number:
  US12/943,795
• Publication Number:
  US 20110057930A1
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: A61B 1/0605 for spatially modulated illumination
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2207/10068 Endoscopic image
• CPC Class Codes: G06T 2210/41 Medical
• CPC Class Codes: G06T 2210/52 Parallel processing
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 23/555 for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes

Abstract

A system and method for providing high-speed, high-resolution three-dimensional imagery for endoscopy, particularly of a tissue surface at a medical procedure site is disclosed. High-resolution imagery provides greater detail of the tissue surface, but requires high-speed depth-frame imaging to provide timely updated depth information. A pattern of light, such as a point of light for example, may be projected onto the tissue surface and a reflected image analyzed to determine depth information. The point of light can be projected and analyzed quickly to produce faster depth-frame image rates. Three-dimensional structured-light depth resolution information may be generated and combined with either a two-dimensional image or a two-dimensional stereo image to provide three-dimensional imagery of the tissue surface. Switching between three-dimensional images and one of the two-dimensional image and a two-dimensional stereo image may also be provided. Further, the three-dimensional structured- light depth information may be further optimized by combining it with three-dimensional stereo-correspondence depth information to generate hybrid three-dimensional imagery.

Claims

1. A method of providing three-dimensional imagery of a surface, comprising:

receiving a first two-dimensional image of the surface, said first two-dimensional image being from a first two-dimensional imager;

receiving a second two-dimensional image of the surface, said second two-dimensional image being from a second two-dimensional imager;

receiving reflective light data related to the surface, said reflective light data being generated by a sensor receiving reflected light being sent from a light source;

generating, using one or more processors, a three-dimensional stereo-correspondence depth map based on the first two-dimensional image and the second two dimensional image;

generating, using one or more processors, a three-dimensional structured-light depth map of the surface based on the light data received from the sensor, said light being a reflection of projected light off the surface; and

generating, using one or more processors, a three-dimensional model of the surface based on the three-dimensional stereo-correspondence depth map and the three-dimensional structured-light depth map.

2. The method of claim 1, wherein the method further comprises rendering the generated three-dimensional model of the surface.

3. The method of claim 1, wherein the method further comprises:

causing a projection of a point of light onto the surface at a medical procedure site resulting in light reflecting off the surface;

determining depth characteristics of the surface based on brightness detected by the sensor, said sensor being other than a two-dimensional array imager; and

wherein generating a three-dimensional structured-light depth map comprises generating a three-dimensional structured-light depth map of the surface from the depth characteristics.

4. The method of claim 3, wherein causing a projection of a point of light comprises directing the projection of the point of light through a first channel of an endoscope.

5. The method of claim 1, wherein generating the three-dimensional model of the surface comprises generating a hybrid three-dimensional image by merging the three-dimension stereo-correspondence depth map and the three-dimensional structured-light depth map.

6. The method of claim 1, wherein generating the three-dimensional model of the surface comprises choosing between the three-dimensional stereo-correspondence depth map and the three-dimensional structured-light depth map.

7. The method of claim 1, wherein the surface is a surface at a medical procedure site.

8. A system for providing three-dimensional imagery of a surface, comprising one or more processors, said one or more processors being configured to:

receive a first two-dimensional image of the surface, said first two-dimensional image being from a first two-dimensional imager;

receive a second two-dimensional image of the surface, said second two-dimensional image being from a second two-dimensional imager;

receive reflective light data related to the surface, said reflective light data being generated by a sensor receiving reflected light being sent from a light source;

generate a three-dimensional stereo-correspondence depth map based on the first two-dimensional image and the second two dimensional image;

generate a three-dimensional structured-light depth map of the surface based on the light data received from the sensor, said light being a reflection of projected light off the surface; and

generate a three-dimensional model of the surface based on the three-dimensional stereo-correspondence depth map and the three-dimensional structured-light depth map.

9. The system of claim 8, wherein the system is further configured to render the generated three-dimensional model of the surface.

10. The system of claim 8, the system being further configured to:

cause a projection of a point of light onto the surface at a medical procedure site resulting in light reflecting off the surface;

determining depth characteristics of the surface based on brightness detected by the sensor, said sensor being other than a two-dimensional array imager; and

wherein generating a three-dimensional structured-light depth map comprises generating a three-dimensional structured-light depth map of the surface from the depth characteristics.

11. The system of claim 10, wherein causing a projection of a point of light comprises directing the projection of the point of light through a first channel of an endoscope.

12. The system of claim 10, wherein the sensor comprises a lateral effect photodiode (LEPD).

13. The system of claim 8, wherein generating three-dimensional model of the surface comprises generating a hybrid three-dimensional image by merging the three-dimensional stereo-correspondence depth map and the three-dimensional structured-light depth map.

14. The system of claim 8, wherein generating the three-dimensional model of the surface comprises choosing between the three-dimensional stereo-correspondence depth map and the three-dimensional structured-light depth map.

15. A non-transient computer-readable medium, said non-transient computer-readable media having computing instructions thereon, said computing instructions, when executed by one or more processors, causing the one or more processors to perform the following method:

receiving a first two-dimensional image of the surface, said first two-dimensional image being from a first two-dimensional imager;

receiving a second two-dimensional image of the surface, said second two-dimensional image being from a second two-dimensional imager;

receiving reflective light data related to the surface, said reflective light data being generated by a sensor receiving reflected light being sent from a light source;

generating, using one or more processors, a three-dimensional stereo-correspondence depth map based on the first two-dimensional image and the second two dimensional image;

generating, using one or more processors, a three-dimensional structured-light depth map of the surface based on the light data received from the sensor, said light being a reflection of projected light off the surface; and

generating, using one or more processors, a three-dimensional model of the surface based on the three-dimensional stereo-correspondence depth map and the three-dimensional structured-light depth map.

16. The computer-readable medium of claim 15, wherein the method further comprises rendering the generated three-dimensional model of the surface.

17. The computer-readable medium of claim 15, wherein the method further comprises:

causing a projection of a point of light onto the surface at a medical procedure site resulting in light reflecting off the surface;

determining depth characteristics of the surface based on brightness detected by the sensor, said sensor being other than a two-dimensional array imager; and

wherein generating a three-dimensional structured-light depth map comprises generating a three-dimensional structured-light depth map of the surface from the depth characteristics.

18. The computer-readable medium of claim 17, wherein causing a projection of a point of light comprises directing the projection of the point of light through a first channel of the endoscope.

19. The computer-readable media of claim 15, wherein generating the three-dimensional model of the surface comprises generating a hybrid three-dimensional image by merging the three-dimensional stereo-correspondence depth map and the three-dimensional structured-light depth map.

20. The computer-readable media of claim 15, wherein generating the three-dimensional model of the surface comprises choosing between the three-dimensional stereo-correspondence depth map and the three-dimensional structured-light depth map.