Method and apparatus for continuous guidance of endoscopy

• US 8,672,836 B2
• Filed: 01/30/2008
• Issued: 03/18/2014
• Est. Priority Date: 01/31/2007
• Status: Active Grant

• Current Assignee: Penn State Research Foundation
• Original Assignee: Penn State Research Foundation
• Inventors: Higgins, William E., Merritt, Scott A., Rai, Lav, Gibbs, Jason D., Yu, Kun-Chang
• Primary Examiner(s): Leubecker, John P
• Assistant Examiner(s): COLQUE, RONALD DAVID
• Application Number:
  US12/022,620
• Publication Number:
  US 20080207997A1
• Time in Patent Office: 2,239 Days
• Field of Search: 600/117, 600/111, 600/103, 600/109, 600/417, 600/118, 600/433, 345/427, 345/630, 345/424, 348/65, 348/45, 606/21
• US Class Current: 600/117
• CPC Class Codes: A61B 1/000094 extracting biological structures
• CPC Class Codes: A61B 1/0005 combining images e.g. side-by-side, superimposed or tiled
• CPC Class Codes: A61B 2034/105 Modelling of the patient, e.g. for ligaments or bones
• CPC Class Codes: A61B 2034/107 Visualisation of planned trajectories or target regions
• CPC Class Codes: A61B 2090/365 augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/25 User interfaces for surgical systems
• CPC Class Codes: A61B 90/36 Image-producing devices or illumination devices not otherwise provided for
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras

Abstract

Methods and apparatus provide continuous guidance of endoscopy during a live procedure. A data-set based on 3D image data is pre-computed including reference information representative of a predefined route through a body organ to a final destination. A plurality of live real endoscopic (RE) images are displayed as an operator maneuvers an endoscope within the body organ. A registration and tracking algorithm registers the data-set to one or more of the RE images and continuously maintains the registration as the endoscope is locally maneuvered. Additional information related to the final destination is then presented enabling the endoscope operator to decide on a final maneuver for the procedure. The reference information may include 3D organ surfaces, 3D routes through an organ system, or 3D regions of interest (ROIs), as well as a virtual endoscopic (VE) image generated from the precomputed data-set. The preferred method includes the step of superimposing one or both of the 3D routes and ROIs on one or both of the RE and VE images. The 3D organ surfaces and routes may correspond to the surfaces and paths of a tracheobronchial airway tree extracted, for example, from 3D MDCT images of the chest.

Claims

1. A method for continuous guidance of endoscopy during a live procedure, comprising the steps of:

a) providing a precomputed data-set based on 3D image data, the data-set including reference information representative of a predefined route through a body organ to a final destination, the reference information including virtual endoscopic (VE) image data representing one or more of the following;

3D organ surfaces, 3D routes through an organ system, and 3D regions of interest (ROIs);

b) displaying a plurality of live real endoscopic (RE) images as an operator maneuvers an endoscope within the body organ;

c) presenting information, corresponding to an initial reference location along the predefined route, which enables an endoscope operator to move the endoscope toward the reference location;

d) invoking a registration/tracking algorithm that registers the VE image data to one or more of the RE images and continuously maintains the registration as the endoscope is locally maneuvered;

e) presenting information corresponding to another reference location along the predefined route, which enables the endoscope operator to move the endoscope close to this new reference location;

f) repeating steps d)-e) a plurality of times until the endoscope is within the vicinity of the final destination; and

g) when the final destination is within the field of view of the endoscope, providing additional information enabling the endoscope operator to decide on a final maneuver for the procedure, the additional information including an icon, other the ROI itself, superimposed on at least one of the VE and RE images to visually indicate a direction from the final destination, including a visual indication of where to penetrate through the wall of the body organ, to the ROI.

2. The method of claim 1, including the step of superimposing one or both of the 3D routes and the ROIs on one or both of the RE and VE image data.

3. The method of claim 2, wherein one superimposed route is highlighted and continuously presented to guide the endoscope operator in steps c)-g).

4. The method of claim 3, wherein the registered VE view is displayed synchronously with the live RE video.

5. The method of claim 1, wherein the 3D organ surfaces correspond to the surfaces of a tracheobronchial airway tree extracted from 3D MDCT images of the chest.

6. The method of claim 1, wherein the 3D routes correspond to paths traversing a tracheobronchial airway tree extracted from 3D MDCT images of the chest or traversing other neighboring anatomical structures.

7. The method of claim 1, wherein the ROIs represent one tissue selected from the group consisting of lymph nodes and suspect tumors.

8. The method of claim 1, including the step of halting the registration/tracking algorithm prior to step e).

9. The method of claim 8, wherein a saved registration view is presented after halting the registration/tracking algorithm.

10. The method of claim 1, wherein step d) is done in real time.

11. The method of claim 1 wherein said introducing final maneuver information comprises rendering a portion of the body lumen wall semi-transparent to enhance visualization of the ROI on a display.

12. A system for continuous guidance of endoscopy during a live procedure, comprising:

an endoscope maneuvered by an operator along a route to a final destination within a body organ;

a display device operative to display live, real endoscopic (RE) images obtained by the endoscope;

a memory storing a precomputed data-set derived from 3D image data, the data-set including reference information representative of a predefined route through a body organ to a final destination, the reference information including virtual endoscopic (VE) image data representing one or more of the following;

3D organ surfaces, 3D routes through an organ system, and 3D regions of interest (ROIs);

a processor in communication with the memory and display, the processor being operative to;

a) continuously register the VE image data to one or more of the RE images by displaying and updating the reference information as the operator maneuvers the endoscope, thereby continuously tracking the endoscope and guiding the operator to the final destination on the display device, and

b) when the final destination is within the field of view of the endoscope, provide additional information enabling the endoscope operator to decide on a final maneuver for the procedure, the additional information including an icon, other than the ROI itself, superimposed on at least one of the VE and RE images to visually indicate a direction from the final destination, including a visual indication of where to penetrate through the wall of the body organ, to the ROI.

13. The system of claim 12, wherein one or both of the 3D routes and ROIs are superimposed on one or both of the RE and VE images.

14. The system of claim 13, wherein the processor is operative to highlight a superimposed route on the display as an operator maneuvers the endoscope.

15. The system of claim 14, wherein the VE view is displayed synchronously with the live RE video.

16. The system of claim 12, wherein the 3D organ surfaces correspond to the surfaces of a tracheobronchial airway tree extracted from 3D MDCT images of the chest.

17. The system of claim 12, wherein the 3D routes correspond to paths traversing the tracheobronchial airway tree extracted from 3D MDCT images of the chest or traversing other neighboring anatomical structures.

18. The system of claim 12, wherein the ROIs represent lymph nodes, suspect tumors, narrowed airways, or any other diagnostically relevant regions.

19. The system of claim 12, further including an operator control to halt the registration process.

20. The system of claim 19, wherein a saved registration view is presented on the display after halting.

21. The system of claim 12, wherein the registration is done in real time.

22. A method for guiding an endoscope through an airway to a ROI in a lung during a live procedure, comprising the steps of:

a) receiving a virtual endoscopic (VE) image corresponding to a location along a precomputed route to said ROI;

b) receiving a live real endoscopic (RE) image as an operator maneuvers an endoscope through said airway;

c) superimposing said precomputed route onto at least one of said RE and VE images;

d) registering said VE image to said RE image as the endoscope is maneuvered through said airway; and

e) when the final destination is within the field of view of the endoscope, displaying information enabling the endoscope operator to decide on a final maneuver for the procedure, the information including an icon, other than the ROI itself, superimposed on at least one of the VE and RE images to visually indicate a direction from the final destination, including a visual indication of where to penetrate through the wall of the body organ, to the ROI.

23. The method of claim 22 wherein said final maneuver information comprises an icon superimposed in at least one of said VE and RE images.

24. The method of claim 23 wherein said icon is adapted to visually indicate a direction to the ROI from an end of the precomputed route.

25. The method of claim 24 wherein said icon is shaped as an arrow.

26. The method of claim 23 wherein said icon appears different as the endoscope moves relative to the ROI.

27. The method of claim 23 wherein said icon is representative of multiple final maneuver information.

28. The method of claim 27 wherein the multiple final maneuver information comprises a) direction and b) proximity to the ROI.

29. The method of claim 22 comprising providing a computer to perform each of said steps a) to e).

30. The method of claim 22 comprising repeating steps a) to d) until said endoscope is in the vicinity of the ROI.

31. The method of claim 22 wherein said preparing final maneuver information comprises rendering a portion of the airway wall semi-transparent to enhance visualization of the ROI on a display.