GUIDANCE METHOD BASED ON 3D-2D POSE ESTIMATION AND 3D-CT REGISTRATION WITH APPLICATION TO LIVE BRONCHOSCOPY

• US 20100280365A1
• Filed: 07/12/2010
• Published: 11/04/2010
• Est. Priority Date: 05/23/2005
• Status: Abandoned Application

• Current Assignee: Penn State Research Foundation
• Original Assignee: Penn State Research Foundation
• Inventors: Higgins, William E., Rai, Lav, Merritt, Scott A.
• Application Number:
  US12/834,076
• Publication Number:
  US 20100280365A1
• Time in Patent Office: Days
• US Class Current: 600/424
• CPC Class Codes: A61B 1/000094 extracting biological structures
• CPC Class Codes: A61B 2034/105 Modelling of the patient, e.g. for ligaments or bones
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2090/364 Correlation of different images or relation of image positions in respect to the body
• CPC Class Codes: A61B 2090/3762 using computed tomography systems [CT]
• CPC Class Codes: A61B 34/10 Computer-aided planning, simulation or modelling of surgical operations
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 5/06 Devices, other than using radiation, for detecting or locating foreign bodies for removing same A61B17/50 ; determining position of probes within or on the body of the patient
• CPC Class Codes: A61B 5/415 the glands, e.g. tonsils, adenoids or thymus
• CPC Class Codes: A61B 5/418 lymph vessels, ducts or nodes
• CPC Class Codes: A61B 6/032 Transmission computed tomography [CT]
• CPC Class Codes: A61B 6/466 adapted to display 3D data
• CPC Class Codes: A61B 6/50 specially adapted for specific body parts; specially adapted for specific clinical applications
• CPC Class Codes: A61B 90/36 Image-producing devices or illumination devices not otherwise provided for
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: G06V 10/754 involving a deformation of the sample pattern or of the reference pattern; Elastic matching
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces

Abstract

A method provides guidance to the physician during a live bronchoscopy or other endoscopic procedures. The 3D motion of the bronchoscope is estimated using a fast coarse tracking step followed by a fine registration step. The tracking is based on finding a set of corresponding feature points across a plurality of consecutive bronchoscopic video frames, then estimating for the new pose of the bronchoscope. In the preferred embodiment the pose estimation is based on linearization of the rotation matrix. By giving a set of corresponding points across the current bronchoscopic video image, and the CT-based virtual image as an input, the same method can also be used for manual registration. The fine registration step is preferably a gradient-based Gauss-Newton method that maximizes the correlation between the bronchoscopic video image and the CT-based virtual image. The continuous guidance is provided by estimating the 3D motion of the bronchoscope in a loop. Since depth-map information is available, tracking can be done by solving a 3D-2D pose estimation problem. A 3D-2D pose estimation problem is more constrained than a 2D-2D pose estimation problem and does not suffer from the limitations associated with computing an essential matrix. The use of correlation-based cost, instead of mutual information as a registration cost, makes it simpler to use gradient-based methods for registration.

Claims

1. A video-based method for providing a pose estimate of an endoscope in conjunction with a live endoscopic procedure, the method comprising:

acquiring 3D image data of a target structure in advance of a live endoscopic procedure;

receiving a frame of live endoscopic video image data including the target structure; and

registering the frame of endoscopic video image data with acquired 3D image data to determine a pose estimate of the endoscope.

2. The method of claim 1, wherein the pose estimate is used to guide the endoscope to a suspect site.

3. The method of claim 2, wherein the suspect site is a mediastinal lymph node.

4. The method of claim 1, wherein the endoscope is a bronchoscope.

5. The method of claim 1, wherein the target structure is the airway tree.

6. The method of claim 5, wherein the registration is performed either near to, or far from, a bifurcation I an airway tree.

7. The method of claim 1, wherein the registration comprises an initial registration step to initiate guidance of the endoscope.

8. The method of claim 1, wherein the registration step includes the steps of:

a) estimating a three-dimensional location of the endoscope using (i) known motion information from said live endoscopic video image data, and (ii) local depth information obtained from the previously acquired 3D image data; and

b) determining a new pose of the endoscope based on the 3D location estimated in step (a).

9. The method of claim 8, further including the step of performing a fine registration step to minimize errors associated with estimating the 3D motion of the endoscope.

10. The method of claim 9, wherein the fine registration step is performed manually.

11. The method of claim 8, wherein the registration step further includes the steps of:

(c) updating the previously acquired 3D image data in accordance with the new pose; and

(d) repeating steps (a) through (c) until the guidance is terminated.

12. The method of claim 1, including the step of receiving a sequence of consecutive video image frames as opposed to a single frame.

13. The method of claim 12, wherein the sequence of video image frames is 5 consecutive video frames.

14. A video-based method for registering previously acquired 3D image data and live endoscopic video image data of a patient to obtain a current pose of an endoscope comprising:

a) performing an initial registration to register the previously acquired 3D image data and a frame of the live video image data to obtain a current pose and a current depth map;

b) selecting a plurality of points associated with the live video image data;

c) tracking the points over a plurality of consecutive frames to estimate the two-dimensional (2D) motion of the tracked points;

d) deriving a three-dimensional (3D) motion of the endoscope using the 2D motion of the tracked points and the current depth map;

e) determining a new pose based on the current depth map and 3D motion of the endoscope; and

f) updating the current pose based upon the new pose.

15. The method of claim 14, wherein steps b) through f) are repeated until reaching a suspect site.

16. The method of claim 15, wherein said site is along an airway.

17. A video-based method for providing a pose estimate of an endoscope in conjunction with a live endoscopic procedure, the method comprising:

receiving a frame of live endoscopic video image data of a target structure from the endoscope; and

registering previously acquired 3D image data of the target structure and the frame of endoscopic video image data of the target structure to provide a pose estimate of the endoscope; and

wherein the step of registering is performed using information arising from said frame of endoscopic video image data and the previously acquired 3D image data.

18. The method of claim 17, wherein the step of registering the previously acquired 3D image data and the live video image data is used for guiding said endoscope to a suspect site.

19. The method of claim 18, wherein said suspect site is a mediastinal lymph node.

20. The method of claim 17, wherein the endoscope is a bronchoscope.

21. The method of claim 17, wherein said target structure is the airway tree.

22. The method of claim 21, wherein said registration is performed either near or far from a bifurcation.

23. The method of claim 17, wherein said registering step is an initial registration step to start guidance of the endoscope.

24. The method of claim 18, wherein the registration step includes the steps of:

a) estimating a three-dimensional location of the endoscope using (i) known motion information from said live endoscopic video image data, and (ii) local depth information obtained from the previously acquired 3D image data; and

b) determining a new pose of the endoscope based on the 3D location estimated in step (a).

25. The method of claim 24, further including the step of performing a fine registration step to minimize errors associated with estimating the 3D motion of the endoscope.

26. The method of claim 25, wherein the fine registration step is performed manually.

27. The method of claim 24, wherein the registration step further includes the steps of:

(c) updating the previously acquired 3D image data in accordance with the new pose; and

(d) repeating steps (a) through (c) until the guidance is terminated.

28. The method of claim 17, wherein said receiving comprises receiving a sequence of consecutive video image frames.

29. The method of claim 28, wherein said sequence of video image frames is 5 consecutive video frames.

30. A video-based method for registering previously acquired 3D image data and live endoscopic video image data of a patient to obtain a current pose of an endoscope comprising:

g) performing an initial registration to register the previously acquired 3D image data and a frame of the live video image data to obtain a current pose and a current depth map;

h) selecting a plurality of points associated with the live video image data;

i) tracking the points over a plurality of consecutive frames to estimate the two-dimensional (2D) motion of the tracked points;

j) deriving a three-dimensional (3D) motion of the endoscope using the 2D motion of the tracked points and the current depth map;

k) determining a new pose based on the current depth map and 3D motion of the endoscope; and

l) updating the current pose based upon the new pose.

31. The method of claim 30 comprising repeating steps b) through f) until reaching a suspect site.

32. The method of claim 31, wherein said site is along an airway.