Guidance method based on 3D-2D pose estimation and 3D-CT registration with application to live bronchoscopy

• US 7,756,563 B2
• Filed: 05/19/2006
• Issued: 07/13/2010
• Est. Priority Date: 05/23/2005
• Status: Active Grant

• Current Assignee: Penn State Research Foundation (Pennsylvania State University)
• Original Assignee: Penn State Research Foundation (Pennsylvania State University)
• Inventors: Higgins, William E., Rai, Lav, Merritt, Scott A.
• Primary Examiner(s): Le; Long V
• Assistant Examiner(s): Selkin; Saurel J
• Application Number:
  US11/437,229
• Publication Number:
  US 20070015997A1
• Time in Patent Office: 1,516 Days
• Field of Search: 600/407, 600/423
• US Class Current: 600/407
• CPC Class Codes: A61B 1/000094 extracting biological structures
• CPC Class Codes: A61B 2034/105 Modelling of the patient, e.g. for ligaments or bones
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2090/364 Correlation of different images or relation of image positions in respect to the body
• CPC Class Codes: A61B 2090/3762 using computed tomography systems [CT]
• CPC Class Codes: A61B 34/10 Computer-aided planning, simulation or modelling of surgical operations
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 5/06 Devices, other than using radiation, for detecting or locating foreign bodies for removing same A61B17/50 ; determining position of probes within or on the body of the patient
• CPC Class Codes: A61B 5/415 the glands, e.g. tonsils, adenoids or thymus
• CPC Class Codes: A61B 5/418 lymph vessels, ducts or nodes
• CPC Class Codes: A61B 6/032 Transmission computed tomography [CT]
• CPC Class Codes: A61B 6/466 adapted to display 3D data
• CPC Class Codes: A61B 6/50 specially adapted for specific body parts; specially adapted for specific clinical applications
• CPC Class Codes: A61B 90/36 Image-producing devices or illumination devices not otherwise provided for
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: G06V 10/754 involving a deformation of the sample pattern or of the reference pattern; Elastic matching
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces

Abstract

A method provides guidance to the physician during a live bronchoscopy or other endoscopic procedures. The 3D motion of the bronchoscope is estimated using a fast coarse tracking step followed by a fine registration step. The tracking is based on finding a set of corresponding feature points across a plurality of consecutive bronchoscopic video frames, then estimating for the new pose of the bronchoscope. In the preferred embodiment the pose estimation is based on linearization of the rotation matrix. By giving a set of corresponding points across the current bronchoscopic video image, and the CT-based virtual image as an input, the same method can also be used for manual registration. The fine registration step is preferably a gradient-based Gauss-Newton method that maximizes the correlation between the bronchoscopic video image and the CT-based virtual image. The continuous guidance is provided by estimating the 3D motion of the bronchoscope in a loop. Since depth-map information is available, tracking can be done by solving a 3D-2D pose estimation problem. A 3D-2D pose estimation problem is more constrained than a 2D-2D pose estimation problem and does not suffer from the limitations associated with computing an essential matrix. The use of correlation-based cost, instead of mutual information as a registration cost, makes it simpler to use gradient-based methods for registration.

Claims

1. A method of providing guidance in real-time or near real-time in conjunction with an endoscopic procedure, comprising the steps of:

providing previously acquired three-dimensional (3D) image data of a body lumen;

receiving a continuous live sequence of endoscopic video image data of the body lumen during a procedure from an endoscope; and

registering the previously acquired 3D image data of the body lumen and the live endoscopic video image data of the body lumen to provide guidance to a user of the endoscope wherein the step of registering includes estimating the 3D location of the endoscope using the live sequence of endoscopic video image data and the previously acquired 3D image data.

2. The method of claim 1, wherein the step of registering the previously acquired 3D image data and the live video image data is used to guide said endoscope during the procedure.

3. The method of claim 1, wherein the previously acquired 3D image data is derived from virtual image data.

4. The method of claim 1, wherein the previously acquired 3D image data is derived from computerized tomographic (CT) image data.

5. The method of claim 1, wherein the previously acquired 3D image data is derived from a prerecorded video source.

6. The method of claim 1, wherein the registration step includes the steps of:

a) estimating a three-dimensional location of the endoscope using (i) known motion information from said received video image data, and (ii) local depth information obtained from the previously acquired 3D image data; and

b) determining a new pose of the endoscope based on the 3D location estimated in step (a).

7. The method of claim 1, wherein said registration step is performed after a sequence of consecutive video image frames is received.

8. The method of claim 2, wherein the endoscope is a bronchoscope.

9. The method of claim 8, further comprising the step of advancing the bronchoscope from a bifurcation in an airway tree.

10. The method of claim 6, wherein the step of determining a new pose is based on linearization of a rotation matrix.

11. The method of claim 6, further including the step of performing a fine registration step to minimize errors associated with estimating the 3D motion of the endoscope.

12. The method of claim 6, wherein the registration step further includes the steps of:

(c) updating the previously acquired 3D image data in accordance with the new pose; and

(d) repeating steps (a) through (c) until the guidance is terminated.

13. The method of claim 6, wherein the previously acquired 3D image data are derived from a magnetic resonance image (MRI) image data or computerized tomographic (CT) slices.

14. The method of claim 6, further comprising the step of rendering a virtual image frame of the body lumen corresponding to the new pose of the endoscope.

15. The method of claim 11, wherein the fine registration step is based upon a gradient-based Gauss-Newton method.

29. The method of claim 7, wherein said sequence of video image frames is 5 consecutive video frames.

16. A method of providing guidance in conjunction with an endoscopic procedure, comprising the steps of:

providing previously acquired three-dimensional (3D) image data of a body lumen;

acquiring a continuous live sequence of endoscopic video image data of the body lumen during the procedure from an endoscope; and

registering and tracking the previously acquired 3D image data and the live endoseopic video image data in real time or near real-time to obtain a current pose of the endoscope; and

wherein the step of registering and tracking is performed by;

a) estimating the 3D location of the endoscope using the steps of;

registering the previously acquired 3D image data and live video image data to obtain a current depth map;

choosing a plurality of points associated with the live video image data;

tracking the points over a plurality of consecutive frames to estimate the two-dimensional (2D) motion of the tracked points; and

using the 2D motion of the tracked points and the current depth map to derive the three-dimensional (3D) motion of the instrument;

b) determining a new pose based on the estimating step;

c) updating the current pose based upon the new pose; and

d) repeating steps a) through c) to complete guidance.

17. The method of claim 16, wherein:

3D locations of the plurality of points and corresponding 2D images are determined using perspective projection.

18. The method of claim 16, wherein the estimating step is based on a linearization of a rotation matrix.

19. A system for guiding an endoscope through a body lumen, comprising:

a database storing previously acquired 3D image data of a body lumen;

an input for receiving a continuous live sequence of endoscopic video image data of the body lumen; and

a processor in communication with the database and the input, the processor being configured to perform the following functions;

estimate the 3D location of the actual physical location of the endoscope based on the live endoscopic video image data and the previously acquired 3D image data, and

register the previously acquired 3D image data and the live endoscopic video image data in real time or near real-time, thereby providing guidance to a user of the endoscope.

20. The system of claim 19, wherein the database storing previously acquired 3D image data of the body lumen includes depth information.

21. The system of claim 19, wherein the processor is further configured to perform a fine registration step to minimize errors associated with estimating three-dimensional (3D) motion.

22. The system of claim 19, wherein the previously acquired 3D image data are derived from one of a magnetic resonance image (MRI) image data and computerized tomographic (CT) slices.

23. The system of claim 19, wherein the endoscope input is a bronchoscope input to obtain the live video image data.

24. The system of claim 20, wherein the estimate function is further configured to:

a) estimate three-dimensional (3D) location using the depth information and tracking of a plurality of points associated with the live video image data; and

b) perform a pose estimation to determine a new pose of the endoscope.

25. The system of claim 24, wherein the processor is further configured to linearize a rotation matrix to perform the pose estimation.

26. The method of claim 24, wherein said plurality of points is 20 points.

27. The system of claim 21, wherein the fine registration uses a gradient-based Gauss-Newton method.

28. The system of claim 21, wherein the fine registration includes:

computing a virtual image of a body lumen;

applying a correlation-based cost to register the virtual image and the live video image data.