Guidance method based on 3D-2D pose estimation and 3D-CT registration with application to live bronchoscopy

• US 20070015997A1
• Filed: 05/19/2006
• Published: 01/18/2007
• Est. Priority Date: 05/23/2005
• Status: Active Grant

• Current Assignee: Penn State Research Foundation
• Original Assignee: Penn State Research Foundation
• Inventors: Rai, Lav, Higgins, William, Merritt, Scott
• Granted Patent: US 7,756,563 B2
• Time in Patent Office: Days
• US Class Current: 600/407
• CPC Class Codes: A61B 1/000094 extracting biological structures
• CPC Class Codes: A61B 2034/105 Modelling of the patient, e.g. for ligaments or bones
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2090/364 Correlation of different images or relation of image positions in respect to the body
• CPC Class Codes: A61B 2090/3762 using computed tomography systems [CT]
• CPC Class Codes: A61B 34/10 Computer-aided planning, simulation or modelling of surgical operations
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 5/06 Devices, other than using radiation, for detecting or locating foreign bodies for removing same A61B17/50 ; determining position of probes within or on the body of the patient
• CPC Class Codes: A61B 5/415 the glands, e.g. tonsils, adenoids or thymus
• CPC Class Codes: A61B 5/418 lymph vessels, ducts or nodes
• CPC Class Codes: A61B 6/032 Transmission computed tomography [CT]
• CPC Class Codes: A61B 6/466 adapted to display 3D data
• CPC Class Codes: A61B 6/50 specially adapted for specific body parts; specially adapted for specific clinical applications
• CPC Class Codes: A61B 90/36 Image-producing devices or illumination devices not otherwise provided for
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: G06V 10/754 involving a deformation of the sample pattern or of the reference pattern; Elastic matching
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces

Abstract

A method provides guidance to the physician during a live bronchoscopy or other endoscopic procedures. The 3D motion of the bronchoscope is estimated using a fast coarse tracking step followed by a fine registration step. The tracking is based on finding a set of corresponding feature points across a plurality of consecutive bronchoscopic video frames, then estimating for the new pose of the bronchoscope. In the preferred embodiment the pose estimation is based on linearization of the rotation matrix. By giving a set of corresponding points across the current bronchoscopic video image, and the CT-based virtual image as an input, the same method can also be used for manual registration. The fine registration step is preferably a gradient-based Gauss-Newton method that maximizes the correlation between the bronchoscopic video image and the CT-based virtual image. The continuous guidance is provided by estimating the 3D motion of the bronchoscope in a loop. Since depth-map information is available, tracking can be done by solving a 3D-2D pose estimation problem. A 3D-2D pose estimation problem is more constrained than a 2D-2D pose estimation problem and does not suffer from the limitations associated with computing an essential matrix. The use of correlation-based cost, instead of mutual information as a registration cost, makes it simpler to use gradient-based methods for registration.

Claims

1. A method of providing guidance in conjunction with a diagnostic procedure, comprising the steps of:

providing previously acquired image data of a body lumen;

acquiring live image data of the body lumen; and

registering the previously acquired image data and the live image data in real time or near real-time.

2. The method of claim 1, wherein the step of registering the previously acquired image data and the live image data is used to guide an instrument during the diagnostic procedure.

3. The method of claim 2, wherein the instrument is an endoscope, bronchoscope, colonoscope or laparoscope.

4. The method of claim 1, wherein the previously acquired image data is derived from virtual image data.

5. The method of claim 1, wherein the previously acquired image data is derived from computerized tomographic (CT) image data.

6. The method of claim 1, wherein the previously acquired image data is derived from a prerecorded video source.

7. The method of claim 1, wherein the live image data is derived from video data acquired during the diagnostic procedure.

8. The method of claim 1, wherein the live image data is derived from a stream of incoming virtual images.

9. The method of claim 1, including the steps of:

a) providing previously acquired image data including depth information;

b) using an instrument to provide the live image data;

c) estimating the three-dimensional (3D) motion of the instrument using the depth information and the tracking of points associated with the live video image;

d) determining a new pose by solving a 3D-2D pose estimation problem;

e) updating the previously acquired image data based upon the new pose; and

f) repeating steps c) through e) in system to complete the procedure.

10. The method of claim 9, wherein pose estimation is based on the linearization of a rotation matrix.

11. The method of claim 9, further including the step of performing a fine registration step to minimize errors associated with estimating the 3D motion of the instrument.

12. The method of claim 11, wherein the fine registration step is based upon a gradient-based Gauss-Newton method.

13. The method of claim 11, wherein the fine registration step includes the steps of:

computing a virtual image of a body lumen; and

minimizing the cost between the virtual image and a live video image of the body lumen.

14. The method of claim 9, wherein the previously acquired images are derived from computerized tomographic (CT) slices.

15. The method of claim 9, wherein the instrument is an endoscope, bronchoscope, colonoscope or laparoscope.

16. The method of claim 9, including the steps of:

registering the previously acquired and live image data to obtain a current depth map;

choosing a plurality of points associated with the live image data;

tracking the points over a plurality of consecutive frames to estimate the two-dimensional (2D) motion of the tracked points; and

using the 2D motion of the tracked points and the current depth map to derive the three-dimensional (3D) motion of the instrument.

17. The method of claim 16, wherein:

the 3D locations (W_i) of n points in one of the consecutive frames and their 2D images (u_i,v_i) are known through perspective projection in another reference frame; and

pose estimation solves for rotation and translation (R,T) using W_i and (u_i,v_i).

18. The method of claim 16, wherein R is a three-dimensional matrix that is linearized during pose estimation.

19. A system for guiding a viewing instrument through a body lumen, comprising:

a database storing previously acquired image data of the body lumen;

an input for receiving live image data of the body lumen; and

a processor in communication with the database and the input, the processor being operative to register the previously acquired image data and the live image data in real time or near real-time.

20. The system of claim 19, wherein the database storing previously acquired image data of the body lumen includes depth information.

21. The system of claim 19, wherein the processor is further operative to:

a) estimate three-dimensional motion using the depth information and the tracking of points associated with the live image data; and

b) solve a 3D-2D pose estimation problem to determine a new pose.

22. The system of claim 21, wherein the processor is further operative to linearize a rotation matrix to solve the 3D-2D pose estimation problem.

23. The system of claim 19, wherein the processor is further operative to perform a fine registration step to minimize errors associated with estimating three-dimensional motion.

24. The system of claim 23, wherein the fine registration uses a gradient-based Gauss-Newton method.

25. The system of claim 23, wherein the fine registration includes:

computing a virtual image of a body lumen;

minimizing the cost between the virtual image and the live image data.

26. The system of claim 19, wherein the previously acquired image data are derived from computerized tomographic (CT) slices.

27. The system of claim 19, further including an endoscope, bronchoscope, colonoscope or laparoscope to obtain the live image data.