Guidance method based on 3D-2D pose estimation and 3D-CT registration with application to live bronchoscopy
First Claim
1. A method of providing guidance in real-time or near real-time in conjunction with an endoscopic procedure, comprising the steps of:
- providing previously acquired three-dimensional (3D) image data of a body lumen;
receiving a continuous live sequence of endoscopic video image data of the body lumen during a procedure from an endoscope; and
registering the previously acquired 3D image data of the body lumen and the live endoscopic video image data of the body lumen to provide guidance to a user of the endoscope wherein the step of registering includes estimating the 3D location of the endoscope using the live sequence of endoscopic video image data and the previously acquired 3D image data.
10 Assignments
0 Petitions
Accused Products
Abstract
A method provides guidance to the physician during a live bronchoscopy or other endoscopic procedures. The 3D motion of the bronchoscope is estimated using a fast coarse tracking step followed by a fine registration step. The tracking is based on finding a set of corresponding feature points across a plurality of consecutive bronchoscopic video frames, then estimating for the new pose of the bronchoscope. In the preferred embodiment the pose estimation is based on linearization of the rotation matrix. By giving a set of corresponding points across the current bronchoscopic video image, and the CT-based virtual image as an input, the same method can also be used for manual registration. The fine registration step is preferably a gradient-based Gauss-Newton method that maximizes the correlation between the bronchoscopic video image and the CT-based virtual image. The continuous guidance is provided by estimating the 3D motion of the bronchoscope in a loop. Since depth-map information is available, tracking can be done by solving a 3D-2D pose estimation problem. A 3D-2D pose estimation problem is more constrained than a 2D-2D pose estimation problem and does not suffer from the limitations associated with computing an essential matrix. The use of correlation-based cost, instead of mutual information as a registration cost, makes it simpler to use gradient-based methods for registration.
-
Citations
29 Claims
-
1. A method of providing guidance in real-time or near real-time in conjunction with an endoscopic procedure, comprising the steps of:
-
providing previously acquired three-dimensional (3D) image data of a body lumen; receiving a continuous live sequence of endoscopic video image data of the body lumen during a procedure from an endoscope; and registering the previously acquired 3D image data of the body lumen and the live endoscopic video image data of the body lumen to provide guidance to a user of the endoscope wherein the step of registering includes estimating the 3D location of the endoscope using the live sequence of endoscopic video image data and the previously acquired 3D image data. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 29)
-
-
16. A method of providing guidance in conjunction with an endoscopic procedure, comprising the steps of:
-
providing previously acquired three-dimensional (3D) image data of a body lumen; acquiring a continuous live sequence of endoscopic video image data of the body lumen during the procedure from an endoscope; and registering and tracking the previously acquired 3D image data and the live endoseopic video image data in real time or near real-time to obtain a current pose of the endoscope; and wherein the step of registering and tracking is performed by; a) estimating the 3D location of the endoscope using the steps of; registering the previously acquired 3D image data and live video image data to obtain a current depth map; choosing a plurality of points associated with the live video image data; tracking the points over a plurality of consecutive frames to estimate the two-dimensional (2D) motion of the tracked points; and using the 2D motion of the tracked points and the current depth map to derive the three-dimensional (3D) motion of the instrument; b) determining a new pose based on the estimating step; c) updating the current pose based upon the new pose; and d) repeating steps a) through c) to complete guidance. - View Dependent Claims (17, 18)
-
-
19. A system for guiding an endoscope through a body lumen, comprising:
-
a database storing previously acquired 3D image data of a body lumen; an input for receiving a continuous live sequence of endoscopic video image data of the body lumen; and a processor in communication with the database and the input, the processor being configured to perform the following functions; estimate the 3D location of the actual physical location of the endoscope based on the live endoscopic video image data and the previously acquired 3D image data, and register the previously acquired 3D image data and the live endoscopic video image data in real time or near real-time, thereby providing guidance to a user of the endoscope. - View Dependent Claims (20, 21, 22, 23, 24, 25, 26, 27, 28)
-
Specification