Fast 3D-2D image registration system with application to continuously guided endoscopy
First Claim
1. A system for registering images associated with a continuously guided endoscopic procedure, comprising:
- a memory storing a plurality of reference images of a body lumen, each reference image including a viewpoint with a depth map and image gradients;
a connector adapted for connection to an endoscope for receiving therefrom one or more live video frames associated with a body lumen; and
a processor operative to register each video frame received from the endoscope to at least one of the reference images, the processor being programmed to perform the steps of;
a) warping the video frame to the viewpoint of one reference image of the plurality of reference images or warping one reference image of the plurality of reference images at a viewpoint to the video frame;
b) automatically computing an image difference between (i) the warped video frame and the reference image, or (ii) the warped reference image and the video frame;
c) updating a viewpoint of the video frame using an optimization technique; and
d) repeating steps a) through c) for each video frame until the viewpoint of the video frame converges on a final viewpoint or a next video frame becomes available.
8 Assignments
0 Petitions
Accused Products
Abstract
A novel framework for fast and continuous registration between two imaging modalities is disclosed. The approach makes it possible to completely determine the rigid transformation between multiple sources at real-time or near real-time frame-rates in order to localize the cameras and register the two sources. A disclosed example includes computing or capturing a set of reference images within a known environment, complete with corresponding depth maps and image gradients. The collection of these images and depth maps constitutes the reference source. The second source is a real-time or near-real time source which may include a live video feed. Given one frame from this video feed, and starting from an initial guess of viewpoint, the real-time video frame is warped to the nearest viewing site of the reference source. An image difference is computed between the warped video frame and the reference image. The viewpoint is updated via a Gauss-Newton parameter update and certain of the steps are repeated for each frame until the viewpoint converges or the next video frame becomes available. The final viewpoint gives an estimate of the relative rotation and translation between the camera at that particular video frame and the reference source. The invention has far-reaching applications, particularly in the field of assisted endoscopy, including bronchoscopy and colonoscopy. Other applications include aerial and ground-based navigation.
-
Citations
19 Claims
-
1. A system for registering images associated with a continuously guided endoscopic procedure, comprising:
-
a memory storing a plurality of reference images of a body lumen, each reference image including a viewpoint with a depth map and image gradients; a connector adapted for connection to an endoscope for receiving therefrom one or more live video frames associated with a body lumen; and a processor operative to register each video frame received from the endoscope to at least one of the reference images, the processor being programmed to perform the steps of; a) warping the video frame to the viewpoint of one reference image of the plurality of reference images or warping one reference image of the plurality of reference images at a viewpoint to the video frame; b) automatically computing an image difference between (i) the warped video frame and the reference image, or (ii) the warped reference image and the video frame; c) updating a viewpoint of the video frame using an optimization technique; and d) repeating steps a) through c) for each video frame until the viewpoint of the video frame converges on a final viewpoint or a next video frame becomes available. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10)
-
-
11. A system for registering images associated with a continuously guided endoscopic procedure, comprising:
-
a memory storing a plurality of reference images of a body lumen, each reference image including a viewpoint with a depth map and image gradients; an endoscope generating at least one video frame; and a processor operative to register each video frame received from the endoscope to at least one of the reference images, the processor being programmed to perform the steps of; a) warping the video frame to the viewpoint of one reference image of the plurality of reference images or warping one reference image of the plurality of reference images at a viewpoint to the video frame; b) automatically computing an image difference between (i) the warped video frame and the reference image, or (ii) the warped reference image and the video frame; c) updating a viewpoint of the video frame using an optimization technique; and d) repeating steps a) through c) for each video frame until the viewpoint of the video frame converges on a final viewpoint or a next video frame becomes available wherein the final viewpoint gives an estimate of relative rotation and translation between the video frame and the reference image. - View Dependent Claims (12, 13, 14, 15, 16, 17, 18, 19)
-
Specification