Fast 3D-2D image registration method with application to continuously guided endoscopy

• US 7,889,905 B2
• Filed: 05/19/2006
• Issued: 02/15/2011
• Est. Priority Date: 05/23/2005
• Status: Active Grant

• Current Assignee: Penn State Research Foundation
• Original Assignee: Penn State Research Foundation
• Inventors: Higgins, William E., Rai, Lav, Merritt, Scott A.
• Primary Examiner(s): ZARKA, DAVID PETER
• Application Number:
  US11/437,230
• Publication Number:
  US 20070013710A1
• Time in Patent Office: 1,733 Days
• Field of Search: 382/130
• US Class Current: 382/130
• CPC Class Codes: A61B 1/00147 Holding or positioning arrangements
• CPC Class Codes: A61B 1/2676 Bronchoscopes
• CPC Class Codes: A61B 5/415 the glands, e.g. tonsils, adenoids or thymus
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10068 Endoscopic image
• CPC Class Codes: G06T 2207/10081 Computed x-ray tomography [CT]
• CPC Class Codes: G06T 2207/30061 Lung
• CPC Class Codes: G06T 3/153 using elastic snapping
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06V 10/754 involving a deformation of the sample pattern or of the reference pattern; Elastic matching
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces

Abstract

A novel framework for fast and continuous registration between two imaging modalities is disclosed. The approach makes it possible to completely determine the rigid transformation between multiple sources at real-time or near real-time frame-rates in order to localize the cameras and register the two sources. A disclosed example includes computing or capturing a set of reference images within a known environment, complete with corresponding depth maps and image gradients. The collection of these images and depth maps constitutes the reference source. The second source is a real-time or near-real time source which may include a live video feed. Given one frame from this video feed, and starting from an initial guess of viewpoint, the real-time video frame is warped to the nearest viewing site of the reference source. An image difference is computed between the warped video frame and the reference image. The viewpoint is updated via a Gauss-Newton parameter update and certain of the steps are repeated for each frame until the viewpoint converges or the next video frame becomes available. The final viewpoint gives an estimate of the relative rotation and translation between the camera at that particular video frame and the reference source. The invention has far-reaching applications, particularly in the field of assisted endoscopy, including bronchoscopy and colonoscopy. Other applications include aerial and ground-based navigation.

Claims

1. A method of registering video frames of a body lumen received from an endoscope inserted into the body lumen to previously acquired images of the body lumen, the method comprising the steps of:

computing a plurality of reference images of a body lumen, each reference image including a viewpoint with a depth map and image gradients;

receiving video frames from the endoscope wherein said receiving step comprises receiving a live bronchoscopic video frame associated with an airway; and

providing a processor to register each video frame received from the endoscope to at least one of the reference images, said processor being programmed to perform the steps of;

a) warping the video frame to the viewpoint of one reference image of the plurality of reference images or warping one reference image of the plurality of reference images at a viewpoint to the video frame;

b) automatically computing an image difference between (i) the warped video frame and the reference image, or (ii) the warped reference image and the video frame;

c) updating a viewpoint of the video frame using an optimization technique; and

d) repeating steps a) through c) for each video frame until the viewpoint of the video frame converges on a final viewpoint or a next video frame becomes available.

2. The method of claim 1, wherein at least one reference image is updated before, during, or after registration.

3. The method of claim 1, including the step of guessing an initial viewpoint of the video frame.

4. The method of claim 1, wherein the final viewpoint gives an estimate of the relative rotation and translation between the video frame of the video feed and the reference image.

5. The method of claim 1, further including the step of using a resolution pyramid wherein all images, depth maps, and gradients are down-sampled.

6. The method of claim 1, further including the step of using a weighted normalized cross-correlation objective function.

7. The method of claim 1, wherein the optimization technique is based on a Gauss-Newton parameter update.

8. The method of claim 1 wherein the computed set of reference images of a body lumen comprises a 3-D model of a bronchial tree.

9. The method of claim 1 wherein the body lumen is an airway.

10. A method of registering video frames of a body lumen received from an endoscope inserted into the body lumen to previously acquired images of the body lumen, the method comprising the steps of:

computing a plurality of reference images of a body lumen, each reference image including a viewpoint with a depth map and image gradients;

receiving at least one video frame from the endoscope; and

providing a processor to register each video frame received from the endoscope to at least one of the reference images using of the plurality, said processor being programmed to perform the steps of;

a) warping the video frame to the viewpoint of one reference image of the plurality of reference images or warping one reference image of the plurality of reference images at a viewpoint to the video frame;

b) automatically computing an image difference between (i) the warped video frame and the reference image, or (ii) the warped reference image and the video frame;

c) updating a viewpoint of the video frame using an optimization technique; and

d) repeating steps a) through c) for each video frame until the viewpoint of the video frame converges on a final viewpoint or a next video frame becomes available wherein the final viewpoint gives an estimate of relative rotation and translation between the video frame and the reference image.

11. The method of claim 10, wherein at least one reference image is updated before, during, or after registration.

12. The method of claim 10, including the step of guessing an initial viewpoint of the video frame.

13. The method of claim 10, further including the step of using a resolution pyramid wherein all images, depth maps, and gradients are down-sampled.

14. The method of claim 10, further including the step of using a weighted normalized cross-correlation objective function.

15. The method of claim 10 wherein the optimization technique is based on a Gauss-Newton parameter update.

16. The method of claim 10 wherein the step of receiving at least one video frame from an endoscope comprises receiving a live bronchoscopic video frame.

17. The method of claim 16 wherein the computed set of reference images of the body lumen comprises a 3-D model of a bronchial tree.