Fast 3D-2D image registration method with application to continuously guided endoscopy

• US 20070013710A1
• Filed: 05/19/2006
• Published: 01/18/2007
• Est. Priority Date: 05/23/2005
• Status: Active Grant

• Current Assignee: Penn State Research Foundation
• Original Assignee: Penn State Research Foundation
• Inventors: Rai, Lav, Higgins, William, Merritt, Scott
• Granted Patent: US 7,889,905 B2
• Time in Patent Office: Days
• US Class Current: 345/581
• CPC Class Codes: A61B 1/00147 Holding or positioning arrangements
• CPC Class Codes: A61B 1/2676 Bronchoscopes
• CPC Class Codes: A61B 5/415 the glands, e.g. tonsils, adenoids or thymus
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10068 Endoscopic image
• CPC Class Codes: G06T 2207/10081 Computed x-ray tomography [CT]
• CPC Class Codes: G06T 2207/30061 Lung
• CPC Class Codes: G06T 3/153 using elastic snapping
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06V 10/754 involving a deformation of the sample pattern or of the reference pattern; Elastic matching
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces

Abstract

A novel framework for fast and continuous registration between two imaging modalities is disclosed. The approach makes it possible to completely determine the rigid transformation between multiple sources at real-time or near real-time frame-rates in order to localize the cameras and register the two sources. A disclosed example includes computing or capturing a set of reference images within a known environment, complete with corresponding depth maps and image gradients. The collection of these images and depth maps constitutes the reference source. The second source is a real-time or near-real time source which may include a live video feed. Given one frame from this video feed, and starting from an initial guess of viewpoint, the real-time video frame is warped to the nearest viewing site of the reference source. An image difference is computed between the warped video frame and the reference image. The viewpoint is updated via a Gauss-Newton parameter update and certain of the steps are repeated for each frame until the viewpoint converges or the next video frame becomes available. The final viewpoint gives an estimate of the relative rotation and translation between the camera at that particular video frame and the reference source. The invention has far-reaching applications, particularly in the field of assisted endoscopy, including bronchoscopy and colonoscopy. Other applications include aerial and ground-based navigation.

Claims

1. A method of registering an image, comprising the steps of:

providing a set of one or more reference images with depth maps;

registering the image to at least one of the reference images of the set using the depth map for that reference image.

2. The method of claim 1, wherein:

the image is real and the set of reference images are real;

the image is real and the set of reference images are virtual;

the image is virtual and the set of reference images are real;

or

the image is virtual and the set of reference images are virtual.

3. The method of claim 1, wherein the image or the set of reference images are endoscopic.

4. The method of claim 1, wherein the registration occurs in real-time or near real-time.

5. The method of claim 1, wherein at least one of the images in the set of reference images further includes an image gradient.

6. The method of claim 1, wherein the images are cross-correlated.

7. The method of claim 1, wherein the image being registered is a video frame.

8. The method of claim 1, wherein the image being registered is warped prior to registration.

9. The method of claim 1, wherein the registration includes the step of computing an image difference.

10. The method of claim 1, wherein the registration is updated using a Gauss-Newton method.

11. The method of claim 1, wherein the set of reference images can be updated before, during, or after registration.

12. The method of claim 1, wherein:

the set of reference images represents viewpoints with depth maps and image gradients;

the image to be registered is derived from a video feed having a plurality of consecutive frames; and

the method includes the steps of;

a) warping a frame of the video to the nearest viewpoint of the reference source;

b) computing an image difference between the warped video frame and the reference image;

c) updating the viewpoint using a Gauss-Newton parameter update; and

d) repeating steps a) through c) for each frame until the viewpoint converges or the next video frame becomes available.

13. The method of claim 12, wherein one or more of the images in the set of reference images can be updated before, during, or after registration.

14. The method of claim 12, wherein the video is real-time or near real-time.

15. The method of claim 12, including the step of guessing an initial viewpoint.

16. The method of claim 12, wherein the final viewpoint gives an estimate of the relative rotation and translation between the current frame of the video feed and the reference source.

17. The method of claim 12, further including the step of using a resolution pyramid wherein all images, depth maps, and gradients are down-sampled.

18. The method of claim 12, further including the step of using a weighted normalized cross-correlation objective function that allows images of different mean intensities and intensity ranges to be compared while allowing weighting of individual pixel values.