System and method of generating and updatng a three dimensional model of a luminal network

• US 10,478,143 B2
• Filed: 08/02/2016
• Issued: 11/19/2019
• Est. Priority Date: 08/02/2016
• Status: Active Grant

• Current Assignee: Covidien LP (Medtronic Plc)
• Original Assignee: Covidien LP (Medtronic Plc)
• Inventors: Merlet, Nicolas J., Weingarten, Oren P.
• Primary Examiner(s): Pehlke, Carolyn A
• Application Number:
  US15/226,111
• Publication Number:
  US 20180035966A1
• Time in Patent Office: 1,204 Days
• CPC Class Codes: A61B 1/2676 Bronchoscopes
• CPC Class Codes: A61B 2017/00809 Lung operations
• CPC Class Codes: A61B 2034/107 Visualisation of planned trajectories or target regions
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2034/2068 using pointers, e.g. pointers having reference marks for determining coordinates of body points
• CPC Class Codes: A61B 2090/364 Correlation of different images or relation of image positions in respect to the body
• CPC Class Codes: A61B 2090/3966 Radiopaque markers visible in an X-ray image
• CPC Class Codes: A61B 2090/3983 Reference marker arrangements for use with image guided surgery
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 5/0035 adapted for acquisition of images from more than one imaging mode, e.g. combining MRI and optical tomography
• CPC Class Codes: A61B 5/055 involving electronic [EMR] or nuclear [NMR] magnetic resonance, e.g. magnetic resonance imaging
• CPC Class Codes: A61B 5/062 using magnetic field
• CPC Class Codes: A61B 6/032 Transmission computed tomography [CT]
• CPC Class Codes: A61B 6/12 Arrangements for detecting or locating foreign bodies
• CPC Class Codes: A61B 6/4417 related to combined acquisition of different diagnostic modalities
• CPC Class Codes: A61B 6/4441 the rigid structure being a C-arm or U-arm
• CPC Class Codes: A61B 6/463 characterised by displaying multiple images or images and diagnostic data on one display
• CPC Class Codes: A61B 6/466 adapted to display 3D data
• CPC Class Codes: A61B 6/487 involving fluoroscopy
• CPC Class Codes: A61B 6/5205 involving processing of raw data to produce diagnostic data
• CPC Class Codes: A61B 6/5229 : combining image data of a patient, e.g. combining a functional image with an anatomical image
• CPC Class Codes: A61B 6/5235 : combining images from the same or different ionising radiation imaging techniques, e.g. PET and CT
• CPC Class Codes: A61B 6/5247 : combining images from an ionising-radiation diagnostic technique and a non-ionising radiation diagnostic technique, e.g. X-ray and ultrasound
• CPC Class Codes: A61B 8/5261 : combining images from different diagnostic modalities, e.g. ultrasound and X-ray
• CPC Class Codes: A61B 90/10 : for stereotaxic surgery, e.g. frame-based stereotaxis
• CPC Class Codes: G06T 2207/10081 : Computed x-ray tomography [CT]
• CPC Class Codes: G06T 2207/10121 : Fluoroscopy
• CPC Class Codes: G06T 2207/30061 : Lung
• CPC Class Codes: G06T 7/33 : using feature-based methods

Abstract

A method of updating a model of a luminal network. The method includes performing first imaging of a target, generating 3D image data including locations of fiducial markers using images captured in the first imaging, performing second imaging of the target from a first viewpoint and a second viewpoint, generating 2D image data including locations of the fiducial markers from the first viewpoint and the second viewpoint using images captured in the second imaging, determining locations of the fiducial markers in 3D space according to locations of the fiducial markers within the 2D image data of the first viewpoint, locations of the fiducial markers within the 2D image data of the second viewpoint, and the known angle, and registering the 3D image data generated using images captured in the first imaging with the determined locations of the fiducial markers in 3D space. The first viewpoint and the second viewpoint are oriented at a known angle relative to each other.

Claims

1. A method of updating a model of a luminal network, comprising:

generating 3D image data including fiducial markers disposed at a first location using images captured in a first imaging of a target;

generating 2D image data including the fiducial markers disposed at a second location using images captured in a second imaging of the target, the second imaging of the target including imaging of the target from a first viewpoint and imaging the target from a second viewpoint, the first and second viewpoints oriented at a known angle relative to each other;

determining the second location of the fiducial markers in 3D space based on the second location of the fiducial markers within the 2D image data of the first viewpoint, the second location of the fiducial markers within the 2D image data of the second viewpoint, and the known angle;

comparing the first location of the fiducial markers in the 3D image data to the second location of the fiducial markers in 3D space;

determining a difference between the first location of the fiducial markers in the 3D image data and the second location of the fiducial markers in 3D space based on the comparison;

updating the 3D image data generated using images captured in the first imaging with the determined second location of the fiducial markers in 3D space based on the determined difference;

displaying the updated 3D image data including the fiducial markers disposed at the second location of the fiducial markers in 3D space;

determining movement distances between the second location of the fiducial markers in 3D space and the first location of the fiducial markers in the 3D image data from the first imaging;

determining which of the movement distances exceeds a threshold; and

registering the 3D image data generated using images captured in the first imaging with the determined second location of the fiducial markers in 3D space, wherein the registering excludes the second location in 3D space of the fiducial marker corresponding to movement distances that exceed the threshold.

2. The method of claim 1, wherein the first imaging is performed using CT imaging.

3. The method of claim 1, wherein the second imaging is performed using fluoroscopic imaging.

4. The method of claim 1, wherein determining the second location of the fiducial markers in 3D space includes:

projecting rays from the first viewpoint through endpoints of each of the fiducial markers,

projecting rays from the second viewpoint through the endpoints of each of the fiducial markers; and

determining intersections of the rays projected from the first viewpoint and the rays projected from the second viewpoint as locations of fiducial marker endpoints in 3D space.

5. The method of claim 4, further comprising:

determining a distance between adjacent fiducial marker endpoints; and

comparing the distance between adjacent fiducial marker endpoints with a known length of at least one of the fiducial markers.

6. A system comprising:

a display configured to display;

3D image data of a target, the 3D image data including fiducial markers disposed at a first location;

first 2D image data of the target imaged from a first viewpoint; and

second 2D image data of the target imaged from a second viewpoint, the second viewpoint oriented at a known angle from the first viewpoint, the first 2D image data and the second 2D image data including the fiducial markers disposed at a second location;

a computing device including a processor and a memory storing instructions which, when executed by the processor, cause the computing device to;

determine the second location of the fiducial markers in 3D space according to the second location of the fiducial markers within the first 2D image data, the second location of the fiducial markers within the second 2D image data, and the known angle;

compare the first location of the fiducial markers in the 3D image data to the second location of the fiducial markers in 3D space;

determine a difference between the first location of the fiducial markers in the 3D image data and the second location of the fiducial markers in 3D space;

update the 3D image data with the determined second location of the fiducial markers in 3D space based on the determined difference;

display the updated 3D image data including the fiducial markers disposed at the second location of the fiducial markers in 3D space via the display;

determine movement distances between the second location of the fiducial markers in 3D space and the first location of the fiducial markers in the 3D image data;

determine which of the movement distances exceeds a threshold; and

register the 3D image data generated using images captured in the first imaging with the determined second location of the fiducial markers in 3D space, wherein the registration excludes the second location in 3D space of the fiducial marker corresponding to movement distances that exceed the threshold.

7. The system of claim 6, wherein the 3D image data is generated using CT imaging.

8. The system of claim 6, wherein the first 2D image data and the second 2D image data are generated using fluoroscopic imaging.

9. The system of claim 6, wherein in the determination of the second location of the fiducial markers in 3D space, the instructions further cause the computing device to:

project rays from the first viewpoint through endpoints of each of the fiducial markers;

project rays from the second viewpoint through the endpoints of each of the fiducial markers; and

determine intersections of the rays projected from the first viewpoint and the rays projected from the second viewpoint as locations of fiducial marker endpoints in 3D space.

10. The system of claim 9, wherein the instructions further cause the computing device to:

determine a distance between adjacent fiducial marker endpoints, and

compare the distance between adjacent fiducial marker endpoints with a known length of at least one of the fiducial markers.

11. A method of updating a 3D model of a luminal network, comprising:

generating a 3D model of a luminal network of a patient based on CT imaging of a target and at least one fiducial marker disposed at a first location relative to the target;

generating 2D image data based on fluoroscopic imaging of the target and the at least one fiducial marker from a first viewpoint and fluoroscopic imaging of the target and the at least one fiducial marker from a second viewpoint orientated at a known angle relative to the first viewpoint;

determining a second location of the at least one fiducial marker in 3D space based on a second location of the at least one fiducial marker within the 2D image data;

determining a difference between the first location of the at least one fiducial marker in the CT imaging of the luminal network and the second location of the at least one fiducial marker in 3D space;

updating the 3D model of the luminal network based on the determined difference between the first location of the at least one fiducial marker in the CT imaging of the luminal network and the second location of the at least one fiducial marker in 3D space;

displaying the updated 3D model of the luminal network including the at least one fiducial marker disposed at the second location of the at least one fiducial marker in 3D space;

determining movement distances between the second location of the fiducial marker in 3D space and the first location of the fiducial marker in the CT imaging;

determining which of the movement distances exceed a threshold; and

registering the 3D model generated using images captured in the CT imaging with the determined second location of the fiducial markers in 3D space, wherein the registering excludes the second location in 3D space of the fiducial marker corresponding to movement distances that exceed the threshold.