Systems and methods for planning and navigation

• US 9,498,182 B2
• Filed: 05/22/2012
• Issued: 11/22/2016
• Est. Priority Date: 05/22/2012
• Status: Active Grant

• Current Assignee: Covidien LP (Medtronic Plc)
• Original Assignee: Covidien LP (Medtronic Plc)
• Inventors: Case, Jason A., Frank, Kevin, Ladtkow, Casey M.
• Primary Examiner(s): Park, Patricia
• Application Number:
  US13/477,374
• Publication Number:
  US 20130317352A1
• Time in Patent Office: 1,645 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: A61B 18/1477 Needle-like probes
• CPC Class Codes: A61B 18/1492 having a flexible, catheter-like structure, e.g. for heart ablation A61B18/1477 takes precedence
• CPC Class Codes: A61B 2017/00477 Coupling A61B2017/0046 takes precedence
• CPC Class Codes: A61B 2018/00577 Ablation
• CPC Class Codes: A61B 2018/00595 Cauterization
• CPC Class Codes: A61B 2018/00601 Cutting
• CPC Class Codes: A61B 2018/0063 Sealing
• CPC Class Codes: A61B 2018/00702 Power or energy
• CPC Class Codes: A61B 2018/00761 Duration
• CPC Class Codes: A61B 2018/00982 combined with or comprising means for visual or photographic inspections inside the body, e.g. endoscopes
• CPC Class Codes: A61B 2018/1425 Needle
• CPC Class Codes: A61B 2018/1869 with an instrument interstitially inserted into the body, e.g. needles
• CPC Class Codes: A61B 2034/104 Modelling the effect of the tool, e.g. the effect of an implanted prosthesis or for predicting the effect of ablation or burring
• CPC Class Codes: A61B 2034/107 Visualisation of planned trajectories or target regions
• CPC Class Codes: A61B 2034/2051 Electromagnetic tracking systems
• CPC Class Codes: A61B 2034/2055 Optical tracking systems
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2090/0463 against cooling or freezing
• CPC Class Codes: A61B 2090/0472 against ultrasound energy
• CPC Class Codes: A61B 2090/0481 against EM radiation, e.g. microwave
• CPC Class Codes: A61B 2090/363 : Use of fiducial points
• CPC Class Codes: A61B 2090/365 : augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 2090/378 : using ultrasound
• CPC Class Codes: A61B 2090/3966 : Radiopaque markers visible in an X-ray image
• CPC Class Codes: A61B 2090/3983 : Reference marker arrangements for use with image guided surgery
• CPC Class Codes: A61B 34/10 : Computer-aided planning, simulation or modelling of surgical operations
• CPC Class Codes: A61B 34/20 : Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/25 : User interfaces for surgical systems
• CPC Class Codes: A61B 8/0841 : for locating instruments
• CPC Class Codes: A61B 8/4263 : using sensors not mounted on the probe, e.g. mounted on an external reference frame
• CPC Class Codes: A61B 8/483 : involving the acquisition of a 3D volume of data
• CPC Class Codes: A61B 8/565 : involving data transmission via a network
• CPC Class Codes: A61B 90/04 : Protection of tissue around surgical sites against effects of non-mechanical surgery, e.g. laser surgery protection of eyes against laser radiation A61F9/022
• CPC Class Codes: A61B 90/37 : Surgical systems with images on a monitor during operation
• CPC Class Codes: A61B 90/39 : Markers, e.g. radio-opaque or breast lesions markers
• CPC Class Codes: G06T 2207/10016 : Video; Image sequence
• CPC Class Codes: G06T 2207/10132 : Ultrasound image
• CPC Class Codes: G06T 2207/30021 : Catheter; Guide wire
• CPC Class Codes: G06T 2207/30208 : Marker matrix
• CPC Class Codes: G06T 2207/30244 : Camera pose
• CPC Class Codes: G06T 7/73 : using feature-based methods

Abstract

The present disclosure is directed to a planning and navigation method. The planning and method includes obtaining and rendering a plurality of images. The plurality of images are segmented to demarcate a target area. A treatment plan is determined based on the target area. The navigation method includes obtaining an ultrasound image of a scan plane including the target and obtaining a fiducial image of a fiducial pattern disposed on an ultrasound device. The obtained fiducial image is corrected and a correspondence between the fiducial image and a model image is found. A position of the surgical device is transformed to model coordinates. Then the ultrasound image and a virtual image of the surgical device is displayed to allow a surgeon to navigate the surgical device to the target using the displayed ultrasound image and the virtual image. The target is treated based on the treatment plan.

Claims

1. A method of navigating a surgical device using a fiducial pattern disposed on an ultrasound device and an image capture device disposed on the surgical device, the method comprising:

obtaining an ultrasound image of a scan plane;

obtaining a fiducial image of the fiducial pattern;

correcting the fiducial image for lens distortion;

identifying a plurality of first unique identifiers and a plurality of second unique identifiers in the fiducial image, wherein the plurality of first unique identifiers includes circles of a first color and the plurality of second unique identifiers includes circles of a second color and wherein identifying includes;

applying a first threshold to the fiducial image substantially in parallel with applying a second threshold to an inverted fiducial image;

performing a connected component analysis on the first and second thresholds; and

applying a geometric filter to results of the connected component analysis to identify a region having the plurality of first unique identifiers and to determine weighted centroids of the plurality of first unique identifiers and the plurality of second unique identifiers,

finding correspondence between the fiducial image and a model image using the plurality of first unique identifiers and the plurality of second unique identifiers by;

selecting a plurality of the circles of the first color;

determining an amount of the circles of the first color disposed in a first portion of the region of the fiducial pattern;

determining whether the determined amount of the circles of the first color disposed in the first portion matches a predetermined amount of the circles of the first color; and

in response to a determination that the determined amount of the circles of the first color disposed in the first portion does not match the predetermined amount of the circles of the first color, determining an amount of the circles of the first color disposed in a second portion of the region of the fiducial pattern;

estimating a camera pose;

transforming a position of the surgical device to model coordinates; and

displaying the ultrasound image and a virtual image of the surgical device based on the model coordinates.

2. A method of tracking a first device having an image capture device in relation to a second device having a fiducial pattern, the method comprising:

obtaining a fiducial image of the fiducial pattern;

correcting the fiducial image for lens distortion;

identifying a plurality of first unique identifiers and a plurality of second unique identifiers in the fiducial image, wherein the plurality of first unique identifiers includes circles of a first color and the plurality of second unique identifiers includes circles of a second color and wherein identifying includes;

applying a first threshold to the fiducial image substantially in parallel with applying a second threshold to an inverted fiducial image;

performing a connected component analysis on the first and second thresholds; and

applying a geometric filter to results of the connected component analysis to identify a region having the plurality of first unique identifiers and to determine weighted centroids of the plurality of first unique identifiers and the plurality of second unique identifiers,

finding correspondence between the fiducial image and a model image using the plurality of first unique identifiers and the plurality of second unique identifiers by;

selecting a plurality of the circles of the first color;

determining an amount of the circles of the first color disposed in a first portion of the region of the fiducial pattern;

determining whether the determined amount of the circles of the first color disposed in the first portion matches a predetermined amount of the circles of the first color; and

in response to a determination that the determined amount of the circles of the first color disposed in the first portion does not match the predetermined amount of the circles of the first color, determining an amount of the circles of the first color disposed in a second portion of the region of the fiducial pattern;

estimating a camera pose; and

transforming a position of the surgical device to model coordinates.

3. The method of claim 2, wherein the fiducial pattern includes the plurality of first unique identifiers disposed in the region and the plurality of second unique identifiers.

4. The method of claim 2, wherein finding correspondence between the fiducial image and the model image further comprises:

selecting the plurality of first unique identifiers from the fiducial image;

arranging the plurality of first unique identifiers in clockwise order;

arranging a plurality of model fiducials in clockwise order;

computing a planar homography;

transforming the plurality of model fiducials into image coordinates using the computed planar homography;

finding a model fiducial from the plurality of model fiducials that matches the fiducial image; and

computing the residual error.

5. The method of claim 4, wherein selecting the plurality of first unique identifiers from the fiducial image, comprises:

selecting the plurality of first unique identifiers;

selecting the region having the plurality of first unique identifiers;

counting the number of first unique identifiers in the selected region; and

comparing the number of first unique identifiers in the selected region to a predetermined number,

wherein if the number of first unique identifiers in the selected region equals the predetermined number, the method proceeds to arranging the plurality of first unique identifiers in clockwise order, and

wherein if the number of first unique identifiers in the selected region does not equal the predetermined number, a new region is selected and the number of first unique identifiers in the new region is counted.

6. The method of claim 2, wherein the first threshold and/or the second threshold are dynamic thresholds.

7. A planning and navigation method, comprising:

obtaining a plurality of images;

rendering the plurality of images in three dimensions;

automatically segmenting the plurality of images to demarcate a target area;

automatically determining a treatment plan based on the target area;

obtaining an ultrasound image of a scan plane including the target;

obtaining a fiducial image of a fiducial pattern disposed on an ultrasound device using an image capture device on a surgical device;

correcting the fiducial image for lens distortion;

identifying a plurality of first unique identifiers and a plurality of second unique identifiers in the fiducial image, wherein the plurality of first unique identifiers includes circles of a first color and the plurality of second unique identifiers includes circles of a second color and wherein identifying includes;

applying a first threshold to the fiducial image substantially in parallel with applying a second threshold to an inverted fiducial image;

performing a connected component analysis on the first and second thresholds; and

applying a geometric filter to results of the connected component analysis to identify a region having the plurality of first unique identifiers and to determine weighted centroids of the plurality of first unique identifiers and the plurality of second unique identifiers,

finding correspondence between the fiducial image and a model image using the plurality of first unique identifiers and the plurality of second unique identifiers by;

selecting a plurality of the circles of the first color;

determining an amount of the circles of the first color disposed in a first portion of the region of the fiducial pattern;

determining whether the determined amount of the circles of the first color disposed in the first portion matches a predetermined amount of the circles of the first color; and

in response to a determination that the determined amount of the circles of the first color disposed in the first portion does not match the predetermined amount of the circles of the first color, determining an amount of the circles of the first color disposed in a second portion of the region of the fiducial pattern;

estimating a camera pose;

transforming a position of the surgical device to model coordinates;

displaying the ultrasound image and a virtual image of the surgical device based on the model coordinates;

navigating the surgical device to the target using the displayed ultrasound image and the virtual image; and

treating the target based on the treatment plan.

8. The method of claim 7, wherein automatically segmenting the plurality of images further comprises:

selecting a seed point;

creating a region of interest around the seed point;

comparing a first plurality of pixels in the region of interest to a predetermined threshold;

selecting a second plurality of pixels from the first plurality of pixels, wherein the second plurality of pixels are connected to the seed point and are less than the predetermined threshold; and

applying a geometric filter to the second plurality of pixels.

9. The method of claim 8, further comprising:

determining if the second plurality of pixels forms a predetermined object,

wherein if the second plurality of pixels does not form a predetermined object, the predetermined threshold is increased and the steps of comparing a first plurality of pixels, selecting a second plurality of pixels, applying a geometric filter, and determining if the second plurality of pixels forms the predetermined object are repeated.

10. The method of claim 7, wherein automatically determining a treatment plan further comprises:

performing a volumetric analysis on the target area;

calculating an energy level and treatment duration based on the target area and the surgical device.

11. The method of claim 7, further comprising:

displaying the rendered plurality of images;

displaying the target area; and

displaying the treatment plan.

12. The method of claim 7, further comprising:

automatically segmenting at least one vessel;

adjusting the treatment plan based on a proximity of the at least one vessel to the target; and

displaying the adjusted treatment plan.

13. The method of claim 7, further comprising:

automatically segmenting at least one organ;

adjusting the treatment plan based on a location of the target in relation to the at least one organ; and

displaying the adjusted treatment plan.

14. The method of claim 7 wherein the fiducial pattern includes a plurality of first unique identifiers disposed in a region and a plurality of second unique identifiers.

15. The method of claim 7, wherein finding correspondence between the fiducial image and the model image comprises:

selecting the plurality of first unique identifiers from the fiducial image;

arranging the plurality of first unique identifiers in clockwise order;

arranging a plurality of model fiducials in clockwise order;

computing a planar homography;

transforming the plurality of model fiducials into image coordinates using the computed planar homography;

finding a model fiducial from the plurality of model fiducials that matches the fiducial image; and

computing the residual error.

16. The method of claim 15, wherein selecting the plurality of first unique identifiers from the fiducial image, comprises:

selecting the plurality of first unique identifiers;

selecting the region having the plurality of first unique identifiers;

counting the number of first unique identifiers in the selected region; and

comparing the number of first unique identifiers in the selected region to a predetermined number,

wherein if the number of first unique identifiers in the selected region equals the predetermined number, the method proceeds to arranging the plurality of first unique identifiers in clockwise order, and

wherein if the number of first unique identifiers in the selected region does not equal the predetermined number, a new region is selected and the number of first unique identifiers in the new region is counted.

17. The method of claim 7, wherein the first threshold and/or the second threshold are dynamic thresholds.