Navigation of tubular networks

• US 9,727,963 B2
• Filed: 09/16/2016
• Issued: 08/08/2017
• Est. Priority Date: 09/18/2015
• Status: Active Grant

• Current Assignee: Auris Health, Inc. (Johnson & Johnson)
• Original Assignee: Auris Surgical Robotics, Inc. (Johnson & Johnson)
• Inventors: Stewart, Charles V., Ghoreyshi, Atiyeh, Jeevan, Prasanth, Mintz, David S., Xu, Yiliang, Yang, Gehua, Leotta, Matthew Joseph
• Primary Examiner(s): PONTIUS, JAMES M
• Application Number:
  US15/268,238
• Publication Number:
  US 20170084027A1
• Time in Patent Office: 326 Days
• CPC Class Codes: A61B 1/000094 extracting biological structures
• CPC Class Codes: A61B 1/00147 Holding or positioning arrangements
• CPC Class Codes: A61B 1/00149 using articulated arms
• CPC Class Codes: A61B 1/0016 using motor drive units
• CPC Class Codes: A61B 1/00194 adapted for three-dimensional imaging
• CPC Class Codes: A61B 1/2676 Bronchoscopes
• CPC Class Codes: A61B 2017/00809 Lung operations
• CPC Class Codes: A61B 2034/105 Modelling of the patient, e.g. for ligaments or bones
• CPC Class Codes: A61B 2034/107 Visualisation of planned trajectories or target regions
• CPC Class Codes: A61B 2034/2048 using an accelerometer or inertia sensor
• CPC Class Codes: A61B 2034/2051 Electromagnetic tracking systems
• CPC Class Codes: A61B 2034/2055 Optical tracking systems
• CPC Class Codes: A61B 2034/252 indicating steps of a surgical procedure
• CPC Class Codes: A61B 2034/301 for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes
• CPC Class Codes: A61B 2090/3614 using optical fibre
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/25 User interfaces for surgical systems
• CPC Class Codes: A61B 5/061 Determining position of a probe within the body employing means separate from the probe, e.g. sensing internal probe position employing impedance electrodes on the surface of the body A61B1/00158 takes precedence
• CPC Class Codes: A61B 6/032 Transmission computed tomography [CT]
• CPC Class Codes: A61B 90/30 Devices for illuminating a surgical field, the devices having an interrelation with other surgical devices or with a surgical procedure
• CPC Class Codes: G06T 15/205 : Image-based rendering
• CPC Class Codes: G06T 17/00 : Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/10068 : Endoscopic image
• CPC Class Codes: G06T 2207/10081 : Computed x-ray tomography [CT]
• CPC Class Codes: G06T 2207/30061 : Lung
• CPC Class Codes: G06T 2207/30196 : Human being; Person
• CPC Class Codes: G06T 7/0012 : Biomedical image inspection
• CPC Class Codes: G06T 7/149 : involving deformable models, e.g. active contour models
• CPC Class Codes: G06T 7/248 : involving reference images or patches
• CPC Class Codes: G06T 7/32 : using correlation-based methods
• CPC Class Codes: G06T 7/73 : using feature-based methods

Abstract

Methods and apparatuses provide improved navigation through tubular networks such as lung airways by providing improved estimation of location and orientation information of a medical instrument (e.g., an endoscope) within the tubular network. Various input data such as image data, EM data, and robot data are used by different algorithms to estimate the state of the medical instrument, and the state information is used to locate a specific site within a tubular network and/or to determine navigation information for what positions/orientations the medical instrument should travel through to arrive at the specific site. Probability distributions together with confidence values are generated corresponding to different algorithms are used to determine the medical instrument'"'"'s estimated state.

Claims

1. A method comprising:

accessing robotic data regarding physical manipulation of an elongated medical instrument inserted into a tubular network of a patient;

accessing image data captured by an imaging device located proximal to an instrument tip of the elongated medical instrument;

accessing electromagnetic (EM) data captured using an EM sensor located proximal to the instrument tip as well as using at least on external EM sensor or EM generated located external to the patient;

determining a robot-based estimated state for the instrument tip based on the robotic data, the robot-based estimated state associated with a first confidence value;

determining a image-based estimated state for the instrument tip based on the image data, the image-based estimated state associated with a second confidence value;

determining a EM-based estimated state for the instrument tip based on the EM data, the EM-based estimated state associated with a third confidence value; and

determining an estimated state for the instrument tip based on the robot-based, image-based, and EM-based estimated states and confidence values.

2. The method of claim 1, wherein the elongated medical instrument is a flexible endoscope comprising an endolumenal structure and a plurality of cables that may be variously retracted or related to change a direction of the instrument tip, the endoscope coupled to an instrument device manipulator (IDM) to variously insert, withdraw, and roll the endoscope from or within the tubular network.

3. The method of claim 1, wherein the robot data includes at least one of:

pitch, roll, and yaw of the instrument tip, the pitch, the roll, and the yaw based on retraction of at least one of a plurality of cables of the elongated medical instrument; and

insertion distance of the elongated medical instrument into the tubular network.

4. The method of claim 1, wherein the image data is based on a series of images captured sequentially in time, each of the series of images associated with a timestamp.

5. The method of claim 1, wherein the EM sensor comprises a coil of conductive material.

6. The method of claim 1, wherein the estimated state indicates location of the instrument tip within the tubular network, including at least one of:

position in three-dimensional (3D) space, orientation in 3D space, absolute depth of the instrument tip within the tubular network, relative depth of the instrument tip within a branch of the tubular network, and branch position with respect to a corresponding 3D model of the tubular network.

7. The method of claim 6, wherein the estimated state for the instrument tip is a combined estimated state based on the robot-based, EM-based, and image-based estimated states, and wherein the combined estimated state is generated by, for each of the robot-based, EM-based, and image-based estimated states:

representing the corresponding estimated state as a probability distribution including a set of possible values; and

weighting the corresponding estimated state represented by the set of possible values using the corresponding confidence value applied to each value of the set of possible values.

8. The method of claim 7, wherein the confidence value for a corresponding one of the estimated states represents a degree of confidence in the determination of the corresponding estimated state, and wherein the confidence value is based on at least one of the following factors:

a process used to generate the corresponding estimated state,

received data used to generate the corresponding estimated state, and

a possible area where the instrument tip is located within the tubular network.

9. The method of claim 1, further comprising:

accessing a prior estimated state for the instrument tip based on data captured at a prior instant in time; and

determining at least one of the robot-based, image-based, and EM-based estimated states based on the prior estimated state.

10. The method of claim 1, wherein determining the EM-based estimated state further comprises:

performing registration of an EM system to a 3D model, the EM system including coordinates based on received EM data and the 3D model including coordinates based on a 3D model of the tubular network; and

generating the EM-based estimated state based on the performed registration.

11. The method of claim 10, wherein the 3D model of the tubular network is generated based on computerized axial tomography (CT) scans of the tubular network.

12. The method of claim 1, wherein determining the EM-based estimated state further comprises:

determining an estimate of position and orientation of the instrument tip relative to possible branches within the tubular network.

13. The method of claim 12, wherein the EM-based estimated state is represented as a discrete probability distribution, each of values of the probability distribution indicating a likelihood of the instrument tip being inside a corresponding branch among the possible branches.

14. The method of claim 1, wherein determining the image-based estimated state further comprises:

measuring movement of the elongated medical instrument within the tubular network based on the accessed image data; and

generating the image-based estimated state based on the measured movement.

15. The method of claim 14, wherein the image-based estimated state is represented as a continuous probability distribution of at least one of:

absolute depth of the instrument tip within the tubular network,

depth of the instrument tip relative to a corresponding branch, and

roll of the instrument tip relative to the tubular network.

16. The method of claim 1, wherein the object detection further comprises:

detecting at least one object; and

generating the image-based estimated based on the detected at least one object.

17. The method of claim 16, wherein the at least one object is represented as a two-dimensional shape.

18. The method of claim 1, wherein determining the image-based estimated state further comprises:

mapping between a physically detected object in an actual image and a virtual object generated by a 3D model of the tubular network, the virtual object corresponding to the physically detected object; and

generating the image-based estimated state based on the mapped information.

19. The method of claim 1, wherein determining the image-based estimated state further comprises:

determining a topology of an area where the instrument tip is located within the tubular network; and

generating image-based estimated state based on the determined topology.

20. The method of claim 19, wherein the determined topology is a division into multiple branches faced by the instrument tip.

21. The method of claim 1, further comprises:

accessing a navigation path to a target location in the tubular network;

determining a navigation instruction to direct the instrument tip towards the target location based on the estimated state; and

providing the navigation instruction for presentation to an operator.

22. A non-transitory computer readable storage medium comprising computer program instructions that when executed by a processor cause the processor to perform steps comprising:

accessing robotic data regarding physical manipulation of an elongated medical instrument inserted into a tubular network of a patient;

accessing image data captured by an imaging device located proximal to an instrument tip of the elongated medical instrument;

accessing electromagnetic (EM) data captured using an EM sensor located proximal to the instrument tip as well as using at least on external EM sensor or EM generated located external to the patient;

determining a robot-based estimated state for the instrument tip based on the robotic data, the robot-based estimated state associated with a first confidence value;

determining an image-based estimated state for the instrument tip based on the image data, the image-based estimated state associated with a second confidence value;

determining an EM-based estimated state for the instrument tip based on the EM data, the EM-based estimated state associated with a third confidence value; and

determining an estimated state for the instrument tip based on the robot-based, image-based, and EM-based estimated states and confidence values.