Image-based branch detection and mapping for navigation

• US 10,555,778 B2
• Filed: 10/13/2017
• Issued: 02/11/2020
• Est. Priority Date: 10/13/2017
• Status: Active Grant

• Current Assignee: Auris Health, Inc. (Johnson & Johnson)
• Original Assignee: Auris Health, Inc. (Johnson & Johnson)
• Inventors: Ummalaneni, Ritwik
• Primary Examiner(s): Newton, Alexandra L
• Application Number:
  US15/783,903
• Publication Number:
  US 20190110843A1
• Time in Patent Office: 851 Days
• Field of Search: 600114-118
• CPC Class Codes: A61B 1/00 Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes examination of body cavities or body tracts using ultrasonic, sonic or infrasonic waves A61B8/12; endoscopic instruments for taking cell samples or for biopsy A61B10/04; for surgical purposes A61B17/00; endoscopic surgical cutting instruments A61B17/32; surgical instruments using a laser beam being directed along or through a flexible conduit A61B18/22; Illuminating arrangements therefor for the eyes A61B3/00
• CPC Class Codes: A61B 1/000094 extracting biological structures
• CPC Class Codes: A61B 1/0002 provided with data storages
• CPC Class Codes: A61B 1/00149 using articulated arms
• CPC Class Codes: A61B 1/05 characterised by the image sensor, e.g. camera, being in the distal end portion
• CPC Class Codes: A61B 1/2676 Bronchoscopes
• CPC Class Codes: A61B 1/2736 Gastroscopes
• CPC Class Codes: A61B 1/307 for the urinary organs, e.g. urethroscopes, cystoscopes
• CPC Class Codes: A61B 1/3132 for laparoscopy
• CPC Class Codes: A61B 10/0233 Pointed or sharp biopsy instruments
• CPC Class Codes: A61B 2017/00053 Mapping
• CPC Class Codes: A61B 2017/00477 Coupling A61B2017/0046 takes precedence
• CPC Class Codes: A61B 2017/00809 Lung operations
• CPC Class Codes: A61B 2034/105 Modelling of the patient, e.g. for ligaments or bones
• CPC Class Codes: A61B 2034/2051 Electromagnetic tracking systems
• CPC Class Codes: A61B 2034/2059 Mechanical position encoders
• CPC Class Codes: A61B 2034/2061 using shape-sensors, e.g. fiber shape sensors with Bragg gratings
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2034/301 for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes
• CPC Class Codes: A61B 2090/306 using optical fibres
• CPC Class Codes: A61B 2090/309 : using white LEDs
• CPC Class Codes: A61B 2090/3614 : using optical fibre
• CPC Class Codes: A61B 2090/376 : using X-rays, e.g. fluoroscopy
• CPC Class Codes: A61B 2217/005 : with suction drainage system
• CPC Class Codes: A61B 2217/007 : with irrigation system
• CPC Class Codes: A61B 34/20 : Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/30 : Surgical robots
• CPC Class Codes: A61B 46/10 : specially adapted for instruments, e.g. microscopes
• CPC Class Codes: A61B 50/13 : Trolleys, e.g. carts
• CPC Class Codes: A61G 13/04 : tiltable around transverse or longitudinal axis
• CPC Class Codes: A61G 13/08 : the table being divided into different adjustable sections
• CPC Class Codes: A61G 13/10 : Parts, details or accessories surgical drape sheets A61B46/00
• CPC Class Codes: A61G 13/12 : Rests specially adapted therefor; Arrangements of patient-supporting surfaces rests for the surgeon A61B90/60; restraining devices A61F5/37
• CPC Class Codes: A61G 2210/50 : for radiography
• CPC Class Codes: G06T 2207/10068 : Endoscopic image
• CPC Class Codes: G06T 2207/20076 : Probabilistic image processing
• CPC Class Codes: G06T 2207/30061 : Lung
• CPC Class Codes: G06T 2207/30084 : Kidney; Renal
• CPC Class Codes: G06T 2207/30092 : Stomach; Gastric
• CPC Class Codes: G06T 2207/30244 : Camera pose
• CPC Class Codes: G06T 7/11 : Region-based segmentation
• CPC Class Codes: G06T 7/136 : involving thresholding
• CPC Class Codes: G06T 7/73 : using feature-based methods
• CPC Class Codes: G06T 7/74 : involving reference images or patches

Abstract

Navigation of an instrument within a luminal network can include image-based branch detection and mapping. Image-based branch detection can include identifying within an image one or more openings associated with one or more branches of a luminal network. Image-based branch mapping can include mapping the detected one or more openings to corresponding branches of the luminal network. Mapping may include comparing features of the openings to features of a set of expected openings. A position state estimate for the instrument can be determined from the mapped openings, which can facilitate navigation of the luminal network.

Claims

1. A non-transitory computer readable storage medium having stored thereon instructions that, when executed, cause a processor of a device to at least:

determine a position state estimate of an instrument positioned within a current branch of a luminal network, the instrument having an elongate body configured to be inserted into the luminal network and an imaging device positioned on a distal portion of the elongate body, the instrument attached to an instrument positioning device configured to move the instrument through the luminal network;

determine a set of expected subsequent branches based at least in part on the position state estimate and a preoperative model of the luminal network;

capture an image of the current branch with the imaging device positioned on the instrument;

detect within the image a plurality of openings connecting subsequent branches of the luminal network to the current branch;

determine an estimate of a roll of the instrument based on comparing an orientation of the detected openings within the image to an orientation of the set of expected subsequent branches;

determine a first feature of the detected plurality of openings;

determine a second feature of the set of expected subsequent branches;

calculate a roll correction value based on the estimate of the roll of the instrument;

correct the second feature based on the roll correction value;

compare the first feature to the corrected second feature to determine a mapping of each of the plurality of openings to one of expected subsequent branches; and

based at least in part on the mapping, provide an updated position state estimate.

2. The non-transitory computer readable storage medium of claim 1, wherein the updated position state estimate comprises a probability that the position state estimate is correct.

3. The non-transitory computer readable storage medium of claim 2, wherein the probability is determined based in part on the comparison between the first feature to the corrected second feature.

4. The non-transitory computer readable storage medium of claim 1, wherein the updated position state estimate comprises an estimate of which subsequent branch the instrument will be moved into.

5. The non-transitory computer readable storage medium of claim 1, wherein:

the instructions, when executed, cause the processor of the device to determine which opening of the plurality of detected openings is closer to a center of the image,

the updated position state estimate comprises a probability that the instrument will be moved into the opening that is determined to be closer to the center of the image.

6. The non-transitory computer readable storage medium of claim 1, wherein the instructions, when executed, cause the processor of the device to determine the first feature of the detected openings selected from the group consisting of:

a centroid of an opening, a profile of an opening, and a vector connecting centroids of two openings.

7. The non-transitory computer readable storage medium of claim 6, wherein the instructions, when executed, cause the processor of the device to determine the second feature of the set of expected subsequent branches from the preoperative model, wherein the information comprises at least one of centroid of an opening, a profile of an opening, and a vector connecting centroids of two openings.

8. The non-transitory computer readable storage medium of claim 7, wherein the instructions, when executed, cause the processor of the device to compare one or more features of the detected plurality of openings to the set of expected subsequent branches by:

for each of the detected openings, iteratively matching the first feature of the detected opening to the corrected second feature of the set of expected subsequent branches,

wherein the highest match is used to map the detected opening to the one of the expected subsequent branches.

9. A robotic system for navigating a luminal network of a patient, the robotic system comprising:

an instrument having

an elongate body configured to be inserted into the luminal network, and

an imaging device positioned on a distal portion of the elongate body;

an instrument positioning device attached to the instrument, the instrument positioning device configured to move the instrument through the luminal network;

at least one non-transitory computer-readable memory having stored thereon executable instructions; and

one or more processors in communication with the at least one non-transitory computer-readable memory and configured to execute the instructions to cause the system to at least;

determine a position state estimate of the instrument positioned within a current branch of the luminal network;

determine a set of expected subsequent branches based at least in part on the initial state estimate and a preoperative model of the luminal network;

capture an image of the current branch of the luminal network with the imaging device positioned on the instrument;

detect within the image a plurality of openings connecting subsequent branches of the luminal network to the current branch;

determine an estimate of a roll of the instrument based on comparing an orientation of the detected openings within the image to an orientation of the set of expected subsequent branches;

determine a first feature of the detected plurality of openings;

determine a second feature of the set of expected subsequent branches;

calculate a roll correction value based on the estimate of the roll of the instrument;

correct the second feature based on the roll correction value;

compare the first feature to the corrected second feature to determine a mapping of each of the plurality of openings to one of the expected subsequent branches; and

based at least in part on the mapping, provide an updated position state estimate.

10. The system of claim 9, wherein the instrument comprises an endoscope.

11. The system of claim 9, wherein the instrument positioning device comprises a robotic arm.

12. The system of claim 9, wherein the luminal network comprises a bronchial network of a lung, a gastrointestinal tract, or a renal network of a kidney.

13. The system of claim 9, wherein the instructions, when executed, cause the one or more processors to determine which opening of the plurality of detected openings is closer to a center of the image, and wherein the updated position state estimate comprises a probability that the instrument will be moved into the opening that is determined to be closer to the center of the image.

14. The system of claim 9, wherein the instructions, when executed, cause the one or more processors to determine the first feature of the detected openings, wherein the first feature is selected from the group consisting of:

a centroid of an opening, a profile of an opening, and a vector connecting centroids of two openings.

15. The system of claim 14, wherein the instructions, when executed, cause the one or more processors of the device to determine the second feature of the set of expected subsequent branches from the preoperative model, wherein the second feature comprises at least one of centroid of an opening, a profile of an opening, and a vector connecting centroids of two openings.

16. The system of claim 15, wherein the instructions, when executed, cause the one or more processors to compare one or more features of the detected plurality of openings to the set of expected subsequent branches by:

for each of the detected openings, iteratively matching the first feature of the detected opening to the corrected second feature of the set of expected subsequent branches,

wherein the highest match is used to map the detected opening to the one of the expected subsequent branches.

17. A method for navigating a luminal network, the method comprising:

inserting an instrument into a current branch of the luminal network, the instrument having an elongate body configured to be inserted into the luminal network and an imaging device positioned on a distal portion of the elongate body, the instrument attached to an instrument positioning device configured to move the instrument through the luminal network;

receiving a position state estimate for the instrument;

determining a set of expected subsequent branches based at least in part on the initial state estimate and a preoperative model of the luminal network;

capturing an image of the current branch with the imaging device positioned on the instrument;

analyzing the image to detect a plurality of openings connecting subsequent branches to the current branch;

determining an estimate of a roll of the instrument based on comparing an orientation of the detected openings within the image to an orientation of the set of expected subsequent branches;

determining a first feature of the detected plurality of openings;

determining a second feature of the set of expected subsequent branches;

calculating a roll correction value based on the estimate of the roll of the instrument;

correcting the second feature based on the roll correction value;

comparing the first feature to the corrected second feature to determine a mapping of each of the plurality of openings to one of the expected subsequent branches; and

based at least in part on the mapping, provide an updated position state estimate.

18. The method of claim 17, wherein the updated position state estimate comprises a probability that the position state estimate is correct.

19. The method of claim 18, wherein the probability is determined based in part on the comparison between the first feature to the corrected second feature.

20. The method of claim 19, wherein the probability is determined based in part on the degree to which the one or more features of the detected plurality of openings match the set of expected subsequent branches.

21. The method of claim 17, further comprising:

determining which opening of the plurality of detected openings is closer to a center of the image; and

wherein the updated position state estimate comprises a probability that the instrument will be moved into the opening that is determined to be closer to the center of the image.

22. The method of claim 17, further comprising:

determining the first feature of the detected openings, wherein the first feature is selected from the group consisting of;

a centroid of an opening, a profile of an opening, and a vector connecting centroids of two openings; and

determining the second feature of the set of expected subsequent branches from the preoperative model, wherein the information comprises at least one of centroid of an opening, a profile of an opening, and a vector connecting centroids of two openings; and

wherein comparing the first feature to the corrected second feature comprises;

for each of the detected openings, iteratively matching the first feature of the detected opening to the corrected second feature of the set of expected subsequent branches,

wherein the highest match is used to map the detected opening to the one of the expected subsequent branches.