IMAGE-BASED BRANCH DETECTION AND MAPPING FOR NAVIGATION

• US 20190110843A1
• Filed: 10/13/2017
• Published: 04/18/2019
• Est. Priority Date: 10/13/2017
• Status: Active Grant

• Current Assignee: Auris Health, Inc. (Johnson & Johnson)
• Original Assignee: Auris Health, Inc. (Johnson & Johnson)
• Inventors: Ummalaneni, Ritwik
• Granted Patent: US 10,555,778 B2
• Time in Patent Office: Days
• CPC Class Codes: A61B 1/00 Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes examination of body cavities or body tracts using ultrasonic, sonic or infrasonic waves A61B8/12; endoscopic instruments for taking cell samples or for biopsy A61B10/04; for surgical purposes A61B17/00; endoscopic surgical cutting instruments A61B17/32; surgical instruments using a laser beam being directed along or through a flexible conduit A61B18/22; Illuminating arrangements therefor for the eyes A61B3/00
• CPC Class Codes: A61B 1/000094 extracting biological structures
• CPC Class Codes: A61B 1/0002 provided with data storages
• CPC Class Codes: A61B 1/00149 using articulated arms
• CPC Class Codes: A61B 1/05 characterised by the image sensor, e.g. camera, being in the distal end portion
• CPC Class Codes: A61B 1/2676 Bronchoscopes
• CPC Class Codes: A61B 1/2736 Gastroscopes
• CPC Class Codes: A61B 1/307 for the urinary organs, e.g. urethroscopes, cystoscopes
• CPC Class Codes: A61B 1/3132 for laparoscopy
• CPC Class Codes: A61B 10/0233 Pointed or sharp biopsy instruments
• CPC Class Codes: A61B 2017/00053 Mapping
• CPC Class Codes: A61B 2017/00477 Coupling A61B2017/0046 takes precedence
• CPC Class Codes: A61B 2017/00809 Lung operations
• CPC Class Codes: A61B 2034/105 Modelling of the patient, e.g. for ligaments or bones
• CPC Class Codes: A61B 2034/2051 Electromagnetic tracking systems
• CPC Class Codes: A61B 2034/2059 Mechanical position encoders
• CPC Class Codes: A61B 2034/2061 using shape-sensors, e.g. fiber shape sensors with Bragg gratings
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2034/301 for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes
• CPC Class Codes: A61B 2090/306 using optical fibres
• CPC Class Codes: A61B 2090/309 : using white LEDs
• CPC Class Codes: A61B 2090/3614 : using optical fibre
• CPC Class Codes: A61B 2090/376 : using X-rays, e.g. fluoroscopy
• CPC Class Codes: A61B 2217/005 : with suction drainage system
• CPC Class Codes: A61B 2217/007 : with irrigation system
• CPC Class Codes: A61B 34/20 : Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/30 : Surgical robots
• CPC Class Codes: A61B 46/10 : specially adapted for instruments, e.g. microscopes
• CPC Class Codes: A61B 50/13 : Trolleys, e.g. carts
• CPC Class Codes: A61G 13/04 : tiltable around transverse or longitudinal axis
• CPC Class Codes: A61G 13/08 : the table being divided into different adjustable sections
• CPC Class Codes: A61G 13/10 : Parts, details or accessories surgical drape sheets A61B46/00
• CPC Class Codes: A61G 13/12 : Rests specially adapted therefor; Arrangements of patient-supporting surfaces rests for the surgeon A61B90/60; restraining devices A61F5/37
• CPC Class Codes: A61G 2210/50 : for radiography
• CPC Class Codes: G06T 2207/10068 : Endoscopic image
• CPC Class Codes: G06T 2207/20076 : Probabilistic image processing
• CPC Class Codes: G06T 2207/30061 : Lung
• CPC Class Codes: G06T 2207/30084 : Kidney; Renal
• CPC Class Codes: G06T 2207/30092 : Stomach; Gastric
• CPC Class Codes: G06T 2207/30244 : Camera pose
• CPC Class Codes: G06T 7/11 : Region-based segmentation
• CPC Class Codes: G06T 7/136 : involving thresholding
• CPC Class Codes: G06T 7/73 : using feature-based methods
• CPC Class Codes: G06T 7/74 : involving reference images or patches

Abstract

Navigation of an instrument within a luminal network can include image-based branch detection and mapping. Image-based branch detection can include identifying within an image one or more openings associated with one or more branches of a luminal network. Image-based branch mapping can include mapping the detected one or more openings to corresponding branches of the luminal network. Mapping may include comparing features of the openings to features of a set of expected openings. A position state estimate for the instrument can be determined from the mapped openings, which can facilitate navigation of the luminal network.

Claims

1. A non-transitory computer readable storage medium having stored thereon instructions that, when executed, cause a processor of a device to at least:

determine a position state estimate of an instrument positioned within a current branch of a luminal network;

determine a set of expected subsequent branches based at least in part on the position state estimate and a preoperative model of the luminal network;

capture an image of the current branch with an imaging device positioned on the instrument;

detect within the image a plurality of openings connecting subsequent branches of the luminal network to the current branch;

compare one or more features of the detected plurality of openings to the set of expected subsequent branches to map each of the plurality of openings to one of the set of expected subsequent branches; and

based at least in part on the comparison, provide an updated position state estimate.

2. The non-transitory computer readable storage medium of claim 1, wherein the updated position state estimate comprises a probability that the position state estimate is correct.

3. The non-transitory computer readable storage medium of claim 2, wherein the probability is determined based in part on the comparison between the one or more features of the detected plurality of openings and the set of expected subsequent branches.

4. The non-transitory computer readable storage medium of claim 1, wherein the updated position state estimate comprises an estimate of which subsequent branch the instrument will be moved into.

5. The non-transitory computer readable storage medium of claim 1, wherein:

the instructions, when executed, cause the processor of the device to determine which opening of the plurality of detected openings is closer to a center of the image,

the updated position state estimate comprises a probability that the instrument will be moved into the opening that is determined to be closer to the center of the image.

6. The non-transitory computer readable storage medium of claim 1, wherein the position state estimate comprises an estimate of roll of the instrument about a longitudinal axis of the instrument, and wherein the updated position state estimate comprises a probability that the estimate of roll is correct, wherein the probability is determined by comparing an orientation of the detected openings within the image to an expected orientation of the set of expected subsequent branches based on the preoperative model.

7. The non-transitory computer readable storage medium of claim 1, wherein the instructions, when executed, cause the processor of the device to determine the one or more features of the detected openings selected from the group consisting of:

a centroid of an opening, a profile of an opening, and a vector connecting centroids of two openings.

8. The non-transitory computer readable storage medium of claim 7, wherein the instructions, when executed, cause the processor of the device to obtain information related to the set of expected subsequent branches from the preoperative model, wherein the information comprises at least one of centroid of an opening, a profile of an opening, and a vector connecting centroids of two openings.

9. The non-transitory computer readable storage medium of claim 8, wherein the instructions, when executed, cause the processor of the device to compare one or more features of the detected plurality of openings to the set of expected subsequent branches by:

for each of the detected openings, iteratively matching the one or more features of the detected opening to the information related to the set of expected subsequent branches,

wherein the highest match is used to map the detected opening to the one of the expected subsequent branches.

10. A robotic system for navigating a luminal network of a patient, the robotic system comprising:

an instrument having

an elongate body configured to be inserted into the luminal network, and

an imaging device positioned on a distal portion of the elongate body;

an instrument positioning device attached to the instrument, the instrument positioning device configured to move the instrument through the luminal network;

at least one computer-readable memory having stored thereon executable instructions; and

one or more processors in communication with the at least one computer-readable memory and configured to execute the instructions to cause the system to at least;

determine a position state estimate of the instrument positioned within a current branch of a luminal network;

determine a set of expected subsequent branches based at least in part on the initial state estimate and a preoperative model of the luminal network;

capture an image of the current branch of the luminal network with an imaging device positioned on the instrument;

detect within the image a plurality of openings connecting subsequent branches of the luminal network to the current branch;

compare features of the detected plurality of openings to the set of expected subsequent branches to map each of the plurality of openings to one of the expected subsequent branches; and

based at least in part on the comparison, provide an updated position state estimate.

11. The system of claim 10, wherein the instrument comprises an endoscope.

12. The system of claim 10, wherein the instrument positioning device comprises a robotic arm.

13. The system of claim 10, wherein the luminal network comprises a bronchial network of a lung, a gastrointestinal tract, or a renal network of a kidney.

14. The system of claim 10, wherein the instructions, when executed, cause the one or more processors to determine which opening of the plurality of detected openings is closer to a center of the image, and wherein the updated position state estimate comprises a probability that the instrument will be moved into the opening that is determined to be closer to the center of the image.

15. The system of claim 10, wherein the position state estimate comprises an estimate of roll of the instrument about a longitudinal axis of the instrument, and wherein the instructions, when executed, cause the one or more processors to determine a corrected estimate of roll by comparing an orientation of the detected openings within the image to an expected orientation of the set of expected subsequent branches based on the preoperative model.

16. The system of claim 10, wherein the instructions, when executed, cause the one or more processors to determine the one or more features of the detected openings, wherein the one or more features are selected from the group consisting of:

a centroid of an opening, a profile of an opening, and a vector connecting centroids of two openings.

17. The system of claim 16, wherein the instructions, when executed, cause the one or more processors of the device to obtain information related to the set of expected subsequent branches from the preoperative model, wherein the information comprises at least one of centroid of an opening, a profile of an opening, and a vector connecting centroids of two openings.

18. The system of claim 17, wherein the instructions, when executed, cause the one or more processors to compare one or more features of the detected plurality of openings to the set of expected subsequent branches by:

for each of the detected openings, iteratively matching the one or more features of the detected opening to the information related to the set of expected subsequent branches,

wherein the highest match is used to map the detected opening to the one of the expected subsequent branches.

19. A method for navigating a luminal network, the method comprising:

inserting an instrument into a current branch of the luminal network;

receiving a position state estimate for the instrument;

determining a set of expected subsequent branches based at least in part on the initial state estimate and a preoperative model of the luminal network;

capturing an image of the current branch with an imaging device positioned on the instrument;

analyzing the image to detect a plurality of openings connecting subsequent branches to the current branch;

comparing features of the detected plurality of openings to the set of expected subsequent branches to map each of the plurality of openings to one of the expected subsequent branches; and

based at least in part on the comparison, provide an updated position state estimate.

20. The method of claim 19, wherein the updated position state estimate comprises a probability that the position state estimate is correct.

21. The method of claim 20, wherein the probability is determined based in part on the comparison between the one or more features of the detected plurality of openings and the set of expected subsequent branches.

22. The method of claim 21, wherein the probability is determined based in part on the degree to which the one or more features of the detected plurality of openings match the set of expected subsequent branches.

23. The method of claim 19, further comprising:

determining which opening of the plurality of detected openings is closer to a center of the image; and

wherein the updated position state estimate comprises a probability that the instrument will be moved into the opening that is determined to be closer to the center of the image.

24. The method of claim 19, wherein the position state estimate comprises an estimate of roll of the instrument about a longitudinal axis of the instrument, and wherein the updated position state estimate comprises a probability that the estimate of roll is correct, the method further comprising:

comparing an orientation of the detected openings within the image to an expected orientation of the set of expected subsequent branches based on the preoperative model to determine the probability.

25. The method of claim 19, further comprising:

determining the one or more features of the detected openings, wherein the one or more features are selected from the group consisting of;

a centroid of an opening, a profile of an opening, and a vector connecting centroids of two openings; and

obtaining information related to the set of expected subsequent branches from the preoperative model, wherein the information comprises at least one of centroid of an opening, a profile of an opening, and a vector connecting centroids of two openings; and

wherein comparing features of the detected plurality of openings to the set of expected subsequent branches comprises;

for each of the detected openings, iteratively matching the one or more features of the detected opening to the information related to the set of expected subsequent branches,

wherein the highest match is used to map the detected opening to the one of the expected subsequent branches.

26. A method for identifying openings of branches of a luminal network, the method comprising:

capturing an image of an interior a branch of a luminal network with an imaging device positioned within the branch;

generating a histogram of pixel intensity values for the image;

identifying pixels below a threshold value as indicating openings within the image.

27. The method of claim 26, wherein determining the threshold value comprises:

identifying at least two peaks within the histogram;

identifying a midpoint between the at least two peaks; and

setting the threshold value equal to the intensity value of the midpoint.

28. The method of claim 26, further comprising, for each of the identified openings within the image, determining a centroid of the opening.

29. The method of claim 26, further comprising:

comparing a number of the identified openings in the image to a bad frame detector threshold; and

if the number of the identified openings exceeds the bad frame detector threshold;

capturing a second image of the interior of the branch; and

analyzing the second image to determine openings within the second image.

30. The method of claim 26, wherein the luminal network is a bronchial network of a lung, a gastrointestinal tract, or a renal network of a kidney.