NAVIGATION OF TUBULAR NETWORKS

• US 20190228525A1
• Filed: 12/17/2018
• Published: 07/25/2019
• Est. Priority Date: 09/18/2015
• Status: Active Grant

• Current Assignee: Auris Health, Inc. (Johnson & Johnson)
• Original Assignee: Auris Health, Inc. (Johnson & Johnson)
• Inventors: Mintz, David S., Ghoreyshi, Atiyeh, Jeevan, Prasanth, Xu, Yiliang, Yang, Gehua, Leotta, Matthew Joseph, Stewart, Charles V.
• Granted Patent: US 10,796,432 B2
• Time in Patent Office: Days
• CPC Class Codes: A61B 1/000094 extracting biological structures
• CPC Class Codes: A61B 1/00147 Holding or positioning arrangements
• CPC Class Codes: A61B 1/00149 using articulated arms
• CPC Class Codes: A61B 1/0016 using motor drive units
• CPC Class Codes: A61B 1/00194 adapted for three-dimensional imaging
• CPC Class Codes: A61B 1/2676 Bronchoscopes
• CPC Class Codes: A61B 2017/00809 Lung operations
• CPC Class Codes: A61B 2034/105 Modelling of the patient, e.g. for ligaments or bones
• CPC Class Codes: A61B 2034/107 Visualisation of planned trajectories or target regions
• CPC Class Codes: A61B 2034/2048 using an accelerometer or inertia sensor
• CPC Class Codes: A61B 2034/2051 Electromagnetic tracking systems
• CPC Class Codes: A61B 2034/2055 Optical tracking systems
• CPC Class Codes: A61B 2034/252 indicating steps of a surgical procedure
• CPC Class Codes: A61B 2034/301 for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes
• CPC Class Codes: A61B 2090/3614 using optical fibre
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/25 User interfaces for surgical systems
• CPC Class Codes: A61B 5/061 Determining position of a probe within the body employing means separate from the probe, e.g. sensing internal probe position employing impedance electrodes on the surface of the body A61B1/00158 takes precedence
• CPC Class Codes: A61B 6/032 Transmission computed tomography [CT]
• CPC Class Codes: A61B 90/30 Devices for illuminating a surgical field, the devices having an interrelation with other surgical devices or with a surgical procedure
• CPC Class Codes: G06T 15/205 : Image-based rendering
• CPC Class Codes: G06T 17/00 : Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/10068 : Endoscopic image
• CPC Class Codes: G06T 2207/10081 : Computed x-ray tomography [CT]
• CPC Class Codes: G06T 2207/30061 : Lung
• CPC Class Codes: G06T 2207/30196 : Human being; Person
• CPC Class Codes: G06T 7/0012 : Biomedical image inspection
• CPC Class Codes: G06T 7/149 : involving deformable models, e.g. active contour models
• CPC Class Codes: G06T 7/248 : involving reference images or patches
• CPC Class Codes: G06T 7/32 : using correlation-based methods
• CPC Class Codes: G06T 7/73 : using feature-based methods

Abstract

Methods and apparatuses provide improved navigation through tubular networks such as lung airways by providing improved estimation of location and orientation information of a medical instrument (e.g., an endoscope) within the tubular network. Various input data such as image data, EM data, and robot data are used by different algorithms to estimate the state of the medical instrument, and the state information is used to locate a specific site within a tubular network and/or to determine navigation information for what positions/orientations the medical instrument should travel through to arrive at the specific site. Probability distributions together with confidence values are generated corresponding to different algorithms are used to determine the medical instrument'"'"'s estimated state.

Claims

1-22. -22. (canceled)

23. A surgical robotic system comprising:

an endoscopic tool having a flexible tip;

an optical sensor coupled to the flexible tip;

a processor coupled to the optical sensor, the processor configured with instructions to;

receive a plurality of images from the optical sensor;

identify, within an image of the plurality of images, a division of the tubular network, the division comprising a plurality of openings to a plurality of respective segments of the tubular network;

determine a first estimated state of the flexible tip within the tubular network based on the identified division of the tubular network; and

show, on a display, an estimated position of the flexible tip within a segment of the plurality of segments of the luminal network, wherein the estimated position is determined based on the first estimated state.

24. The system of claim 23, wherein the processor is configured to identify the division with the image by detecting the plurality of openings within the image.

25. The system of claim 24, wherein processor is configured to detect the plurality of openings based on light reflective intensity within the image.

26. The system of claim 24, wherein processor is configured to:

access model data from a three dimensional model of the tubular network; and

compare the detected plurality of openings with virtual branch information from the model data.

27. The system of claim 26, wherein the three dimensional model of the tubular network is generated based on computerized axial tomography (CT) scans of the tubular network.

28. The system of claim 26, wherein the processor is configured to compare the detected plurality of openings with the virtual branch information from the model data by compare the detected plurality of openings with a virtual image of the division, wherein the virtual image is generated based on the model data.

29. The system of claim 28, wherein the processor is configured to:

show, on the display, the image;

rotate the virtual image to match an orientation of the image; and

show, on the display, the virtual image.

30. The system of claim 24, wherein the processor is configured to access model data comprising a tree diagram representative of the luminal network and determine the first estimated state based on the tree diagram.

31. The system of claim 23, wherein the plurality of openings comprise three openings and the division comprises a trifurcation.

32. The system of claim 23, wherein the plurality of images are captured sequentially in time.

33. The system of claim 24, wherein the processor is configured to track the detected plurality of openings over a set of sequential image frames of the plurality of images to detect a segment into which the flexible tip has moved.

34. The system of claim 24, wherein the processor is configured to track relative positions of the detected plurality of openings over a set of sequential image frames of the plurality of images to determine a measurement of roll of the flexible tip.

35. The system of claim 23, wherein the processor is configured to:

determine a second estimate state of the position of the device within the tubular network based on a second set of sensor data; and

determine the estimated position of the flexible tip based on the first estimated state and the second estimated state.

36. The system of claim 35, wherein the second set of sensor data comprises data derived from a position sensor.

37. The system of claim 35, wherein the second set of sensor data comprises data regarding physical manipulation of the endoscopic tool.

38. A non-transitory computer readable storage medium comprising computer program instructions that when executed by one or more processors cause the one or more processors to:

receive a plurality of images from an optical sensor on a flexible tip of an endoscopic tool inserted into a tubular network;

identify, within an image of the plurality of images, a division of the tubular network, the division comprising a plurality of openings to a plurality of respective segments of the tubular network;

determine a first estimated state of the flexible tip within the tubular network based on the identified division of the tubular network; and

show, on a display, an estimated position of the flexible tip within a segment of the plurality of segments of the luminal network, wherein the estimated position is determined based on the first estimated state.

39. The non-transitory computer readable storage medium of claim 38, wherein the instructions, when executed, cause the one or more processors to:

determine a second estimate state of the position of the device within the tubular network based on a second set of sensor data; and

determine the estimated position of the flexible tip based on the first estimated state and the second estimated state.

40. The non-transitory computer readable storage medium of claim 39, wherein the second set of sensor data comprises data derived from a position sensor.

41. A method for navigating a tubular network, the method comprising:

receiving a plurality of images from an optical sensor on a flexible tip of an endoscopic tool inserted into a tubular network;

identifying, within an image of the plurality of images, a division of the tubular network, the division comprising a plurality of openings to a plurality of respective segments of the tubular network;

determining a first estimated state of the flexible tip within the tubular network based on the identified division of the tubular network; and

showing, on a display, an estimated position of the flexible tip within a segment of the plurality of segments of the luminal network, wherein the estimated position is determined based on the first estimated state.

42. The method of claim 41, further comprising:

determining a second estimate state of the position of the device within the tubular network based on a second set of sensor data; and

determining the estimated position of the flexible tip based on the first estimated state and the second estimated state.