Robotic systems for determining a roll of a medical device in luminal networks

• US 10,159,532 B1
• Filed: 06/29/2018
• Issued: 12/25/2018
• Est. Priority Date: 06/23/2017
• Status: Active Grant

• Current Assignee: Auris Health, Inc. (Johnson & Johnson)
• Original Assignee: Auris Health, Inc. (Johnson & Johnson)
• Inventors: Ummalaneni, Ritwik
• Primary Examiner(s): Lee, Jonathan S
• Application Number:
  US16/023,877
• Publication Number:
  US 20180368920A1
• Time in Patent Office: 179 Days
• CPC Class Codes: A61B 2017/00477 Coupling A61B2017/0046 takes precedence
• CPC Class Codes: A61B 2017/00809 Lung operations
• CPC Class Codes: A61B 2034/105 Modelling of the patient, e.g. for ligaments or bones
• CPC Class Codes: A61B 2034/2051 Electromagnetic tracking systems
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2034/301 for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes
• CPC Class Codes: A61B 2090/306 using optical fibres
• CPC Class Codes: A61B 2090/309 using white LEDs
• CPC Class Codes: A61B 2090/3614 using optical fibre
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/70 Manipulators specially adapted for use in surgery
• CPC Class Codes: A61B 5/06 Devices, other than using radiation, for detecting or locating foreign bodies for removing same A61B17/50 ; determining position of probes within or on the body of the patient
• CPC Class Codes: A61B 5/062 using magnetic field
• CPC Class Codes: A61B 6/032 Transmission computed tomography [CT]
• CPC Class Codes: A61B 6/0487 Motor-assisted positioning
• CPC Class Codes: A61B 6/102 Protection against mechanical damage, e.g. anti-collision devices
• CPC Class Codes: A61B 6/12 Arrangements for detecting or locating foreign bodies
• CPC Class Codes: A61B 6/4441 the rigid structure being a C-arm or U-arm
• CPC Class Codes: A61B 6/463 characterised by displaying multiple images or images and diagnostic data on one display
• CPC Class Codes: A61B 6/466 : adapted to display 3D data
• CPC Class Codes: A61B 6/487 : involving fluoroscopy
• CPC Class Codes: A61B 6/488 : involving pre-scan acquisition
• CPC Class Codes: B25J 9/1694 : characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion

Abstract

Certain aspects relate to systems and techniques for navigation-assisted medical devices. Some aspects relate to correlating features of depth information generated based on captured images of an anatomical luminal network with virtual features of depth information generated based on virtual images of a virtual representation of the anatomical luminal network in order to automatically determine aspects of a roll of a medical device within the luminal network.

Claims

1. A method of facilitating navigation of an anatomical luminal network of a patient, the method, executed by a set of one or more computing devices, comprising:

receiving imaging data captured by an imaging device at a distal end of an instrument positioned within the anatomical luminal network, wherein the anatomical luminal network includes a right bronchus and a left bronchus;

identifying one or more features from the imaging data, the one or more features identified from the imaging data representing a first positioning of the right bronchus and the left bronchus in the imaging data;

accessing one or more features associated with a virtual image, the virtual image simulated from a viewpoint of a virtual imaging device positioned at a virtual location within a virtual luminal network representative of the anatomical luminal network, the virtual luminal network including a virtual right bronchus corresponding to the right bronchus and a virtual left bronchus corresponding to the left bronchus, wherein the one or more features associated with the virtual image represent a second positioning of the virtual right bronchus and the virtual left bronchus in the virtual image;

calculating a correspondence between the first positioning and the second positioning; and

determining a roll of the distal end of the instrument within the anatomical luminal network based on the calculated correspondence.

2. The method of claim 1, wherein the anatomical luminal network includes a trachea leading into the right bronchus and the left bronchus, the method further comprising determining an insertion depth of the distal end of the instrument within the trachea based at least partly on the virtual location of the virtual image.

3. The method of claim 2, further comprising:

generating an electromagnetic field around the anatomical luminal network; and

determining a registration between a coordinate frame of the virtual luminal network and a coordinate frame of the electromagnetic field based at least partly on the insertion depth and roll of the distal end of the instrument.

4. The method of claim 1, further comprising:

generating a depth map based on the imaging data; and

identifying, based on the depth map, the first positioning of the right bronchus and the left bronchus in the imaging data;

wherein the one or more features associated with the virtual image comprise the second positioning derived from a virtual depth map associated with a virtual image.

5. The method of claim 4, further comprising:

generating the depth map by calculating, for each pixel of a plurality of pixels of the imaging data, a depth value representing an estimated distance between the imaging device and a tissue surface within the anatomical luminal network corresponding to the pixel;

identifying a first pixel of the plurality of pixels corresponding to a first depth criterion in the depth map;

determining a first location of the right bronchus in the imaging data based on a position of the first pixel;

identifying a second pixel of the plurality of pixels corresponding to a second depth criterion in the depth map; and

determining a second location of the left bronchus in the imaging data based on a position of the second pixel.

6. The method of claim 5, further comprising:

determining the first pixel has a greater depth value than the second pixel; and

identifying that the first location corresponds to the right bronchus based on determining the first pixel has the greater depth value.

7. The method of claim 5, wherein the first depth criterion represents a most distant imaged tissue within the right bronchus, and wherein the second depth criterion represents a most distant imaged tissue within the left bronchus.

8. The method of claim 5, wherein the first depth criterion corresponds to a first local maximum in a first region of depth values around the first pixel, and wherein the second depth criterion corresponds to a second local maximum in a second region of depth values around the second pixel.

9. The method of claim 4, further comprising determining the roll of the instrument based on an angular distance between a first position of the right bronchus in the imaging data and a second position of the virtual right bronchus in the virtual image.

10. A system configured to facilitate navigation of an anatomical luminal network of a patient, the system comprising:

an imaging device at a distal end of an instrument;

at least one computer-readable memory having stored thereon executable instructions; and

one or more processors in communication with the at least one computer-readable memory and configured to execute the instructions to cause the system to at least;

receive imaging data captured by the imaging device with the distal end of the instrument positioned within the anatomical luminal network, wherein the anatomical luminal network includes a right bronchus and a left bronchus;

identify one or more features from the imaging data, the one or more features identified from the imaging data representing a first positioning of the right bronchus and the left bronchus in the imaging data;

access one or more features associated with a virtual image, the virtual image simulated from a viewpoint of a virtual imaging device positioned at a virtual location within a virtual luminal network representative of the anatomical luminal network, the virtual luminal network including a virtual right bronchus corresponding to the right bronchus and a virtual left bronchus corresponding to the left bronchus, wherein the one or more features associated with the virtual image represent a second positioning of the virtual right bronchus and the virtual left bronchus in the virtual image;

calculate a correspondence between first positioning and the second positioning; and

determine a roll of the distal end of the instrument within the anatomical luminal network based on the calculated correspondence.

11. The system of claim 10, wherein the anatomical luminal network includes a trachea leading into the right bronchus and the left bronchus, and wherein the one or more processors are configured to execute the instructions to cause the system to at least determine an insertion depth of the distal end of the instrument within the trachea based at least partly on the virtual location of the virtual image.

12. The system of claim 11, wherein the one or more processors are configured to execute the instructions to cause the system to at least:

generate an electromagnetic field around the anatomical luminal network; and

determine a registration between a coordinate frame of the virtual luminal network and a coordinate frame of the electromagnetic field based at least partly on the insertion depth and roll of the distal end of the instrument.

13. The system of claim 10, wherein the one or more processors are configured to execute the instructions to cause the system to at least:

generate a depth map based on the imaging data; and

identify, based on the depth map, the first positioning of the right bronchus and the left bronchus in the imaging data;

wherein the one or more features associated with the virtual image comprise the second positioning derived from a virtual depth map associated with a virtual image.

14. The system of claim 13, wherein the one or more processors are configured to execute the instructions to cause the system to at least:

generate the depth map by calculating, for each pixel of a plurality of pixels of the imaging data, a depth value representing an estimated distance between the imaging device and a tissue surface within the anatomical luminal network corresponding to the pixel;

identify a first pixel of the plurality of pixels corresponding to a first depth criterion in the depth map;

determine a first location of the right bronchus in the imaging data based on a position of the first pixel;

identify a second pixel of the plurality of pixels corresponding to a second depth criterion in the depth map; and

determine a second location of the left bronchus in the imaging data based on a position of the second pixel.

15. The system of claim 14, wherein the one or more processors are configured to execute the instructions to cause the system to at least:

determine the first pixel has a greater depth value than the second pixel; and

identify that the first location corresponds to the right bronchus based on determining the first pixel has the greater depth value.

16. The system of claim 14, wherein the first depth criterion represents a most distant imaged tissue within the right bronchus, and wherein the second depth criterion represents a most distant imaged tissue within the left bronchus.

17. The system of claim 14, wherein the first depth criterion corresponds to a first local maximum in a first region of depth values around the first pixel, and wherein the second depth criterion corresponds to a second local maximum in a second region of depth values around the second pixel.

18. The system of claim 10, wherein the one or more processors are configured to execute the instructions to cause the system to at least determine the roll of the instrument based on an angular distance between a first position of the right bronchus in the imaging data and a second position of the virtual right bronchus in the virtual image.

19. A non-transitory computer readable storage medium having stored thereon instructions that, when executed, cause at least one computing device to at least:

receive imaging data captured by the imaging device with the distal end of the instrument positioned within the anatomical luminal network, wherein the anatomical luminal network includes a right bronchus and a left bronchus;

identify one or more features of the imaging data, the one or more features identified from the imaging data representing a first positioning of the right bronchus and the left bronchus in the imaging data;

access one or more features associated with a virtual image, the virtual image simulated from a viewpoint of a virtual imaging device positioned at a virtual location within a virtual luminal network representative of the anatomical luminal network, the virtual luminal network including a virtual right bronchus corresponding to the right bronchus and a virtual left bronchus corresponding to the left bronchus, wherein the one or more features associated with the virtual image represent a second positioning of the virtual right bronchus and the virtual left bronchus in the virtual image;

calculate a correspondence between first positioning and the second positioning; and

determine a roll of the distal end of the instrument within the anatomical luminal network based on the calculated correspondence.

20. The non-transitory computer readable storage medium of claim 19, wherein the anatomical luminal network includes a trachea leading into the right bronchus and the left bronchus, and wherein the instructions, when executed, cause the at least one computing device to at least determine an insertion depth of the distal end of the instrument within the trachea based at least partly on the virtual location of the virtual image.

21. The non-transitory computer readable storage medium of claim 20, wherein the instructions, when executed, cause the at least one computing device to at least:

generate an electromagnetic field around the anatomical luminal network; and

determine a registration between a coordinate frame of the virtual luminal network and a coordinate frame of the electromagnetic field based at least partly on the insertion depth and roll of the distal end of the instrument.

22. The non-transitory computer readable storage medium of claim 19, wherein the instructions, when executed, cause the at least one computing device to at least:

generate a depth map based on the imaging data; and

identify, based on the depth map, the first positioning of the right bronchus and the left bronchus in the imaging data;

wherein the one or more features associated with the virtual image comprise the second positioning derived from a virtual depth map associated with a virtual image.

23. The non-transitory computer readable storage medium of claim 22, wherein the instructions, when executed, cause the at least one computing device to at least:

generate the depth map by calculating, for each pixel of a plurality of pixels of the imaging data, a depth value representing an estimated distance between the imaging device and a tissue surface within the anatomical luminal network corresponding to the pixel;

identify a first pixel of the plurality of pixels corresponding to a first depth criterion in the depth map;

determine a first location of the right bronchus in the imaging data based on a position of the first pixel;

identify a second pixel of the plurality of pixels corresponding to a second depth criterion in the depth map; and

determine a second location of the left bronchus in the imaging data based on a position of the second pixel.

24. The non-transitory computer readable storage medium of claim 23, wherein the instructions, when executed, cause the at least one computing device to at least:

determine the first pixel has a greater depth value than the second pixel; and

identify that the first location corresponds to the right bronchus based on determining the first pixel has the greater depth value.

25. The non-transitory computer readable storage medium of claim 23, wherein the first depth criterion represents a most distant imaged tissue within the right bronchus, and wherein the second depth criterion represents a most distant imaged tissue within the left bronchus.

26. The non-transitory computer readable storage medium of claim 23, wherein the first depth criterion corresponds to a first local maximum in a first region of depth values around the first pixel, and wherein the second depth criterion corresponds to a second local maximum in a second region of depth values around the second pixel.

27. The non-transitory computer readable storage medium of claim 19, wherein the instructions, when executed, cause the at least one computing device to at least determine the roll of the instrument based on an angular distance between a first position of the right bronchus in the imaging data and a second position of the virtual right bronchus in the virtual image.