Endolumenal object sizing

• US 10,136,959 B2
• Filed: 12/28/2016
• Issued: 11/27/2018
• Est. Priority Date: 12/28/2016
• Status: Active Grant

• Current Assignee: Auris Health, Inc. (Johnson & Johnson)
• Original Assignee: Auris Health, Inc. (Johnson & Johnson)
• Inventors: Mintz, David S., Schummers, David M., Jeevan, Prasanth, Rafii-Tari, Hedyeh, Ummalaneni, Ritwik
• Primary Examiner(s): Luong, Peter
• Application Number:
  US15/392,674
• Publication Number:
  US 20180177561A1
• Time in Patent Office: 699 Days
• CPC Class Codes: A61B 1/00009 of image signals during a use of endoscope
• CPC Class Codes: A61B 1/000094 extracting biological structures
• CPC Class Codes: A61B 1/04 combined with photographic or television appliances
• CPC Class Codes: A61B 1/06 with illuminating arrangements
• CPC Class Codes: A61B 1/307 for the urinary organs, e.g. urethroscopes, cystoscopes
• CPC Class Codes: A61B 2034/2051 Electromagnetic tracking systems
• CPC Class Codes: A61B 2034/303 specifically adapted for manipulations within body lumens, e.g. within lumen of gut, spine, or blood vessels
• CPC Class Codes: A61B 2090/061 for measuring dimensions, e.g. length
• CPC Class Codes: A61B 2090/364 Correlation of different images or relation of image positions in respect to the body
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/32 operating autonomously
• CPC Class Codes: A61B 5/062 using magnetic field
• CPC Class Codes: A61B 5/107 Measuring physical dimensions, e.g. size of the entire body or parts thereof
• CPC Class Codes: A61B 90/37 Surgical systems with images on a monitor during operation

Abstract

An object sizing system sizes an object positioned within a patient. The object sizing system identifies a presence of the object. The object sizing system navigates an elongate body of an instrument to a position proximal to the object within the patient. An imaging sensor coupled to the elongate body captures one or more sequential images of the object. The instrument may be further moved around within the patient to capture additional images at different positions/orientations relative to the object. The object sizing system also acquires robot data and/or EM data associated with the positions and orientations of the elongate body. The object sizing system analyzes the captured images based on the acquired robot data to estimate a size of the object.

Claims

1. A method for sizing an object positioned within a patient, comprising:

navigating an elongate body to a location within the patient and proximate to the object, the elongate body comprising an imaging sensor for capturing images of the object;

capturing sequential images of the object with the imaging sensor;

navigating the elongate body into at least one of a different orientation and a different position during the capture of the sequential images;

determining orientations and positions of the elongate body during the capture of the sequential images based on a set of robot data;

comparing the orientation and position of the elongate body determined based on the set of robot data for at least one captured sequential image to the orientation and position determined based on the set of robot data for at least another captured sequential image; and

analyzing, with a processor, the sequential images of the object and the comparison of the set of robot data to estimate a size of the object.

2. The method of claim 1, wherein the set of robot data comprises data from which a depth of the imaging sensor into the patient can be determined.

3. The method of claim 1, wherein the set of robot data comprises data from which position measurements about a position of the imaging sensor relative to the object can be determined.

4. The method of claim 1, wherein the set of robot data comprises information from which an articulation of the elongate body relative to the object can be determined.

5. The method of claim 1, wherein the set of robot data comprises at least one of data collected from a robotic manipulator and data collected from an electromagnetic sensor coupled to the elongate body.

6. The method of claim 1, further comprising:

inserting a guide wire into the patient to measure distance from the imaging sensor to the object.

7. The method of claim 1, wherein analyzing the sequential images of the object comprises:

combining the sequential images via image stitching techniques;

identifying a structure of the object; and

determining an estimated size of the identified object.

8. The method of claim 7, wherein the structure of the object comprises at least one of an outer axial diameter and a radius of the object.

9. The method of claim 1, further comprising:

capturing the object with a basket;

analyzing the sequential images depicting the object captured within the basket; and

determining an estimated size of the object based on the analyzed images.

10. The method of claim 9, wherein analyzing the sequential images of the captured object comprises:

detecting a basket marker in the sequential images; and

determining a distance to the object based on the detected basket marker; and

wherein determining the estimated size of the object is based on the distance to the object.

11. The method of claim 1, wherein:

the accessed set of robot data comprises data from which a position and size of a basket can be determined; and

analyzing the sequential images of the object and the set of robot data comprises;

identifying those images depicting at least a portion of the object and at least a portion of a basket; and

determining an estimated size of the object based on the position and the size of the basket relative to the object in the identified images.

12. The method of claim 11, wherein analyzing the sequential images of the object and the set of robot data comprises segmenting the sequential images to remove the basket from the analyzed images.

13. The method of claim 12, further wherein analyzing the sequential images of the object and the set of robot data comprises determining a structure of the object based on the segmented images.

14. The method of claim 1, further comprising:

illuminating the object with a structured light source;

wherein the sequential images are structured light images of the object;

wherein the analysis of the images of the object comprises

analyzing an illumination pattern projected on the object by the structured light source; and

determining an estimated size of the object based on the analysis of the illumination pattern.

15. The method of claim 14, wherein analyzing the sequential images comprises:

applying optical flow techniques on a plurality of pixels of the images of the object.

16. A system for sizing an object positioned within a patient comprising:

an elongate body, the elongate body comprising an imaging sensor for capturing images of the object, and

a computer comprising instructions that, when executed by a processor of the computer, cause the computer to perform steps of;

navigating the elongate body into the patient and to a location proximate to the object, the navigating enabled by a robotic manipulator;

capturing sequential images of the object with the imaging sensor;

navigating the elongate body into at least one of a different orientation and a different position during the capture of the sequential images;

determining orientations and positions of the elongate body during the capture of the sequential images based on a set of robot data;

comparing the orientation and position of the elongate body determined based on the set of robot data for at least one captured sequential image to the orientation and position determined based on the set of robot data for at least another captured sequential image; and

analyzing the sequential images and the comparison of the set of robot data to estimate a size of the object.

17. The system of claim 16, wherein the set of robot data comprises data from which a depth of the imaging sensor into the lumen can be determined.

18. The system of claim 16, wherein the set of robot data comprises data from which position measurements about a position of the imaging sensor relative to the object can be determined.

19. The system of claim 16, wherein the set of robot data comprises data from which an articulation of the elongate body can be determined.

20. The system of claim 16, wherein the set of robot data comprises at least one of data collected from a robotic manipulator and data collected from an electromagnetic sensor coupled to the elongate body.

21. The system of claim 16, further comprising:

a guide wire;

wherein the computer further comprises instructions that, when executed by the processor of the computer, cause the computer to perform steps of;

inserting a guide wire into the patient to measure distance from the imaging sensor to the object.

22. The system of claim 16, wherein analyzing the sequential images to estimate a size of the object further comprises:

combining the sequential images via image stitching techniques;

identifying a structure of the object; and

determining an estimated size of the object based on the identified structure.

23. The system of claim 22, wherein the identified structure of the object comprises at least one of an outer axial diameter and a radius of the object.

24. The system of claim 16, wherein the computer further comprises instructions that, when executed by the processor of the computer, cause the computer to perform steps of:

instructing capture the object with a basket;

analyzing the sequential images depicting the object captured within the basket; and

determining an estimated size of the object based on the analyzed images.

25. The system of claim 24, wherein analyzing the sequential images of the captured object comprises:

detecting a basket marker in the sequential images; and

determining a distance to the object based on the detected basket marker; and

wherein determining the estimated size of the object is based on the distance to the object.

26. The system of claim 16,

wherein the accessed set of robot data further data from which a position and size of a basket can be determined;

and

wherein analyzing the sequential images of the object and the set of robot data comprises;

identifying those images depicting at least a portion of the object and at least a portion of a basket; and

determining an estimated size of the object based on the position and the size of the basket relative to the object in the identified images.

27. The system of claim 26,

wherein analyzing the sequential images of the object and the comparison of the set of robot data comprises segmenting the sequential images to remove the basket from the analyzed images.

28. The system of claim 16, further comprising:

a structured light source positionable proximate to the object;

wherein the computer further comprises instructions that, when executed by the processor of the computer, cause the computer to perform steps of;

illuminating the object with a structured light source;

wherein the sequential images are structured light images of the object; and

wherein analyzing the sequential images of the object comprises

analyzing an illumination pattern projected on the object by the structured light source; and

determining an estimated size of the object based on the analysis of the illumination pattern.

29. The system of claim 16, wherein analyzing the sequential images comprises:

performing image analysis on the sequential images via applying optical flow techniques on a plurality of pixels of the object in the sequential images.