Automated endoscope calibration

• US 10,314,463 B2
• Filed: 06/23/2016
• Issued: 06/11/2019
• Est. Priority Date: 10/24/2014
• Status: Active Grant

• Current Assignee: Auris Health, Inc. (Johnson & Johnson)
• Original Assignee: Auris Health, Inc. (Johnson & Johnson)
• Inventors: Agrawal, Varun, Ghoreyshi, Atiyeh, Mintz, David S.
• Primary Examiner(s): Newton, Alexandra L
• Application Number:
  US15/191,391
• Publication Number:
  US 20160374541A1
• Time in Patent Office: 1,083 Days
• CPC Class Codes: A61B 1/00006 of control signals
• CPC Class Codes: A61B 1/000094 extracting biological structures
• CPC Class Codes: A61B 1/00057 provided with means for testing or calibration
• CPC Class Codes: A61B 1/00149 using articulated arms
• CPC Class Codes: A61B 1/0016 using motor drive units
• CPC Class Codes: A61B 1/0052 Constructional details of control elements, e.g. handles
• CPC Class Codes: A61B 2034/301 for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes
• CPC Class Codes: A61B 2090/064 for measuring force, pressure or mechanical tension
• CPC Class Codes: A61B 34/30 Surgical robots

Abstract

A surgical robotic system automatically calibrates tubular and flexible surgical tools such as endoscopes. By accounting for nonlinear behavior of an endoscope, the surgical robotic system can accurately model motions of the endoscope and navigate the endoscope while performing a surgical procedure on a patient. The surgical robotic system models the nonlinearities using sets of calibration parameters determined based on images captured by an image sensor of the endoscope. Calibration parameters can describe translational or rotational movements of the endoscope in one or more axis, e.g., pitch and yaw, as well as a slope, hysteresis, or dead zone value corresponding to the endoscope'"'"'s motion. The endoscope can include tubular components referred to as a sheath and leader. An instrument device manipulator of the surgical robotic system actuates pull wires coupled to the sheath or the leader, which causes the endoscope to articulate.

Claims

1. A method for controlling an endoscope of a surgical robotic system, comprising:

receiving a command to move the endoscope using the surgical robotic system;

accessing a set of calibration parameters associated with the endoscope, the set of calibration parameters including at least a first set of calibration parameters based on an image registration process using a plurality of images captured by an image sensor coupled to a tip of the endoscope during a calibration procedure that comprises movement of the endoscope between images captured by the image sensor, the calibration procedure including generating a difference array based on differences between images capture by the image sensor between movement of the endoscope performed before receiving the command to move the endoscope and generating a gradient array based on rates of change between the images captured by the image sensor;

generating an adjusted command based on the command and the set of calibration parameters;

providing the adjusted command to the surgical robotic system to move the endoscope; and

receiving movement information describing motion of the endoscope in response at least to the adjusted command.

2. The method of claim 1, wherein the movement information is received from a strain gauge of an instrument device manipulator of the surgical robotic system.

3. The method of claim 1, wherein the set of calibration parameters describes nonlinear behavior of the endoscope and includes:

a second set of calibration parameters based on a calibration curve fitting process using previously recorded movement information of the endoscope.

4. The method of claim 3, wherein the nonlinear behavior is based at least in part on a stiffness and a length of a helix section of the endoscope.

5. The method of claim 3, wherein generating the adjusted command comprises:

modifying the command based on at least one calibration parameter from the first set; and

modifying the command based on at least one calibration parameter from the second set.

6. The method of claim 3, wherein the plurality of images includes information about a pattern on a surface of a dome-shaped calibration structure.

7. The method of claim 1, wherein the image registration process using the plurality of images comprises:

generating the difference array based on differences in pixel values between a first image of a pair and a second image of the pair;

generating the gradient array based on rates of change in pixel values between a first image of the pair and a second image of the pair; and

wherein the first set of calibration parameters is based at least in part on the difference arrays and the gradient arrays.

8. The method of claim 7, wherein the image sensor includes a plurality of color channels and wherein the image registration process using the plurality of images further comprises applying a weighted average to the difference arrays and the gradient arrays based on a sensitivity of each of the color channels.

9. The method of claim 3, wherein the movement information includes at least:

a first segment describing motion of the endoscope in a forward direction along an axis; and

a second segment describing motion of the endoscope in a backward direction along the axis.

10. The method of claim 9, wherein the calibration curve fitting process using previously recorded movement information comprises:

generating a slope by averaging a first rate of change of position of the endoscope based on the first segment and a second rate of change of position of the endoscope based on the second segment;

generating a hysteresis by determining a difference between the first segment and the second segment; and

wherein the second set of calibration parameters includes the slope and the hysteresis.

11. The method of claim 1, wherein the endoscope in configured to move in one or more axes including at least a pitch axis and a yaw axis, and wherein the command includes at least a translational motion and a rotational motion.

12. A surgical robotic system, comprising:

an endoscope;

a robotic arm including a drive mechanism, the drive mechanism coupled to the endoscope; and

a controller configured to;

receive a command to move the endoscope using the surgical robotic system;

access a set of calibration parameters associated with the endoscope, the set of calibration parameters including at least a first set of calibration parameters based on an image registration process using a plurality of images captured by an image sensor coupled to a tip of the endoscope during a calibration procedure that comprises movement of the endoscope between images captured by the image sensor, the calibration procedure including generating a difference array based on differences between images capture by the image sensor between movement of the endoscope performed before receiving the command to move the endoscope and generating a gradient array based on rates of change between the images captured by the image sensor;

generate an adjusted command based on the command and the set of calibration parameters;

provide the adjusted command to the surgical robotic system to move the endoscope; and

receive movement information describing motion of the endoscope in response at least to the adjusted command.

13. The system of claim 12, wherein the endoscope includes a sheath tubular component coupled to a leader tubular component, the leader tubular component including a helix section.

14. The system of claim 12, wherein the set of calibration parameters is based on an image registration process and calibration curve fitting process, and wherein the set of calibration parameters describe nonlinear behavior of the endoscope.