Methods and system for performing 3-D tool tracking by fusion of sensor and/or camera derived data during minimally invasive robotic surgery

• US 10,555,775 B2
• Filed: 05/16/2005
• Issued: 02/11/2020
• Est. Priority Date: 05/16/2005
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Hoffman, Brian David, Larkin, David Q., Prisco, Giuseppe, Zhang, Guanghua G, Kumar, Rajesh
• Primary Examiner(s): Nganga, Boniface N
• Application Number:
  US11/130,471
• Publication Number:
  US 20060258938A1
• Time in Patent Office: 5,384 Days
• Field of Search: 600423, 600424, 606130, 700245, 700262
• CPC Class Codes: A61B 1/00193 adapted for stereoscopic vision
• CPC Class Codes: A61B 1/00194 adapted for three-dimensional imaging
• CPC Class Codes: A61B 1/04 combined with photographic or television appliances
• CPC Class Codes: A61B 1/3132 for laparoscopy
• CPC Class Codes: A61B 2034/102 Modelling of surgical devices, implants or prosthesis
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2090/0818 Redundant systems, e.g. using two independent measuring systems and comparing the signals
• CPC Class Codes: A61B 2090/364 Correlation of different images or relation of image positions in respect to the body
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 5/06 Devices, other than using radiation, for detecting or locating foreign bodies for removing same A61B17/50 ; determining position of probes within or on the body of the patient
• CPC Class Codes: A61B 5/061 Determining position of a probe within the body employing means separate from the probe, e.g. sensing internal probe position employing impedance electrodes on the surface of the body A61B1/00158 takes precedence
• CPC Class Codes: A61B 5/062 using magnetic field
• CPC Class Codes: A61B 5/725 using specific filters therefor, e.g. Kalman or adaptive filters A61B5/316 take precedence
• CPC Class Codes: A61B 90/36 Image-producing devices or illumination devices not otherwise provided for
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: A61B 90/39 Markers, e.g. radio-opaque or breast lesions markers

Abstract

Methods and system perform tool tracking during minimally invasive robotic surgery. Tool states are determined using triangulation techniques or a Bayesian filter from either or both non-endoscopically derived and endoscopically derived tool state information, or from either or both non-visually derived and visually derived tool state information. The non-endoscopically derived tool state information is derived from sensor data provided either by sensors associated with a mechanism for manipulating the tool, or sensors capable of detecting identifiable signals emanating or reflecting from the tool and indicative of its position, or external cameras viewing an end of the tool extending out of the body. The endoscopically derived tool state information is derived from image data provided by an endoscope inserted in the body so as to view the tool.

Claims

1. A tool tracking and control method for minimally invasive robotic surgery, the method comprising:

a processor determining non-endoscopically derived estimated positions of a tool by using information received from a non-endoscope device while the tool is being robotically manipulated;

the processor determining endoscopically derived estimated positions of the tool by using information received from an endoscope while the tool is being robotically manipulated;

the processor implementing a Bayesian filter which generates optimal estimates of the positions of the tool by processing together the non-endoscopically and endoscopically derived estimated positions of the tool; and

the processor controlling use of one of the tool and the endoscope by using the optimal estimates of the positions of the tool.

2. The method according to claim 1, wherein the Bayesian filter is provided with an initial estimate of an average difference between a non-endoscopically derived tool position and an endoscopically derived tool position.

3. The method according to claim 2, wherein the Bayesian filter updates the estimate of the average difference between the non-endoscopically derived tool position and the endoscopically derived tool position while estimating the positions of the tool using the non-endoscopically derived tool position and the endoscopically derived tool position.

4. The method according to claim 1, wherein the non-endoscopically derived estimated position information is generated from sensor data which is received from the non-endoscope device.

5. The method according to claim 4, wherein the endoscopically derived estimated position information is generated from image data received from at least one camera of the endoscope which captures image information for an area of the body when the tool is inserted therein.

6. The method according to claim 1, wherein the Bayesian filter is a Kalman filter.

7. The method according to claim 6, wherein the Kalman filter is an extended Kalman filter.

8. The method according to claim 1, wherein the Bayesian filter is a particle filter.

9. A tool tracking and control method for minimally invasive robotic surgery, the method comprising:

a processor generating visually derived state information of a tool from image data received from a camera viewing the tool while the tool is being robotically manipulated;

the processor generating state vector information by combining initial values for a set of camera intrinsic parameters with the visually derived state information of the tool, wherein the camera intrinsic parameters include at least one of focal length, principal point, skew, and distortion, and wherein the state vector information is organized into a matrix that is to be multiplied by a state vector including state variables for the tool and the set of camera intrinsic parameters;

the processor implementing a Bayesian filter and providing the state vector information to the Bayesian filter for processing according to the state vector, so the Bayesian filter generates an optimal estimate of a state of the tool and corrected values for the set of camera intrinsic parameters; and

the processor controlling use of one of the tool and the camera by using at least one of the optimal estimate of the state of the tool and the corrected values for the set of camera intrinsic parameters.

10. The method according to claim 9, further comprising:

the processor generating non-visually derived state information of the tool from sensor data received from one or more sensors, the sensor data including information indicative of a position of the tool; and

the processor generating the state vector information by combining the non-visually derived state information of the tool with the initial values for the set of camera intrinsic parameters and the visually derived state information of the tool.

11. The method according to claim 10, wherein the camera is associated with an endoscope.

12. The method according to claim 10, wherein the one or more sensors are associated with a robotic mechanism used for manipulating the tool.

13. A tool tracking and control method for minimally invasive robotic surgery, the method comprising:

a processor generating a first plurality of estimated tool positions for each point in a plurality of points in time by using at least one of sensor data associated with a robotic mechanism for manipulating the tool, sensor data associated with the tool, and image data generated by an external camera positioned so as to view an exposed end of the tool extending out of the body, while the tool is inserted through an incision in a body and being manipulated by the robotic mechanism;

the processor generating a second plurality of estimated tool positions for each point in the plurality of points in time by using image data generated by an endoscope positioned so as to view an effector end of the tool;

the processor determining an optimal estimated tool position for each point in the plurality of points in time by using a Bayesian technique to process the first and second pluralities of estimated tool positions; and

the processor controlling use of one of the tool and the endoscope by using the optimal estimated tool position for each point in the plurality of points in time.

14. The method according to claim 13, wherein the robotic mechanism includes joints and linkages, wherein generating the first plurality of estimated tool positions for each point in the plurality of points in time comprises generating the first plurality of estimated tool positions by using the sensor data associated with the robotic mechanism, and wherein the sensor data associated with the robotic mechanism includes sensor data that is indicative of movement of at least one of the joints and linkages.

15. The method according to claim 13, wherein the processor generating the first plurality of estimated tool positions for each point in the plurality of points in time comprises the processor generating the first plurality of estimated tool positions by using the sensor data associated with the tool, and wherein the sensor data associated with the tool includes sensor data that is indicative of the position of the tool.

16. The method according to claim 13, wherein the Bayesian technique is a Kalman filtering technique.

17. The method according to claim 13, wherein the Bayesian technique is a particle filtering technique.