Methods and system for performing 3-D tool tracking by fusion of sensor and/or camera derived data during minimally invasive robotic surgery

• US 20060258938A1
• Filed: 05/16/2005
• Published: 11/16/2006
• Est. Priority Date: 05/16/2005
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Incorporated (Intuitive Surgical Holdings, LLC)
• Inventors: Larkin, David Q., Prisco, Giuseppe, Kumar, Rajesh, Hoffman, Brian David, Zhang, Guanghua G.
• Granted Patent: US 10,555,775 B2
• Time in Patent Office: Days
• US Class Current: 600/424
• CPC Class Codes: A61B 1/00193 adapted for stereoscopic vision
• CPC Class Codes: A61B 1/00194 adapted for three-dimensional imaging
• CPC Class Codes: A61B 1/04 combined with photographic or television appliances
• CPC Class Codes: A61B 1/3132 for laparoscopy
• CPC Class Codes: A61B 2034/102 Modelling of surgical devices, implants or prosthesis
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2090/0818 Redundant systems, e.g. using two independent measuring systems and comparing the signals
• CPC Class Codes: A61B 2090/364 Correlation of different images or relation of image positions in respect to the body
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 5/06 Devices, other than using radiation, for detecting or locating foreign bodies for removing same A61B17/50 ; determining position of probes within or on the body of the patient
• CPC Class Codes: A61B 5/061 Determining position of a probe within the body employing means separate from the probe, e.g. sensing internal probe position employing impedance electrodes on the surface of the body A61B1/00158 takes precedence
• CPC Class Codes: A61B 5/062 using magnetic field
• CPC Class Codes: A61B 5/725 using specific filters therefor, e.g. Kalman or adaptive filters A61B5/316 take precedence
• CPC Class Codes: A61B 90/36 Image-producing devices or illumination devices not otherwise provided for
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: A61B 90/39 Markers, e.g. radio-opaque or breast lesions markers

Abstract

Methods and system perform tool tracking during minimally invasive robotic surgery. Tool states are determined using triangulation techniques or a Bayesian filter from either or both non-endoscopically derived and endoscopically derived tool state information, or from either or both non-visually derived and visually derived tool state information. The non-endoscopically derived tool state information is derived from sensor data provided either by sensors associated with a mechanism for manipulating the tool, or sensors capable of detecting identifiable signals emanating or reflecting from the tool and indicative of its position, or external cameras viewing an end of the tool extending out of the body. The endoscopically derived tool state information is derived from image data provided by an endoscope inserted in the body so as to view the tool.

Claims

1. A tool tracking method comprising:

tracking a tool by processing non-endoscopically derived tool state information and endoscopically derived tool state information generated while the tool is inserted and being manipulated through a minimally invasive incision in a body.

2. The method according to claim 1, wherein the processing of the non-endoscopically derived tool state information and endoscopically derived tool state information is performed using a Bayesian filter.

3. The method according to claim 2, wherein the endoscopically derived tool state information is provided to the Bayesian filter less frequently than the non-endoscopically derived tool state information.

4. The method according to claim 2, wherein the non-endoscopically derived tool state information is continuously provided at a sampling rate to the Bayesian filter for processing, and the endoscopically-derived tool state information is non-continuously provided to the Bayesian filter for processing.

5. The method according to claim 2, wherein the Bayesian filter is provided with an initial estimate of the average difference between a non-endoscopically derived tool state and an endoscopically derived tool state.

6. The method according to claim 5, wherein the Bayesian filter updates the estimate of the average difference between the non-endoscopically derived tool state and the endoscopically derived tool state while estimating the state of the tool using the non-endoscopically derived tool state and the endoscopically derived tool state.

7. The method according to claim 1, wherein the non-endoscopically derived tool state information is generated from sensor data indicative of at least a position of the tool in a fixed reference frame.

8. The method according to claim 7, wherein the sensor data is indicative of an orientation of the tool in the fixed reference frame.

9. The method according to claim 7, wherein the sensor data is provided by position sensors coupled to a mechanism for manipulating the tool through the incision in the body, and the non-endoscopically derived tool state information is generated from kinematics of the mechanism using the sensor data.

10. The method according to claim 9, wherein the processing of the non-endoscopically derived tool state information and endoscopically derived tool state information is performed using a Bayesian filter.

17. The method according to claim 1, wherein the non-endoscopically derived tool state information originates from an external camera viewing an end of the tool extending out of the body.

18. The method according to claim 1, wherein the endoscopically derived tool state information is generated from image data provided by an endoscope viewing the tool while it is inserted and being manipulated through the minimally invasive incision in the body.

19. The method according to claim 18, wherein the endoscope is a stereoscopic endoscope providing left and right image planes, and generation of the endoscopically derived tool state information includes determining at least the position of the tool in a reference frame of the endoscope by processing information of the left and the right image planes.

20. The method according to claim 19, further comprising:

determining the position of the endoscope in a fixed reference frame, and determining the position of the tool in the fixed reference frame from information of the position of the tool in the reference frame of the endoscope and the position of the endoscope in the fixed reference frame.

21. The method according to claim 18, wherein the generation of the endoscopically derived tool state information includes identifying an orientation dependent tool marker in the image data, and determining a endoscopically derived estimate of at least the orientation of the tool by processing information of the orientation dependent tool marker in the image data.

22. The method according to claim 18, wherein the processing of the non-endoscopically derived tool state information and the endoscopically-derived tool state information comprises:

generating a computer model of the tool positioned and oriented within an image plane defined by the image data according to the non-endoscopically derived tool state information; and

modifying the position and orientation of the computer model with respect to an image of the tool in the image plane until the computer model approximately overlays the image of the tool so as to generate a corrected position and orientation of the tool.

23. The method according to claim 22, wherein the modification of the position and orientation of the computer model with respect to the image of the tool until the computer model approximately overlays the image of the tool, comprises:

determining the modified position and orientation of the computer model that approximately overlays the tool image by minimizing a difference between the computer model and the image of the tool.

24. The method according to claim 2, wherein the Bayesian filter is a Kalman filter.

25. The method according to claim 24, wherein the Kalman filter is an extended Kalman filter.

26. The method according to claim 2, wherein the Bayesian filter is a particle filter.

11. The method according to claim 11, wherein quality measures are respectively derived for the non-endoscopically and the endoscopically derived tool states, and weightings of contributions of the non-endoscopically and the endoscopically derived tool states in the Bayesian filter are determined by the quality measures.

12. The method according to claim 11, wherein the quality measure for the non-endoscopically derived tool state is determined from a difference between the position in the fixed reference frame indicated by the sensor data and a position in the fixed reference frame indicated by a command signal controlling a robotic mechanism for manipulating the tool.

13. The method according to claim 11, wherein the sensor data is provided by a detector detecting a signal indicative of the position of the tool in a fixed reference frame.

14. The method according to claim 13, wherein the signal emanates from the tool.

15. The method according to claim 13, wherein the signal reflects off of the tool.

16. The method according to claim 11, wherein the sensor data is processed so as to determine whether the tool is being manipulated at the time so that processing of the endoscopically derived tool state information is only performed if the tool is being manipulated at the time.

27. A tool tracking method comprising:

receiving sensor information indicative of a position and orientation of a tool when the tool is inserted through an incision in a body;

receiving image information for the tool; and

determining the position and orientation of the tool using both the sensor and the image information.

28. The method according to claim 27, wherein the sensor information is associated with a mechanism used for robotically manipulating the tool through the incision in the body.

29. The method according to claim 28, wherein the mechanism includes a robotic arm for manipulating the tool, and a plurality of sensors generating data associated with movement of the robotic arm from which kinematics information is derived.

30. The method according to claim 27, wherein the image information is generated by at least one camera inserted in the body so as to capture images of the tool when inserted therein.

31. The method according to claim 30, wherein the at least one camera is included in one or more endoscopes.

32. The method according to claim 30, wherein the at least one camera consists of two cameras included in a stereoscopic endoscope.

33. The method according to claim 27, wherein the image information is generated by at least one camera external to the body so as to capture images of an end of the tool extending out of the body when the other end of the tool is inserted therein.

34. The method according to claim 27, wherein the determination of the tool position and orientation comprises:

determining one or more kinematic estimated positions and orientations of the tool relative to a fixed reference frame from the sensor information;

determining one or more image estimated positions and orientations of the tool relative to a camera reference frame from the image information;

translating the one or more kinematic estimated positions and orientations of the tool from the fixed reference frame to the camera reference frame; and

processing the one or more kinematic and the one or more image estimated positions and orientations to generate the tool position and orientation relative to the camera reference frame.

35. The method according to claim 34, wherein the processing of the one or more kinematic estimated positions and orientations and the one or more image estimated positions and orientations, comprises:

providing the one or more kinematic and the one or more image estimated positions and orientations to a Bayesian filter.

36. The method according to claim 35, wherein the one or more image estimated positions and orientations are determined less frequently than the one or more kinematic estimated positions and orientations.

37. The method according to claim 35, wherein the Bayesian filter is provided with an initial estimate of the average difference between a sensor derived tool state and an image derived tool state.

38. The method according to claim 37, wherein the Bayesian filter updates the estimate of the average difference between the sensor derived tool state and the image derived tool state while estimating the state of the tool using the sensor derived tool state and the image derived tool state.

39. The method according to claim 36, wherein the one or more kinematic estimated positions and orientations derive from time sampled information provided by one or more sensors coupled to a mechanism for manipulating the tool through the incision in the body, and the one or more image estimated positions and orientations derive from sampled images provided by one or more cameras so as to capture images of the tool.

40. The method according to claim 39, wherein quality measures are respectively derived for the one or more kinematic estimated positions and orientations and the one or more image estimated positions and orientations, and weightings of contributions the one or more kinematic estimated positions and orientations and the one or more image estimated positions and orientations in the Bayesian filter are determined by the quality measures.

41. The method according to claim 40, wherein the quality measure for the one or more kinematic estimated positions and orientations is determined from a difference between one of the kinematic estimated positions and a position being commanded by a command signal controlling the mechanism for manipulating the tool.

42. The method according to claim 41, wherein the one or more cameras are included in an endoscope positioned within the body to capture images of an end of the tool extended into the body.

43. The method according to claim 41, wherein the one or more cameras are positioned outside of the body to capture images of an end of the tool extending out of the body.

44. The method according to claim 37, wherein the Bayesian filter is a Kalman filter.

45. The method according to claim 44, wherein the Kalman filter is an extended Kalman filter.

46. The method according to claim 37, wherein the Bayesian filter is a particle filter.

47. The method according to claim 27, wherein the determination of the tool position and orientation includes processing the image information to identify a marker on the tool, and determine an orientation of the tool using the marker.

48. The method according to claim 27, wherein the determination of the tool position and orientation includes generating a computer model of the tool using the sensor information so as to be positioned and oriented within an image plane defined in the image information, and modifying the position and orientation of the computer model with respect to an image of the tool in the image plane until the computer model substantially overlays the image.

49. The method according to claim 48, wherein the position and orientation of the computer model in the image plane is a modified position and orientation of the computer model determined for a prior in time image plane.

50. The method according to claim 48, wherein the position and orientation of the computer model in the image plane is derived from at least the kinematic information corresponding in time to that image plane.

51. The method according to claim 49, wherein the image information is generated by an endoscope inserted in the body so as to capture images of an end of the tool inserted therein.

52. The method according to claim 51, wherein the endoscope is a stereoscopic endoscope.

53. The method according to claim 49, wherein the image information is generated by at least one camera external to the body so as to capture images of an end of the tool extending out of the body when the other end of the tool is inserted therein.

54. A minimally invasive robotic surgery system with tool tracking, comprising:

one or more non-endoscopic devices providing data from which non-endoscopically derived tool state information is generated when a tool is inserted and robotically manipulated through an incision in a body;

an endoscope capturing images from which endoscopically derived tool state information is generated for an area within the body when the tool is inserted therein; and

a processor configured to process the non-endoscopically and endoscopically derived tool state information for tracking the state of the tool.

55. The system according to claim 54, wherein the endoscope is a stereoscopic endoscope.

56. The system according to claim 54, further comprising a mechanism used for manipulating the tool through the incision in the body, wherein the one or more non-endoscopic devices include one or more sensors providing sensor data representing kinematic information according to such manipulation.

57. The system according to claim 56, wherein the sensor data includes digitized samples of an identifiable signal emanating from or reflecting off the tool so as to indicate the position of the tool.

58. The system according to claim 54, wherein the one or more non-endoscopic devices include a camera viewing an end of the tool extending out of the body.

59. The system according to claim 54, wherein the processor is further configured to track the state of the tool by:

determining one or more non-endoscopic estimated tool states from the non-endoscopically derived tool state information;

determining one or more endoscopic estimated tool states from the endoscopically derived tool state information; and

processing the one or more non-endoscopic and one or more endoscopic estimated tool states to generate the state of the tool.

60. The system according to claim 59, wherein the processor is further configured to use a Bayesian filter for processing the one or more non-endoscopic and one or more endoscopic estimated tool states to generate the state of the tool.

61. The system according to claim 60, wherein the Bayesian filter is a Kalman filter.

62. The system according to claim 61, wherein the Kalman filer is an extended Kalman filter.

63. The system according to claim 60, wherein the Bayesian filter is a particle filter.

64. The system according to claim 54, wherein the processor is further configured to identify a marker on the tool from the endoscope captured images, and determines an orientation of the tool using the marker while tracking the state of the tool.

65. The system according to claim 54, wherein the processor is further configured to generate a computer model of the tool positioned and oriented within an image plane defined in the endoscope captured images, and modify the position and orientation of the computer model with respect to an image of the tool in the image plane until the computer model substantially overlaps the image.

66. The system according to claim 65, wherein the processor is further configured to derive the position and orientation of the computer model in the image plane from a modified position and orientation of the computer model determined by the processor for a prior in time image plane.

67. The system according to claim 65, wherein the processor is further configured to derive the position and orientation of the computer model in the image plane from at least the non-endoscopically derived tool state information corresponding in time to the image plane.

68. A minimally invasive robotic surgery system with tool tracking, comprising:

one or more sensors providing sensor data from which non-visually derived tool state information for a tool is generated when the tool is inserted and robotically manipulated through an incision in a body;

at least one camera capturing image information of the tool when the tool is inserted therein; and

a processor configured to process the non-visually derived tool state information and the image information for tracking the state of the tool.

69. The system according to claim 68, further comprising a mechanism used for manipulating the tool through the incision in the body, wherein the sensor data represents kinematic information according to such manipulation.

70. The system according to claim 68, wherein the sensor data includes digitized samples of an identifiable signal emanating from or reflecting off the tool and indicating the position of the tool.

71. The system according to claim 68, wherein the at least one camera is included in one or more endoscopes positioned so as to view an end of the tool inserted in the body.

72. The system according to claim 68, wherein the at least one camera consists of two cameras included in a stereoscopic endoscope.

73. The system according to claim 68, wherein the processor is further configured to track the state of the tool by:

determining one or more non-visual estimated states of the tool from the non-visually derived tool state information;

determining one or more visual estimated states of the tool from the image information; and

processing the one or more non-visual and the one or more visual estimated states to generate the state of the tool.

74. The system according to claim 73, wherein the processor is further configured to:

determine an error transform from a corresponding pair of the non-visual and visual estimated states of the tool; and

generate a state of the tool for a subsequent time by correcting a selected one of the non-visual or visual estimated states of the tool for the subsequent time using the error transform.

75. The system according to claim 74, wherein the processor receives the image information less frequently than the non-visually derived tool state information for processing.

76. The system according to claim 75, wherein the one or more non-visual estimated states derive from time sampled information provided by the one or more sensors, and the one or more visual estimated states derive from time sampled images provided by one or more cameras.

77. The system according to claim 75, wherein the processor is further configured to use a Bayesian filter for processing the one or more non-visual and the one or more visual estimated states to generate the state of the tool.

78. The system according to claim 77, wherein the Bayesian filter is a Kalman filter.

79. The system according to claim 78, wherein the Kalman filter is an extended Kalman filter.

80. The system according to claim 77, wherein the Bayesian filter is a particle filter.

81. The system according to claim 77, wherein the processor receives the image information less frequently than the non-visually derived tool state information for processing.

82. The system according to claim 81, wherein the one or more non-visual estimated states derive from time sampled information provided by the one or more sensors, and the one or more visual estimated states derive from time sampled images provided by one or more cameras.

83. The system according to claim 68, wherein the processor is further configured to identify a marker on the tool from the image information, and determine an orientation of the tool using the marker while tracking the state of the tool.

84. The system according to claim 68, wherein the processor is further configured to generate a computer model of the tool positioned and oriented within an image plane defined in the image information, and modify the position and orientation of the computer model with respect to an image of the tool in the image plane until the computer model substantially overlays the image.

85. The system according to claim 80, wherein the processor is configured to derive the position and orientation of the computer model in the image plane from a modified position and orientation of the computer model determined by the processor for a prior in time image plane.

86. The system according to claim 84, wherein the processor is configured to derive the position and orientation of the computer model in the image plane from at least the non-visually derived tool state information corresponding in time to the image plane.

87. A tool tracking method comprising:

determining a computer model of a tool;

receiving a captured image including a view of the tool;

determining an estimated position and orientation of the tool from the captured image, and positioning and orienting the computer model at that estimated position and orientation in reference to the captured image; and

modifying the estimated position and orientation of the computer model with respect to an image of the tool in the captured image until the computer model approximately overlays the image so as to correct the estimated position and orientation of the tool for the captured image.

88. The method according to claim 87, wherein the modification of the estimated position and orientation of the computer model with respect to the image of the tool in the captured image, comprises:

determining the modified position and orientation of the computer model that approximately overlays the tool image by minimizing a difference between the computer model and the image of the tool.

89. The method according to claim 87, wherein the captured image is captured by a stereo endoscope inserted into an area of the body.

90. The method according to claim 87, further comprising:

extracting a silhouette of the computer model of the tool;

extracting edges of the image of the tool in the captured image; and

positioning and rotating the silhouette of the computer model until the silhouette of the computer model approximately overlays the edges of the image of the tool within the captured image.

91. The method according to claim 90, further comprising:

removing lines in the silhouette of the computer model of the tool corresponding to hidden lines in the image of the tool in the captured image prior to positioning and rotating the silhouette of the computer model until the silhouette of the computer model approximately overlays the edges of the image of the tool within the captured image.

92. The method according to claim 90, wherein the positioning and rotating of the silhouette of the computer model until the silhouette approximately overlays the edges of the image of the tool within the captured image, comprises:

determining a difference between the silhouette of the computer model and the image of the tool in the captured image; and

positioning and rotating the silhouette of the computer model until the difference is minimized.

93. The method according to claim 92, wherein the difference to be minimized is a sum of absolute differences between edge pixels extracted from the tool image and their closest silhouette edges.

94. The method according to claim 92, wherein the view of the area where the tool is inserted is captured as a grid of pixels by at least one camera inserted in the area.

95. A tool tracking method comprising:

determining whether sensor data indicative of a tool state is available for a point in time;

determining whether image data indicative of the tool state is available for the point in time; and

determining the tool state using both the sensor data and the image data if both are available for the point in time, or using only the sensor data if only the sensor data is available, or using only the image data if only the image data is available.

96. The method according to claim 95, further comprising:

determining whether a user is manipulating the tool at the point in time; and

performing the sensor data and image data availability determinations only if the user is determined to be manipulating the tool at the point in time.

97. The method according to claim 95, further comprising:

determining whether the state of the tool has changed relative to a prior point in time using the sensor data; and

performing the image data availability and tool state determinations only if the state of the tool is determined to have changed.

98. The method according to claim 95, wherein the tool state indicated by the sensor data includes a position and orientation of the tool in a fixed reference frame.

99. The method according to claim 98, wherein the image data is generated by a camera unit, the sensor data further indicates a position and orientation of the camera unit in the fixed reference frame, and the tool state indicated by the image data includes a position and orientation of the tool in a reference frame related to the camera unit.

100. The method according to claim 99, wherein the determination of the tool state using the sensor data and the image data, comprises:

translating the position and orientation of the tool indicated by the sensor data from the fixed reference frame to the reference frame related to the camera unit before determining the tool state when using both sensor data and the image data.

101. A tool tracking method comprising:

determining a first estimated tool state relative to a landmark for a point in time using first sensor data indicative of the tool state at the point in time;

determining an estimated camera state relative to the landmark for the point in time using second sensor data indicative of the camera state at the point in time;

determining a second estimated tool state relative to the camera for the point in time using image data generated by the camera and indicative of the tool state at the point in time;

translating the first estimated tool state so as to be relative to the camera instead of the landmark; and

computing an error transform between the translated first and the second estimated tool states that at a subsequent point in time if image data indicative of the tool state at the subsequent point in time is not available, then the tool state is determined by applying the error transform to a third estimated tool state determined using sensor data indicative of the tool state at the subsequent point in time translated so as to be relative to the camera instead of the landmark.

102. The method according to claim 101, further comprising if the image data indicative of the tool state at the subsequent point in time is available:

determining a fourth estimated tool state relative to the camera for the subsequent point in time using image data generated by the camera and indicative of the tool state at the subsequent point in time; and

determining the tool state at the subsequent point in time as being the fourth estimated tool state.

103. The method according to claim 102, further comprising if the image data indicative of the tool state at the subsequent point in time is available:

computing a second error transform between the translated third and the fourth estimated tool states so that at a subsequent, subsequent point in time if image data indicative of the tool state at the subsequent, subsequent point in time is not available, then the tool state is determined by applying the second error transform to a fifth estimated tool state determined using sensor data indicative of the tool state at the subsequent, subsequent point in time translated so as to be relative to the camera instead of the landmark.

104. A tool tracking method comprising:

determining non-endoscopically derived estimated state information for a tool at a given time;

determining endoscopically estimated state information for the tool at the given time; and

providing the non-endoscopically and endoscopically derived estimated states for the tool to a Bayesian filter configured so as to generate an optimal estimate of the state of the tool.

105. The method according to claim 104, wherein the Bayesian filter is provided with an initial estimate of an average difference between a non-endoscopically derived tool state and an endoscopically derived tool state.

106. The method according to claim 105, wherein the Bayesian filter updates the estimate of the average difference between the non-endoscopically derived tool state and the endoscopically derived tool state while estimating the state of the tool using the non-endoscopically derived tool state and the endoscopically derived tool state.

107. The method according to claim 104, wherein the non-endoscopically derived estimated state information is generated from sensor data indicative of at least a position of the tool in a fixed reference frame at the given time.

108. The method according to claim 107, wherein the endoscopically derived estimated state information is generated from image data received from at least one camera capturing image information for an area of the body when the tool is inserted therein.

109. The method according to claim 104, wherein the Bayesian filter is a Kalman filter.

110. The method according to claim 109, wherein the Kalman filter is an extended Kalman filter.

111. The method according to claim 104, wherein the Bayesian filter is a particle filter.

112. A tool tracking and calibration method comprising:

generating visually derived state information from image data received from a camera viewing a tool;

generating state vector information by combining initial values for a set of camera parameters with the visually derived state information; and

providing the state vector information to a Bayesian filter for processing so as to generate an optimal estimate of a state of the tool and corrected values for the set of camera parameters.

113. The method according to claim 112, further comprising:

generating non-visually derived state information from sensor data received from one or more sensors and including information indicative of a position of the tool; and

generating the state vector information by combining the non-visually derived state information with the initial values for the set of camera parameters and the visually derived state information.

114. The method according to claim 113, wherein the camera is an endoscope.

115. The method according to claim 113, wherein the one or more sensors are associated with a robotic mechanism used for manipulating the tool through a minimally invasive incision in a body.

116. A camera tracking method comprising:

determining a position of a tool in a fixed reference frame from non-visually derived tool state information generated from sensor data indicative of the position of the tool;

determining a position of the tool in a camera frame moveable with a camera using visually derived tool state information generated from image data provided by the camera while viewing the tool; and

determining a position of the camera in the fixed reference frame using the position of the tool in the fixed reference frame and the position of the tool in the moveable camera frame.

117. A tool tracking method comprising:

determining a position of a camera in a fixed reference frame from non-visually derived camera state information generated from sensor data indicative of the position of the camera;

determining a position of a tool in a camera frame moveable with the camera using visually derived tool state information generated from image data provided by the camera while viewing the tool; and

determining a position of the tool in the fixed reference frame using the position of the camera in the fixed reference frame and the position of the tool in the moveable camera frame.

118. A tool tracking method comprising:

generating a plurality of estimated tool states for each point in a plurality of points in time, while the tool is inserted and being manipulated through an incision in a body; and

determining an optimal estimated tool state for each point in the plurality of points in time by processing the plurality of estimated tool states using Bayesian techniques.

119. The method according to claim 118, wherein the plurality of estimated tool states include an estimated tool state determined using only sensor data associated with a robotic mechanism for manipulating the tool, so as to be indicative of movement of the robotic mechanism.

120. The method according to claim 119, wherein the robotic mechanism includes joints and linkages, and the sensor data is indicative of movement of the joints and linkages.

121. The method according to claim 118, wherein the plurality of estimated tool states include an estimated tool state determined using only sensor data associated with the tool, so as to be indicative of a position of the tool.

122. The method according to claim 121, wherein the sensor data is further indicative of an orientation of the tool.

123. The method according to claim 118, wherein the plurality of estimated tool states include an estimated tool state determined using only image data generated by an endoscope positioned so as to view an effector end of the tool as the tool is inserted and being manipulated through the incision in the body.

124. The method according to claim 118, wherein the plurality of estimated tool states include an estimated tool state determined using only image data generated by an external camera positioned so as to view an exposed end of the tool extending out of the body as the tool is inserted and being manipulated through the incision in the body.

125. The method according to claim 118, wherein the Bayesian technique is a Kalman filtering technique.

126. The method according to claim 118, wherein the Bayesian technique is a particle filtering technique.