Methods of locating and tracking robotic instruments in robotic surgical systems

• US 8,147,503 B2
• Filed: 09/30/2007
• Issued: 04/03/2012
• Est. Priority Date: 09/30/2007
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Zhao, Wenyi, Hasser, Christopher J., Nowlin, William C., Hoffman, Brian D.
• Primary Examiner(s): Vanore, David A
• Application Number:
  US11/865,015
• Publication Number:
  US 20090088773A1
• Time in Patent Office: 1,647 Days
• Field of Search: 600/407, 600/424, 600/427, 600/437, 606/130, 700/245
• US Class Current: 606/130
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2090/371 with simultaneous use of two cameras
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/25 User interfaces for surgical systems
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 2201/034 of medical instruments

Abstract

In one embodiment of the invention, a method is disclosed to locate a robotic instrument in the field of view of a camera. The method includes capturing sequential images in a field of view of a camera. The sequential images are correlated between successive views. The method further includes receiving a kinematic datum to provide an approximate location of the robotic instrument and then analyzing the sequential images in response to the approximate location of the robotic instrument. An additional method for robotic systems is disclosed. Further disclosed is a method for indicating tool entrance into the field of view of a camera.

Claims

1. A method to locate a robotic instrument in a field of view of a camera, the method comprising:

capturing sequential images in a field of view of a camera, the sequential images being correlated between successive views;

receiving a kinematic datum to provide an approximate location of the robotic instrument; and

analyzing the sequential images in response to the approximate location of the robotic instrument.

2. The method of claim 1, further comprising:

locating the robotic instrument in the field of view in response to the analysis of the sequential images.

3. The method of claim 1, further comprising:

estimating a current location of the robotic instrument in the field of view in response to the analysis of the sequential images and a portion of the robotic instrument being hidden from the field of view.

4. The method of claim 1, further comprising:

synthesizing a model of the robotic instrument; and

further analyzing the sequential images in response to the model of the robotic instrument.

5. The method of claim 1, wherein

the kinematic datum is a current kinematic datum, and the method further comprises

correcting the current kinematic datum to correct future kinematic datum without regard to corrections of past kinematic datum.

6. The method of claim 1, wherein

the kinematic datum is a current kinematic datum, and the method further comprises

correcting the current kinematic datum in response to corrections of past kinematic datum such that future kinematic datum can be corrected.

7. The method of claim 1, wherein

the sequential images form a depth map, and

the depth map is analyzed in response to the approximate location of the robotic instrument to locate the robotic instrument in the field of view.

8. A method for robotic systems, the method comprising:

receiving one or more video images of at least one robotic instrument in reference to a camera coordinate system;

receiving kinematics information associated with the at least one robotic instrument; and

localizing the at least one robotic instrument in the camera coordinate system in response to the one or more video images and the kinematics information.

9. The method of claim 8, wherein

the localizing of the at least one robotic instrument includes

predicting the position of the at least one robotic instrument in response to the kinematics information.

10. The method of claim 9, wherein

the localizing of the at least one robotic instrument includes

synthesizing an image of the at least one robotic instrument in response to a computer aided design (CAD) model of the at least one robotic instrument;

and

analyzing the video images to match the synthesized tool image to an actual tool image to determine a position of the robotic instrument.

11. The method of claim 10, wherein

the analyzing to determine the position includes

minimizing a cost function of comparing the synthesized tool image with the actual tool image, and

minimizing a sum of squared differences between geometries of the synthesized tool image and the actual tool image.

12. The method of claim 11, wherein

the minimizing of the sum of squared differences includes

non-linearly mapping a three dimensional synthesized tool image into a two dimensional synthesized tool image.

13. The method of claim 12, further comprising:

learning a change in appearance of the at least one robotic instrument in response to a different view of the at least one robotic instrument.

14. The method of claim 13, wherein

the change in appearance includes a change in one or more of lighting, occlusion, wear, and the surrounding environment of the robotic instrument.

15. The method of claim 8, further comprising:

after localizing the at least one robotic instrument,

tracking the motion of the at least one robotic instrument in the camera coordinate system in response to the one or more video images and the kinematics information.

16. The method of claim 15, wherein

if while tracking the motion the at least one instrument is occluded so as to become untrackable, then

the localizing of the at least one robotic instrument in the camera coordinate system is repeated to locate the robotic instrument in the camera coordinate system.

17. The method of claim 15, wherein

the tracking of the motion of the at least one robotic instrument includes

adaptively fusing the video images and the kinematics information together to determine a position of the at least one robotic instrument.

18. The method of claim 17, wherein

the adaptively fusing includes

weighting the kinematics information more than the video images if the video images are unreliable indicators of a location of the at least one robotic instrument.

19. The method of claim 17, wherein

the adaptively fusing includes

weighting the video images more than kinematics information if the kinematics information are unreliable indicators of a location of the at least one robotic instrument.

20. The method of claim 17, wherein

the adaptively fusing includes

weighting the video images more than kinematics information if the video images are reliable indicators of a location of the at least one robotic instrument.

21. The method of claim 17, wherein

the adaptively fusing includes

using a covariance matrix with a state space model to adaptively fuse the video images and kinematics information together to provide a reliable indicator of a location of the at least one robotic instrument.

22. A method for indicating tool entrance into a field of view of a camera, the method comprising:

tracking a robotic surgical tool in and out of a field of view of a camera;

determining that the robotic surgical tool is outside the field of view of the camera in response to the tracking of the robotic surgical tool; and

displaying one of a plurality of compass icons in a display of the field of view to indicate a direction of tool reentrance into the field of view.

23. The method of claim 22, further comprising:

altering the one compass icon in the display by flashing, adding arrowheads, or changing color to indicate an increase and a decrease in a distance between an edge of the field of view of the camera and the robotic surgical tool.

24. The method of claim 22, wherein

the camera has moved away from the position of the robotic surgical tool in a surgical site such that the robotic surgical tool is outside the field of view of the camera.

25. The method of claim 22, wherein

the robotic surgical tool has moved away from a position of the camera in a surgical site such that the robotic surgical tool is outside the field of view of the camera.