Methods and systems for robotic instrument tool tracking with adaptive fusion of kinematics information and image information

• US 8,108,072 B2
• Filed: 09/30/2007
• Issued: 01/31/2012
• Est. Priority Date: 09/30/2007
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Zhao, Wenyi, Nowlin, William C., Hoffman, Brian D., Hasser, Christopher J J
• Primary Examiner(s): Black, Thomas
• Assistant Examiner(s): BROWDER, LINDSAY
• Application Number:
  US11/865,014
• Publication Number:
  US 20090088897A1
• Time in Patent Office: 1,584 Days
• Field of Search: 700/245, 700/250, 700/253, 700/254, 700/258, 700/259, 700/262, 901/46, 901/57, 382/128, 382/153, 318/568.11, 318/568.16, 318/568.22, 318/568.24, 600/101, 600/103, 600/407, 600/410, 600/424, 600/429, 606/130, 348/45, 348/65, 348/73, 703/6, 703/7
• US Class Current: 700/250
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2090/3612 with images taken automatically
• CPC Class Codes: A61B 2090/371 with simultaneous use of two cameras
• CPC Class Codes: A61B 2090/3784 both receiver and transmitter being in the instrument or receiver being also transmitter
• CPC Class Codes: A61B 2090/3937 Visible markers
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: G05B 2219/40557 Tracking a tool, compute 3-D position relative to camera
• CPC Class Codes: G05B 2219/45117 Medical, radio surgery manipulator
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 2201/034 of medical instruments

Abstract

In one embodiment of the invention, a method for a robotic system is disclosed to track one or more robotic instruments. The method includes generating kinematics information for the robotic instrument within a field of view of a camera; capturing image information in the field of view of the camera; and adaptively fusing the kinematics information and the image information together to determine pose information of the robotic instrument. Additionally disclosed is a robotic medical system with a tool tracking sub-system. The tool tracking sub-system receives raw kinematics information and video image information of the robotic instrument to generate corrected kinematics information for the robotic instrument by adaptively fusing the raw kinematics information and the video image information together.

Claims

1. A method for a robotic system, the method comprising:

generating kinematics information for a robotic instrument within a field of view of a camera;

capturing image information in the field of view of the camera; and

adaptively fusing the kinematics information and the image information together to determine pose information of the robotic instrument;

wherein the adaptive fusing of the kinematics information and the image information together includes synthesizing an image of a model of the robotic instrument in response to the kinematics information, matching the image of the model to the image information to generate match results, filtering the match results, and forming a plurality of states of the pose information for the robotic instrument in response to the filtered match results and kinematics information.

2. The method of claim 1, wherein

the pose information of the robotic instrument includes location and orientation of the robotic instrument with respect to the camera and has a quality greater than the kinematics information alone.

3. The method of claim 1, further comprising:

sensing first kinematics information of the robotic instrument at a first position and capturing first image information of the robotic instrument at the first position;

sensing second kinematics information of the robotic instrument at a second position and capturing second image information of the robotic instrument at the second position; and

wherein the first kinematics information, the first image information, the second kinematics information, and the second image information are adaptively fused together to track the robotic instrument.

4. The method of claim 1, wherein

the adaptive fusing of the kinematics information and the image information together includes

preferring the kinematics information over the image information in generating the pose information if the image information is unreliable.

5. The method of claim 4, wherein

the adaptive fusing of the kinematics information and the image information together further includes

preferring the image information over the kinematics information in generating the pose information if the image information is reliable or if the kinematics information is unreliable.

6. The method of claim 1, comprising:

if an image of the robotic instrument is unavailable, the method further comprises predicting the pose of the robotic instrument in response to prior information.

7. The method of claim 6, wherein

the pose of the robotic instrument is predicted in response to prior kinematics information.

8. The method of claim 7, wherein

the pose of the robotic instrument is predicted in further response to current kinematics information.

9. The method of claim 6, wherein

the pose of the robotic instrument is predicted in response to prior image information.

10. The method of claim 1, wherein

the adaptive fusing of the kinematics information and the image information together includes

adaptively fusing together first image information of a first video image source with second image information of a second video image source.

11. The method of claim 1, wherein

the matching of the image of the model of the robotic instrument to the image information to generate match results includes

reading a computer aided design model of the robotic instrument to determine one or more markers thereof, and

analyzing the image information for the one or more markers to generate a match.

12. The method of claim 11, wherein

the one or more markers form a pattern and the match of the one or more markers in the image information is a pattern match.

13. The method of claim 11, wherein

the one or more markers are artificial markers.

14. The method of claim 13, wherein

the one or more artificial markers are a pattern of dots.

15. The method of claim 11, wherein

the one or more markers are natural markers consisting of geometry information of the computer aided design model.

16. The method of claim 1, wherein

the camera is a stereo camera and the capturing of image information in the field of view of the camera includes

capturing left and right stereo images in the field of view of the stereo camera.

17. The method of claim 1, wherein

the camera includes a first camera in a first position over the robotic instrument and a second camera in a second position differing from the first over the robotic instrument, and

the capturing of image information in the field of view of the plurality of cameras includes

capturing first images in the field of view of the first camera, and

capturing second images in the field of view of the second camera.

18. The method of claim 1, wherein

the robotic instrument includes a marker with a geometric relationship, and

the capturing of image information in the field of view of the camera includes

capturing an image of the marker with the geometric relationship to improve image matching.

19. The method of claim 1, wherein

the robotics instrument is coupled to a robotic arm of a robotic system to generate the kinematics information.

20. A robotic medical system comprising:

a robotic instrument coupled to a robotic arm to be manipulated and generate raw kinematics information;

a camera to capture video image information of a portion of the robotic instrument; and

a tool tracking sub-system to receive the raw kinematics information and the video image information of the robotic instrument, the tool tracking sub-system to generate corrected kinematics information for the robotic instrument by adaptively fusing the raw kinematics information and the video image information together, wherein the adaptive fusing of the kinematics information and the video image information together includes synthesizing an image of a model of the robotic instrument in response to the kinematics information, matching the image of the model to the video image information to generate match results, filtering the match results, and forming a plurality of states of the pose information for the robotic instrument in response to the filtered match results and kinematics information.

21. The robotic medical system of claim 20, wherein

the tool tracking sub-system adaptively fuses the raw kinematics information and the video image information together to localize and automatically track the robotic instrument.

22. The robotic medical system of claim 20, further comprising:

a video display coupled to the camera and the tool tracking sub-system, the video display to display an operative image aligned to an image of a tissue surface to guide the robotic instrument into a patient'"'"'s body.

23. The robotic medical system of claim 22, wherein

the operative image is a pre-operative image, and

the pre-operative image is a magnetic resonance image (MRI) of a portion of the patient'"'"'s body or a computed tomography (CT) image of a portion of the patient'"'"'s body.

24. The robotic medical system of claim 22, wherein

the robotic instrument is an ultrasound probe,

the operative image is an intra-operative image, and

the intra-operative image is an ultrasound image of a portion of the patient'"'"'s body generated by the ultrasound probe.