Tool tracking systems, methods and computer products for image guided surgery

• US 8,073,528 B2
• Filed: 09/30/2007
• Issued: 12/06/2011
• Est. Priority Date: 09/30/2007
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Zhao, Wenyi, Hasser, Christopher J., Nowlin, William C., Hoffman, Brian D.
• Primary Examiner(s): Casler, Brian
• Assistant Examiner(s): LAURITZEN, AMANDA L
• Application Number:
  US11/865,016
• Publication Number:
  US 20090088634A1
• Time in Patent Office: 1,528 Days
• Field of Search: 600/424, 600/427, 600/437, 600/407, 700/245, 606/130
• US Class Current: 600/424
• CPC Class Codes: A61B 1/00193 adapted for stereoscopic vision
• CPC Class Codes: A61B 1/00194 adapted for three-dimensional imaging
• CPC Class Codes: A61B 1/018 for receiving instruments
• CPC Class Codes: A61B 1/04 combined with photographic or television appliances
• CPC Class Codes: A61B 1/3132 for laparoscopy
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2090/061 for measuring dimensions, e.g. length
• CPC Class Codes: A61B 2090/364 Correlation of different images or relation of image positions in respect to the body
• CPC Class Codes: A61B 2090/3983 Reference marker arrangements for use with image guided surgery
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 5/061 Determining position of a probe within the body employing means separate from the probe, e.g. sensing internal probe position employing impedance electrodes on the surface of the body A61B1/00158 takes precedence
• CPC Class Codes: A61B 5/1076 for measuring dimensions inside body cavities, e.g. using catheters A61B3/1005 takes precedence
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: B25J 13/089 Determining the position of the robot with reference to its environment
• CPC Class Codes: B25J 15/0019 End effectors other than grippers
• CPC Class Codes: B25J 9/1656 characterised by programming, planning systems for manipulators
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10068 : Endoscopic image
• CPC Class Codes: G06T 2207/30004 : Biomedical image processing
• CPC Class Codes: G06T 2207/30241 : Trajectory
• CPC Class Codes: G06T 7/246 : using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/255 : Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 2201/034 : of medical instruments
• CPC Class Codes: G09B 23/28 : for medicine
• CPC Class Codes: G16H 20/40 : relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
• CPC Class Codes: G16H 30/40 : for processing medical images, e.g. editing
• CPC Class Codes: G16H 40/67 : for remote operation

Abstract

In one embodiment of the invention, a tool tracking system is disclosed including a computer usable medium having computer readable program code to receive images of video frames from at least one camera and to perform image matching of a robotic instrument to determine video pose information of the robotic instrument within the images. The tool tracking system further includes computer readable program code to provide a state-space model of a sequence of states of corrected kinematics information for accurate pose information of the robotic instrument. The state-space model receives raw kinematics information of mechanical pose information and adaptively fuses the mechanical pose information and the video pose information together to generate the sequence of states of the corrected kinematics information for the robotic instrument. Additionally disclosed are methods for image guided surgery.

Claims

1. A non-transitory computer usable medium for use with a tool tracking system comprising:

(a) non-transitory computer readable code programmed to receive images of video frames from at least one camera;

(b) non-transitory computer readable code programmed to receive kinematics information related to robotic movement of a robotic instrument, determine mechanical pose information from the kinematics information, and synthesize model pose information of a computer aided design model of the robotic instrument using the mechanical pose information;

(c) non-transitory computer readable code programmed to determine video pose information of the robotic instrument using the synthesized model pose information as a pattern for pattern searching within the images; and

(d) non-transitory computer readable code programmed to provide a state-space model of a sequence of states of corrected kinematics information for accurate pose information of the robotic instrument, the state-space model to receive raw kinematics information of mechanical pose information and to adaptively fuse the mechanical pose information and the video pose information together to generate the sequence of states of the corrected kinematics information for the robotic instrument.

2. The computer usable medium of claim 1, wherein the synthesized model pose information of the model of the robotic instrument generated by the non-transitory computer readable code includes one or more markers of the robotic instrument forming a pattern.

3. The computer usable medium of claim 2, wherein the one or more markers include artificial markers consisting of a pattern of dots.

4. The computer usable medium of claim 2, wherein the one or more markers include natural markers represented by consisting of geometry information of the computer aided design model.

5. A non-transitory computer usable medium for use with a tool tracking system comprising:

(a) non-transitory computer readable code programmed to receive images of video frames from at least one camera, determine video pose information of a robotic instrument within the images, and estimate uncertainty in the determined video pose information in light of video information in response to view geometry statistics; and

(b) non-transitory computer readable code programmed to receive kinematics information related to robotic movement of the robotic instrument, determine mechanical pose information from the kinematics information, and fuse the mechanical pose information and the video pose information in a state-space model using a covariance matrix configured to compensate for the estimated uncertainty in the video pose information to generate estimated pose information for the robotic instrument.

6. The computer usable medium of claim 5, wherein the state-space model includes a dynamic model and an observation model respectively including dynamic noise and observation noise that respectively have dynamic and observation Gaussian distributions respectively characterized by dynamic and observation covariance matrices, wherein the observation covariance matrix includes a sub-matrix for vision;

and

the non-transitory computer readable code is programmed to configure the sub-matrix for vision to compensate for the estimated uncertainty of video information by modifying elements of the sub-matrix so as to adjust a standard deviation for the estimated pose information for the robotic instrument accordingly.

7. The computer usable medium of claim 5, wherein assuming an independent Gaussian noise model, the view geometry statistics are computed for one or more of digitization error/image resolution, feature/algorithm related image matching error, distance from object to camera, angle between object surface normal and line of sight, illumination, and specularity.