TOOL TRACKING SYSTEMS AND METHODS FOR IMAGE GUIDED SURGERY

• US 20090088634A1
• Filed: 09/30/2007
• Published: 04/02/2009
• Est. Priority Date: 09/30/2007
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Incorporated (Intuitive Surgical Holdings, LLC)
• Inventors: Nowlin, William C., Hasser, Christopher J., Zhao, Wenyi, Hoffman, Brian D.
• Granted Patent: US 8,073,528 B2
• Time in Patent Office: Days
• US Class Current: 600/427
• CPC Class Codes: A61B 1/00193 adapted for stereoscopic vision
• CPC Class Codes: A61B 1/00194 adapted for three-dimensional imaging
• CPC Class Codes: A61B 1/018 for receiving instruments
• CPC Class Codes: A61B 1/04 combined with photographic or television appliances
• CPC Class Codes: A61B 1/3132 for laparoscopy
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2090/061 for measuring dimensions, e.g. length
• CPC Class Codes: A61B 2090/364 Correlation of different images or relation of image positions in respect to the body
• CPC Class Codes: A61B 2090/3983 Reference marker arrangements for use with image guided surgery
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 5/061 Determining position of a probe within the body employing means separate from the probe, e.g. sensing internal probe position employing impedance electrodes on the surface of the body A61B1/00158 takes precedence
• CPC Class Codes: A61B 5/1076 for measuring dimensions inside body cavities, e.g. using catheters A61B3/1005 takes precedence
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: B25J 13/089 Determining the position of the robot with reference to its environment
• CPC Class Codes: B25J 15/0019 End effectors other than grippers
• CPC Class Codes: B25J 9/1656 characterised by programming, planning systems for manipulators
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10068 : Endoscopic image
• CPC Class Codes: G06T 2207/30004 : Biomedical image processing
• CPC Class Codes: G06T 2207/30241 : Trajectory
• CPC Class Codes: G06T 7/246 : using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/255 : Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 2201/034 : of medical instruments
• CPC Class Codes: G09B 23/28 : for medicine
• CPC Class Codes: G16H 20/40 : relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
• CPC Class Codes: G16H 30/40 : for processing medical images, e.g. editing
• CPC Class Codes: G16H 40/67 : for remote operation

Abstract

In one embodiment of the invention, a tool tracking system is disclosed including a computer usable medium having computer readable program code to receive images of video frames from at least one camera and to perform image matching of a robotic instrument to determine video pose information of the robotic instrument within the images. The tool tracking system further includes computer readable program code to provide a state-space model of a sequence of states of corrected kinematics information for accurate pose information of the robotic instrument. The state-space model receives raw kinematics information of mechanical pose information and adaptively fuses the mechanical pose information and the video pose information together to generate the sequence of states of the corrected kinematics information for the robotic instrument. Additionally disclosed are methods for image guided surgery.

Claims

1. A tool tracking system comprising:

a computer usable medium having

computer readable program code to receive images of video frames from at least one camera and to perform image matching of a robotic instrument to determine video pose information of the robotic instrument within the images; and

computer readable program code to provide a state-space model of a sequence of states of corrected kinematics information for accurate pose information of the robotic instrument, the state-space model to receive raw kinematics information of mechanical pose information and to adaptively fuse the mechanical pose information and the video pose information together to generate the sequence of states of the corrected kinematics information for the robotic instrument.

2. The tool tracking system of claim 1, wherein

the computer readable program code to perform image matching of the robotic instrument to determine video pose information of the robotic instrument within the images by performing image matching between one or more of

a first video image of the robotic instrument at a first time and a second video image of the robotic instrument at a second time,

a third video image of the robotic instrument and a synthesized image of the robotic instrument, and

a first viewpoint video image of the robotic instrument from a first camera and a second viewpoint video image of the robotic instrument from a second camera spaced apart from the first camera.

3. The tool tracking system of claim 1, wherein

the computer usable medium further has

computer readable program code to synthesize model pose information of a computer aided design model of a robotic instrument.

4. The tool tracking system of claim 3, wherein

the synthesized model pose information of the model of the robotic instrument generated by the computer readable program code is one or more markers of the robotic instrument forming a pattern;

and the computer readable program code to match the pattern of the one or more markers of the robotic instrument to images of the robotic instrument to determine video pose information.

5. The tool tracking system of claim 4, wherein

the one or more markers are artificial markers consisting of a pattern of dots.

6. The tool tracking system of claim 4, wherein

the one or more markers are natural markers consisting of geometry information of the computer aided design model.

7. The tool tracking system of claim 1, wherein

the state-space model includes a state model and an observation model, and

the computer readable program code to generate the states of the state model to inter-relate a plurality of states associated with the pose of the robotic instrument at different time instances and to relate the state model and the observation model by relating the plurality of states with respective observations of the robotic tool at respective time instances.

8. The tool tracking system of claim 1, wherein

the state-space model includes a plurality of states, and

the computer usable medium further has

computer readable program code to generate states of the state space model when information is uncertain if at least one state of the state-space model is certain.

9. The tool tracking system of claim 8, wherein

the image information for a state is unavailable, and

the computer usable medium further has

computer readable program code to predict the pose of the robotic tool in response to prior states of the state space model.

10. The tool tracking system of claim 1, wherein

the computer usable medium further has

computer readable program code to estimate uncertainty of video information in response to view geometry statistics, and

wherein the state-space model to adaptively fuse the mechanical pose information and the video pose information together in response to the estimated uncertainty of video information.

11. The tool tracking system of claim 10, wherein

the estimated uncertainty of video information forms a covariance matrix that is used by the state space model to adaptively fuse the mechanical pose information and the video pose information together.

12. The tool tracking system of claim 10, wherein

assuming an independent Gaussian noise model, the view geometry statistics are computed for one or more of digitization error/image resolution, feature/algorithm related image matching error, distance from object to camera, angle between object surface normal and line of sight, illumination, and specularity.

13. A method of image guided surgery, the method comprising:

during surgery capturing a sequence of images of a surgical site of the body and one or more robotic instruments; and

adaptively fusing kinematics information and image information of the one or more robotic instruments together to accurately track tool pose in the surgical site.

14. The method of claim 13, further comprising:

automatically guiding the one or more robotic instruments within the surgical site.

15. The method of claim 14, wherein

a camera is automatically guided to maintain the one or more robotic instruments within the field of view of the camera.

16. The method of claim 14, wherein

a robotic instrument is automatically guided to move it to a point on tissue within the surgical site.

17. The method of claim 13, further comprising:

touching surface features in the surgical site to determine depths of surface features in the surgical site to form a depth map.

18. The method of claim 17, further comprising:

prior to surgery, scanning a portion of a body to obtain tissue images;

forming a three dimensional image of an organ in response to the tissue images; and

overlaying a portion of the three dimensional image of the organ onto the depth map of the tissue surface to visually display an outline of the organ to guide a surgeon into and around the organ within the surgical site.

19. The method of claim 13, further comprising:

capturing ultrasound images of a portion of the surgical site with respect to a robotic instrument;

translating the ultrasound images with respect to the robotic instrument to a camera coordinate system;

overlaying the ultrasound images of the portion of the surgical site onto the video images of the surgical site;

wherein the ultrasound images are two dimensional ultrasound images or three dimensional ultrasound images; and

wherein the three dimensional ultrasound images may be formed from the two dimensional ultrasound images by rotating an ultrasound transducer over tissue in the surgical site.

20. The method of claim 13, further comprising:

touching one or more robotic instruments at points of interest on the tissue surface in the surgical site to overlay virtual marks onto the images of the tissue surface

21. The method of claim 20, further comprising:

displaying the virtual marks overlaid onto the images of the tissue surface on an external display to illustrate areas of interest to student surgeons.

22. The method of claim 20, wherein

a first tracked tool touches points of interest on the tissue surface to overlay the virtual marks onto the images of the tissue surface;

and

a second tool differing from the first is used to perform the surgical procedure.

23. A method of image guided surgery, the method comprising:

touching tissue in a surgical site with a first tool tip of a first robotic tool;

touching tissue in the surgical site with a second tool tip of a second robotic tool;

capturing a sequence of images of the surgical site including the first robotic tool and the second robotic tool;

adaptively fusing kinematics information and image information of the first robotic tool together to accurately determine a first tool tip location;

adaptively fusing kinematics information and image information of the second robotic tool together to accurately determine a second tool tip location; and

comparing the first tool tip location and the second tool tip location to determine a distance between them.

24. The method of claim 23, wherein

the first tool tip touches a first external surface of a tumor;

the second tool tip touches a second external surface of the tumor; and

the comparing determines a diameter of the tumor.

25. The method of claim 23, wherein

the first tool tip touches a first external surface of an organ;

the second tool tip touches a second external surface of the organ; and

the comparing determines a diameter of the organ.