Visual force feedback in a minimally invasive surgical procedure

• US 8,594,841 B2
• Filed: 04/22/2009
• Issued: 11/26/2013
• Est. Priority Date: 12/31/2008
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Zhao, Wenyi, Zhao, Tao, Larkin, David Q.
• Primary Examiner(s): Marc, McDieunel
• Application Number:
  US12/428,142
• Publication Number:
  US 20100169815A1
• Time in Patent Office: 1,679 Days
• Field of Search: 700/245, 700/259, 700/260, 606/1, 606/42, 606/139, 606/143, 606/164, 606/174, 606/205, 606/208, 600/102, 600/104, 318/566, 318/568.11, 813/29, 813/31, 901/41, 435/6.11
• US Class Current: 700/245
• CPC Class Codes: A61B 2034/102 Modelling of surgical devices, implants or prosthesis
• CPC Class Codes: A61B 2090/065 for measuring contact or contact pressure
• CPC Class Codes: A61B 2090/365 augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/35 for telesurgery
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 34/76 Manipulators having means for providing feel, e.g. force or tactile feedback
• CPC Class Codes: B25J 9/1633 compliant, force, torque control, e.g. combined with position control
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: Y10S 901/47 Optical

Abstract

Methods of and a system for providing a visual representation of force information in a robotic surgical system. A real position of a surgical end effector is determined. A projected position of the surgical end effector if no force were applied against the end effector is also determined. Images representing the real and projected positions are output superimposed on a display. The offset between the two images provides a visual indication of a force applied to the end effector or to the kinematic chain that supports the end effector. In addition, tissue deformation information is determined and displayed.

Claims

1. A method of providing a visual representation of force information in a robotic surgical system, the method comprising:

displaying a first image of a tool on a display screen, wherein the first image shows an actual position of the tool; and

displaying a second image of the tool on the display screen such that an offset between the displayed first and second images of the tool provides a visual indication of a force being applied against the tool.

2. The method of claim 1, wherein at least some of the force is applied to the tool as a result of movement of the tool between a first position and a second position, wherein other of the force is applied to a linkage supporting the tool, and wherein the offset between the first and second images of the tool disregards at least some of the other force.

3. The method of claim 1, wherein the second image represents kinematic position information derived from joint states of a linkage supporting the tool.

4. The method of claim 1, wherein the first image comprises an image of the tool which has been captured by an image capturing device.

5. The method of claim 1, wherein the display screen is a viewer of a surgeon console of the robotic surgical system.

6. The method of claim 1, wherein the second image comprises a synthetic representation of the tool showing a commanded position and orientation of the tool.

7. The method of claim 6, wherein the synthetic representation comprises a computer model of the tool.

8. The method of claim 7, wherein the computer model comprises a three-dimensional computer model of the tool.

9. The method of claim 1, wherein the force being applied against the tool is determinable from force information provided by force sensors.

10. The method of claim 9, wherein the force sensors are proximate to an end effector of the tool.

11. The method of claim 9, wherein the tool is supported by a linkage having joints, and wherein the force sensors are proximate to the joints.

12. The method of claim 1, wherein the tool is supported by a linkage having joints, and further comprising:

determining by a controller joint errors from positions of the joints with respect to a commanded position of the joints;

correcting by the controller the joints toward the commanded position; and

wherein the force is determinable from information derived from the correcting of the joints.

13. A system of providing a visual representation of force information in a robotic system, the system comprising:

an image input for capturing an image of an end effector while a force is applied to the end effector;

a display coupled to the image input so as to present an actual position of the end effector under the applied force; and

a processor coupled to the display, the processor generating a second image representing a projected position of the end effector offset from the first position so as to visually indicate the force, the processor transmitting the second image to the display so that the second image is superimposed with the first image.

14. A method of providing a visual representation of tissue deformation in a robotic surgical system, the method comprising:

sensing a force which is being applied against a tissue so as to cause deformation of the tissue;

determining by a processor device information indicative of the deformation of the tissue by using the sensed force; and

displaying an image of the tissue and the information indicative of the deformation of the tissue superimposed over the image of the tissue on a display screen.

15. The method of claim 14, wherein the force is determinable from force information provided by force sensors.

16. The method of claim 15, wherein the force sensors are proximate to an end effector applying the force against the tissue.

17. The method of claim 15, wherein an end effector applying the force against the tissue is supported by a linkage having joints, and wherein the force sensors are proximate to the joints.

18. The method of claim 14, wherein an end effector applying the force against the tissue is supported by a linkage having joints, and further comprising:

determining by a controller joint errors from positions of the joints with respect to a commanded position of the joints;

correcting by a controller the joints toward the commanded position; and

wherein the force is determinable from information derived from the correcting of the joints.

19. The method of claim 14, wherein the force is determinable from a displacement of an end effector applying the force against the tissue.

20. The method of claim 14, wherein the information indicative of the deformation comprises a color superimposed over the image of the tissue.

21. The method of claim 20, wherein the color changes based upon the amount of force being applied to the tissue.

22. The method of claim 20, wherein the color is varied based upon how far the tissue is from contact with an end effector.

23. The method of claim 14, further comprising:

displaying an ultrasound image of the tissue on the display screen as the force is being applied to the tissue.