SYNTHETIC REPRESENTATION OF A SURGICAL ROBOT

• US 20090192524A1
• Filed: 03/31/2009
• Published: 07/30/2009
• Est. Priority Date: 06/29/2006
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Incorporated (Intuitive Surgical Holdings, LLC)
• Inventors: Larkin, David Q., Zhao, Tao, Mohr, Paul W., Mohr, Catherine J., Itkowitz, Brandon D., Halabe, Daniel J., Hasser, Christopher J., Dimaio, Simon, Zhao, Wenyi, Hoffman, Brian D.
• Granted Patent: US 9,789,608 B2
• Time in Patent Office: Days
• US Class Current: 606/130
• CPC Class Codes: A61B 2034/2059 Mechanical position encoders
• CPC Class Codes: A61B 2034/2061 using shape-sensors, e.g. fiber shape sensors with Bragg gratings
• CPC Class Codes: A61B 2090/371 with simultaneous use of two cameras
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/25 User interfaces for surgical systems
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 90/36 Image-producing devices or illumination devices not otherwise provided for
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: A61B 90/37 Surgical systems with images on a monitor during operation
• CPC Class Codes: B25J 9/1666 Avoiding collision or forbidden zones
• CPC Class Codes: B25J 9/1671 characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: B25J 9/1692 Calibration of manipulator
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 19/4202 preparation of the programme medium using a drawing, a model
• CPC Class Codes: G05B 2219/36432 By putting some constraints on some DOF, move within limited volumes, areas, planes, limits motion in x, y or z planes, virtual reality constraints
• CPC Class Codes: G05B 2219/39083 Robot interference, between two robot arms
• CPC Class Codes: G05B 2219/39096 Self-collision, internal collison, collision between links of one robot
• CPC Class Codes: G05B 2219/39449 Pendant, pda displaying camera images overlayed with graphics, augmented reality
• CPC Class Codes: G05B 2219/40607 : Fixed camera to observe workspace, object, workpiece, global
• CPC Class Codes: G05B 2219/45117 : Medical, radio surgery manipulator
• CPC Class Codes: G05B 2219/45123 : Electrogoniometer, neuronavigator, medical robot used by surgeon to operate
• CPC Class Codes: G16H 20/40 : relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
• CPC Class Codes: G16H 40/67 : for remote operation

Abstract

A synthetic representation of a robot tool for display on a user interface of a robotic system. The synthetic representation may be used to show the position of a view volume of an image capture device with respect to the robot. The synthetic representation may also be used to find a tool that is outside of the field of view, to display range of motion limits for a tool, to remotely communicate information about the robot, and to detect collisions.

Claims

1. A robotic surgical system, comprising:

a robot including a linkage supporting at least one surgical tool;

a kinematic component coupled to the robot so as to obtain kinematic position information associated with the linkage;

a display; and

a first component coupling the display with the kinematic component so as to display a synthetic representation of the robot that includes a graphical three-dimensional model of at least a portion of the linkage based upon;

linkage structure data regarding the linkage, and

the kinematic position information.

2. The system of claim 1, further comprising:

an image capture device having a field of view encompassing the tool; and

a surgeon console comprising a display and an input disposed for receiving a commend to move the tool;

wherein the display is coupled to receive an image within the field of view of the image capture device.

3. The system of claim 2, further comprising:

a second display;

wherein the second display is remote of the surgeon console; and

wherein the second display displays the synthetic representation of the robot.

4. The system of claim 3, further comprising a second component coupling the second display with the kinematic component so as to display on the second display a synthetic representation of the robot that includes a graphical representation of at least a portion of the linkage based upon:

the linkage structure data regarding the linkage, and the kinematic position information.

5. The system of claim 1, wherein the synthetic representation comprises a model of the tool.

6. The system of claim 5, wherein the model comprises a three-dimensional model of the tool.

7. The system of claim 1, further comprising:

an image capture device that captures an image of the tool within a field of view of the image capture device; and

a second component coupling the image capture device to the display so as to display the image from the image capture device on the display.

8. The system of claim 7, wherein the synthetic representation is displayed to appear on the display outside the displayed image from the capture device.

9. A robotic surgical system, comprising:

a robot including a tool for performing surgery;

an image capture device that captures a field of view encompassing the tool;

data for providing a synthetic representation of at least a portion of the robot, the portion including the image capture device;

a display; and

a first component coupling the data to the display so as to display the synthetic representation of the robot including a three-dimensional representation of the field of view of the image capture device.

10. The system of claim 9, wherein the display comprises a viewer of a surgeon console of the robotic surgical system.

11. The system of claim 9, wherein the synthetic representation comprises a model of the tool.

12. The system of claim 11, wherein the model comprises a three-dimensional model of the tool.

13. A robotic surgical system, comprising:

a robot including a first linkage supporting a first surgical tool and a second linkage supporting a second surgical tool or an image capture device;

a kinematic component coupled to the first linkage and the second linkage so as to obtain joint state information regarding the linkages;

data regarding structures of the first linkage, the second linkage, the first surgical tool, and the second surgical tool or the image capture device; and

a collision detection component coupled to the data and to the kinematic component so as to generate a warning signal about a potential or actual collision of (i) the first linkage or first surgical tool with (ii) the second linkage or the second surgical tool or the image capture device.

14. The system of claim 13, wherein the collision detection component is configured to generate a warning signal as a result of a proximity of the first linkage or the first surgical tool with the second linkage or the second surgical tool or the image capture device, as determined by the data and the kinematic component.

15. The system of claim 13, further comprising a display;

wherein the collision detection component is configured to transmit the signal to the display so as to display information about the potential or actual collision.

16. The system of claim 15, wherein the information about the potential or actual collision comprises a synthetic representation of the potential or actual collision between (i) the first linkage or the first surgical tool and (ii) the second linkage or the second surgical tool or the image capture device.

17. The system of claim 15, wherein the information about the potential or actual collision comprises information about avoiding or correcting a collision.

18. The system of claim 15, wherein the information about the potential or actual collision comprises highlighting a location of the potential or actual collision.

19. The system of claim 18, wherein the information about the potential or actual collision comprises a text message adjacent the location.

20. A method of controlling a position of a tool in a robotic system, the method comprising:

displaying a first image on a display, the first image comprising a video feed of a tool or linkage of a robot within a field of view of an image capture device;

displaying a second image on the display, the second image representing a three-dimensional model of the tool or linkage, wherein the tool or linkage represented in the second image is aligned with the tool or linkage in the first image; and

receiving a command from an input device moved with reference to the first and second images on the display so as to control movement of the tool or linkage.

21. A method of providing a range of motion indication for a tool of a robotic system, the method comprising:

displaying a first image representing a position of the tool; and

superimposing on the first image a second image representing a limit of motion of the tool.

22. In a robotic system, a method comprising:

maintaining information about a position of a tool of a robotic system; and

generating a signal as a result of the tool being within a threshold distance from a limit of motion of the tool.

23. A robotic surgical system, comprising:

a robot including a linkage supporting at least one surgical tool;

an image capture device having a field of view encompassing the tool;

a kinematic component coupled to the robot so as to obtain kinematic position information associated with the linkage;

a display coupled to receive and display an image within the field of view of the image capture device; and

a first component coupling the display with the kinematic component so as to display information collocated with the tool represented in the displayed image, the position of the information being based upon;

linkage structure data regarding the linkage, and

the kinematic position information.

24. The system of claim 23, further comprising:

a stereoscopic image system coupled to the image capture device and the display to generate a three-dimensional view of the tool in the field of view of the image capture device; and

a calibrated camera model coupled to the first component and the display to provide a three-dimensional appearance of the display information matching the three-dimensional view of the tool.

25. The system of claim 23, wherein the information collocated with the tool comprises text.

26. In a robotic system, a method comprising:

displaying in a first display a first image comprising a video feed of a tool supported by a robot within a field of view of an image capture device; and

displaying in the first display a synthetic three-dimensional representation of at least a portion of the robot including the tool.

27. The system of claim 26, wherein the three-dimensional representation is displayed to appear outside the displayed field of view of the image capture device.

28. In a robotic system, a method comprising:

displaying a first image comprising a video feed of a tool supported by a robot within a field of view of an image capture device, wherein the first image comprises a first portion of the robot; and

displaying a synthetic three-dimensional representation of at least a portion of the robot including the tool, wherein the synthetic three-dimensional representation comprises a second portion of the robot that is larger than the first portion.

29. In a robotic system, a method comprising:

displaying a first image comprising a video feed of a tool supported by a robot within a field of view of an image capture device, the first image comprising a first portion of the robot as viewed from a first direction; and

displaying a synthetic three-dimensional representation of at least a portion of the robot including the tool, wherein the displayed synthetic three-dimensional representation appears as viewed from a second direction, different from the first direction.

30. The system of claim 29, further comprising changing the display of the three-dimensional representation to appear as viewed from a third direction, different from the first and the second directions.

31. The system of claim 29, further comprising changing the display of the three-dimensional representation to appear as viewed from the first direction.

32. A robotic surgical system, comprising:

a robot including a linkage supporting at least one tool or an image capture device;

a kinematic component coupled to the linkage so as to obtain kinematic position information associated with the linkage;

first data regarding structures of the linkage and said at least one tool or an image capture device;

patient data regarding a position of a patient; and

a collision detection component coupled to receive the first data, coupled to receive the patient data, and coupled to the kinematic component so as to generate a warning signal about a potential or actual collision of the linkage with the patient, or of said at least one tool or an image capture device with the patient.

33. The system of claim 32, further comprising a display;

wherein the collision detection component is configured to transmit the signal to the display so as to display information about the potential or actual collision.

34. The system of claim 33, wherein the information about the potential or actual collision comprises a synthetic representation of the potential or actual collision between the linkage or said at least one tool with the patient.

35. The system of claim 34, wherein the information about the potential or actual collision comprises information about avoiding or correcting a collision.

36. A method of teaching tool movements for a tool in a robotic system, the method comprising:

displaying a first image on a display, the first image comprising a video feed of a tool or linkage of a robot within a field of view of an image capture device;

displaying a second image on the display, the second image representing a model of the tool or linkage;

moving the second image in response to an input from a first controller; and

moving the tool in response to an input from a second controller.

37. A surgical robotic system comprising:

an image capture device having a field of view in which a real time video image is captured;

a linkage that supports a surgical tool;

a display; and

a modeling component;

wherein the modeling component generates a three-dimensional rendering of the field of view and generates a three-dimensional rendering of at least a portion of the surgical tool outside the field of view;

wherein the display simultaneously displays the rendering of the field of view, the rendering of at least the portion of the surgical tool outside the field of view, and the real time video image;

wherein the rendering of at least the portion of the surgical tool is displayed to appear located with reference to the rendering of the field of view in substantially the same location that the surgical tool has with reference to the field of view of the image capture device; and

wherein the real time video image is displayed to appear located with reference to the rendering of the field of view in substantially the same location that the real time video image is captured within the field of view of the image capture device.

38. The surgical robotic system of claim 37:

wherein the displayed location of the real time video image with reference to the rendering of the field of view remains substantially constant as a displayed view angle of the rendering of the field of view is changed in the display.