Synthetic representation of a surgical robot

• US 9,789,608 B2
• Filed: 03/31/2009
• Issued: 10/17/2017
• Est. Priority Date: 06/29/2006
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Itkowitz, Brandon D., Halabe, Daniel J., Zhao, Tao, Dimaio, Simon, Hasser, Christopher J., Mohr, Catherine J., Mohr, Paul W., Larkin, David Q., Zhao, Wenyi, Hoffman, Brian D.
• Primary Examiner(s): Downey, John R
• Application Number:
  US12/415,354
• Publication Number:
  US 20090192524A1
• Time in Patent Office: 3,122 Days
• Field of Search: 606 1, 600424, 600426, 345619, 345629-635, 700245, 382128
• CPC Class Codes: A61B 2034/2059 Mechanical position encoders
• CPC Class Codes: A61B 2034/2061 using shape-sensors, e.g. fiber shape sensors with Bragg gratings
• CPC Class Codes: A61B 2090/371 with simultaneous use of two cameras
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/25 User interfaces for surgical systems
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 90/36 Image-producing devices or illumination devices not otherwise provided for
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: A61B 90/37 Surgical systems with images on a monitor during operation
• CPC Class Codes: B25J 9/1666 Avoiding collision or forbidden zones
• CPC Class Codes: B25J 9/1671 characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: B25J 9/1692 Calibration of manipulator
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 19/4202 preparation of the programme medium using a drawing, a model
• CPC Class Codes: G05B 2219/36432 By putting some constraints on some DOF, move within limited volumes, areas, planes, limits motion in x, y or z planes, virtual reality constraints
• CPC Class Codes: G05B 2219/39083 Robot interference, between two robot arms
• CPC Class Codes: G05B 2219/39096 Self-collision, internal collison, collision between links of one robot
• CPC Class Codes: G05B 2219/39449 Pendant, pda displaying camera images overlayed with graphics, augmented reality
• CPC Class Codes: G05B 2219/40607 : Fixed camera to observe workspace, object, workpiece, global
• CPC Class Codes: G05B 2219/45117 : Medical, radio surgery manipulator
• CPC Class Codes: G05B 2219/45123 : Electrogoniometer, neuronavigator, medical robot used by surgeon to operate
• CPC Class Codes: G16H 20/40 : relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
• CPC Class Codes: G16H 40/67 : for remote operation

Abstract

A synthetic representation of a robot tool for display on a user interface of a robotic system. The synthetic representation may be used to show the position of a view volume of an image capture device with respect to the robot. The synthetic representation may also be used to find a tool that is outside of the field of view, to display range of motion limits for a tool, to remotely communicate information about the robot, and to detect collisions.

Claims

1. A medical system comprising:

first and second tools, the first tool including a first elongated shaft and a first end effector, the first end effector coupled to a distal end of the first elongated shaft, the second tool including a second elongated shaft and a second end effector, the second end effector coupled to a distal end of the second elongated shaft;

an image capture device;

first, second, and third robotic arm assemblies, the first robotic arm assembly detachably coupled to a proximal end of the first elongated shaft of the first tool, the second robotic arm assembly detachably coupled to a proximal end of the second elongated shaft of the second tool, and the third robotic arm assembly supporting the image capture device;

a viewer displaying images captured by the image capture device; and

a modeling component programmed to generate a synthetic image of the first robotic arm assembly, the first tool, the second robotic arm assembly, the second tool, the third robotic arm assembly, the image capture device, and a view volume projecting out of an image capturing end of the image capture device so as to provide a visual indication of a field of view of the image capture device, using kinematic data for the first, second, and third robotic arm assemblies, and to provide the synthetic image to the viewer for displaying on the viewer so that proximity with respect to each other of the first, second, and third robotic arm assemblies is viewable on the viewer.

2. The system of claim 1, further comprising a display;

wherein the modeling component is programmed to cause the synthetic image to be displayed on the display.

3. The system of claim 1, wherein the modeling component is programmed to maintain information about relative locations of the first and second robotic arm assemblies, to predict collisions between the first and second robotic arm assemblies based upon their relative locations, and to generate a first warning signal upon a distance between the first and second robotic arm assemblies reaching a first proximity threshold;

and wherein the first warning signal results in an audible warning.

4. The system of claim 1, wherein the modeling component is programmed to maintain information about relative locations of the first and second tools, to predict collisions between the first and second tools based upon their relative locations, and to generate a second warning signal upon a distance between the first and second tools reaching a second proximity threshold;

and wherein the second warning signal results in a visual warning displayed on the viewer.

5. The system of claim 4, wherein the visual warning comprises changing a color of only an image of at least one of the first and second tools in the image that has been captured by the image capture device and is being displayed on the viewer.

6. The system of claim 4, wherein the visual warning comprises a text message adjacent an image of at least one of the first and second tools in the image that has been captured by the image capture device and is being displayed on the viewer.

7. The system of claim 5, wherein changing the color of the image of at least one of the first and second tools comprises changing the color to yellow to indicate an impending collision between the first and second removable tools.

8. The system of claim 5, wherein changing the color of the image of at least one of the first and second tools comprises changing the color to red to indicate an actual collision between the first and second tools.

9. The system of claim 1, wherein the modeling component is programmed to maintain information about relative locations of the first and second robotic arm assemblies, to predict collisions between the first and second robotic arm assemblies based upon their relative locations, and to generate a first warning signal upon a distance between the first and second robotic arm assemblies reaching a first proximity threshold;

and wherein the first warning signal results in a visual warning.

10. The system of claim 1, wherein the modeling component is programmed to cause the image that has been captured by the image capture device and the synthetic image to be displayed in separate window areas on the viewer.

11. The system of claim 1, further comprising a toggle, wherein the modeling component is programmed to provide only one of the image that has been captured by the image capture device and the synthetic image to the viewer at a time according to a current state of the toggle for displaying on the viewer.

12. The system of claim 1, wherein the modeling component is programmed to generate the synthetic image so as to include a synthetic image of a virtual boundary indicating a range of motion limit for the first end effector of the first tool, and to provide the generated synthetic image to the viewer for displaying on the viewer so that proximity of the first end effector of the first tool to the range of motion limit for the first end effector of the first tool is viewable on the viewer.

13. The system of claim 1, wherein the modeling component is programmed to provide the generated synthetic image to the viewer for displaying on the viewer so that the view volume of the image capture device is viewable on the viewer from a different perspective than that of the image capture device.