Synthetic representation of a surgical robot

• US 10,137,575 B2
• Filed: 06/21/2017
• Issued: 11/27/2018
• Est. Priority Date: 06/29/2006
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Itkowitz, Brandon D., DiMaio, Simon P., Halabe, Daniel J., Hasser, Christopher J., Hoffman, Brian D., Larkin, David Q., Mohr, Catherine J., Mohr, Paul W., Zhao, Tao, Zhao, Wenyi
• Primary Examiner(s): Downey, John R
• Application Number:
  US15/629,533
• Publication Number:
  US 20170282372A1
• Time in Patent Office: 524 Days
• Field of Search: 606 1, 606130
• CPC Class Codes: A61B 2034/2059 Mechanical position encoders
• CPC Class Codes: A61B 2034/2061 using shape-sensors, e.g. fiber shape sensors with Bragg gratings
• CPC Class Codes: A61B 2090/371 with simultaneous use of two cameras
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/25 User interfaces for surgical systems
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 90/36 Image-producing devices or illumination devices not otherwise provided for
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: A61B 90/37 Surgical systems with images on a monitor during operation
• CPC Class Codes: B25J 9/1666 Avoiding collision or forbidden zones
• CPC Class Codes: B25J 9/1671 characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: B25J 9/1692 Calibration of manipulator
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 19/4202 preparation of the programme medium using a drawing, a model
• CPC Class Codes: G05B 2219/36432 By putting some constraints on some DOF, move within limited volumes, areas, planes, limits motion in x, y or z planes, virtual reality constraints
• CPC Class Codes: G05B 2219/39083 Robot interference, between two robot arms
• CPC Class Codes: G05B 2219/39096 Self-collision, internal collison, collision between links of one robot
• CPC Class Codes: G05B 2219/39449 Pendant, pda displaying camera images overlayed with graphics, augmented reality
• CPC Class Codes: G05B 2219/40607 : Fixed camera to observe workspace, object, workpiece, global
• CPC Class Codes: G05B 2219/45117 : Medical, radio surgery manipulator
• CPC Class Codes: G05B 2219/45123 : Electrogoniometer, neuronavigator, medical robot used by surgeon to operate
• CPC Class Codes: G16H 20/40 : relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
• CPC Class Codes: G16H 40/67 : for remote operation

Abstract

A synthetic representation of a robot tool for display on a user interface of a robotic system. The synthetic representation may be used to show the position of a view volume of an image capture device with respect to the robot. The synthetic representation may also be used to find a tool that is outside of the field of view, to display range of motion limits for a tool, to remotely communicate information about the robot, and to detect collisions.

Claims

1. A robotic surgical system, comprising:

a robot including a linkage supporting at least one surgical tool;

an image capture device having a field of view encompassing the tool;

a kinematic component coupled to the robot so as to obtain kinematic position information associated with the linkage;

a display coupled to receive and display an image within the field of view of the image capture device; and

a first component coupling the display with the kinematic component so as to display information collocated with the tool represented in the displayed image, the position of the information being based upon;

linkage structure data regarding the linkage, and

the kinematic position information.

2. The system of claim 1, further comprising:

a stereoscopic image system coupled to the image capture device and the display to generate a three-dimensional view of the tool in the field of view of the image capture device; and

a calibrated camera model coupled to the first component and the display to provide a three-dimensional appearance of the display information matching the three-dimensional view of the tool.

3. The system of claim 1, wherein the information collocated with the tool comprises text.

4. The system of claim 1, further comprising:

an input operatively coupled to the robot for commanding the robot to move the tool;

wherein the kinematic component obtains the kinematic position information associated with the linkage during movement of the tool; and

wherein the first component causes the information to be displayed on the display so as to maintain collocation of information with the tool represented in the displayed image during the movement of the tool.

5. The system of claim 4,

wherein the wherein the first component causes the information to be displayed on the display so as to maintain collocation of information with the tool represented in the displayed image during the movement of the tool by maintaining the position of the information over the tool represented in the displayed image.

6. The system of claim 4,

wherein the wherein the first component causes the information to be displayed on the display so as to maintain collocation of information with the tool represented in the displayed image during the movement of the tool by maintaining the position of the information around or adjacent the tool represented in the displayed image.

7. A robotic surgical system, comprising:

a robotic arm assembly coupled to a tool having an end effector;

an image capture device disposed relative to the tool so as to capture video images of the end effector;

an input device operatively coupled to the robotic arm for commanding the robotic arm to move the tool;

a tool tracking component configured to track movement of the tool;

a display; and

a processor programmed to;

receive tool tracking information from the tool tracking component;

receive video information derived from video images of the end effector which have been captured by the image capture device;

receive tool related information; and

cause the tool related information and the video information to be displayed on the display so that the tool information is continuously collocated with the video information during movement of the tool by using the tool tracking information.

8. The robotic surgical system of claim 7,

wherein the tool related information is in the form of a text message.

9. A method for controlling movement of a tool, the method comprising:

receiving a command to move a tool having an end effector;

causing the tool to move in response to the command;

tracking movement of the tool;

receiving tool tracking information indicative of the movement of the tool from a tool tracking component;

receiving video information derived from video images of the end effector which have been captured by an image capture device;

receiving tool related information; and

causing the tool related information and the video information to be displayed on a display so that the tool information is continuously collocated with the video information during movement of the tool by using the tool tracking information.

10. The method of claim 9,

wherein the tool related information is in the form of a text message.