Rendering tool information as graphic overlays on displayed images of tools

• US 10,008,017 B2
• Filed: 02/15/2013
• Issued: 06/26/2018
• Est. Priority Date: 06/29/2006
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Itkowitz, Brandon D., Halabe, Daniel J., Zhao, Tao, DiMaio, Simon P., Hasser, Christopher J., Mohr, Catherine J., Mohr, Paul W., Larkin, David Q., Hoffman, Brian David, Zhao, Wenyi
• Primary Examiner(s): LUAN, SCOTT
• Application Number:
  US13/768,047
• Publication Number:
  US 20140055489A1
• Time in Patent Office: 1,957 Days
• CPC Class Codes: A61B 2034/2061 using shape-sensors, e.g. fiber shape sensors with Bragg gratings
• CPC Class Codes: A61B 2090/371 with simultaneous use of two cameras
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/25 User interfaces for surgical systems
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 90/36 Image-producing devices or illumination devices not otherwise provided for
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: B25J 9/1666 Avoiding collision or forbidden zones
• CPC Class Codes: B25J 9/1671 characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems
• CPC Class Codes: B25J 9/1692 Calibration of manipulator
• CPC Class Codes: G05B 19/4202 preparation of the programme medium using a drawing, a model
• CPC Class Codes: G05B 2219/36432 By putting some constraints on some DOF, move within limited volumes, areas, planes, limits motion in x, y or z planes, virtual reality constraints
• CPC Class Codes: G05B 2219/39083 Robot interference, between two robot arms
• CPC Class Codes: G05B 2219/39096 Self-collision, internal collison, collision between links of one robot
• CPC Class Codes: G05B 2219/39449 Pendant, pda displaying camera images overlayed with graphics, augmented reality
• CPC Class Codes: G05B 2219/40607 Fixed camera to observe workspace, object, workpiece, global
• CPC Class Codes: G05B 2219/45117 Medical, radio surgery manipulator
• CPC Class Codes: G05B 2219/45123 Electrogoniometer, neuronavigator, medical robot used by surgeon to operate
• CPC Class Codes: G06T 11/40 Filling a planar surface by adding surface attributes, e.g. colour or texture
• CPC Class Codes: G06T 11/60 : Editing figures and text; Combining figures or text
• CPC Class Codes: Y02P 90/02 : Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]

Abstract

An operator telerobotically controls tools to perform a procedure on an object at a work site while viewing real-time images of the work site on a display. Tool information is provided in the operator'"'"'s current gaze area on the display by rendering the tool information over the tool so as not to obscure objects being worked on at the time by the tool nor to require eyes of the user to refocus when looking at the tool information and the image of the tool on a stereo viewer.

Claims

1. A robotic system comprising:

a tool having a working end, the working end adapted to physically interact with an object;

a manipulator coupled to the tool for robotic manipulation of the tool;

a camera spatially disposed relative to the tool so that the camera captures an image of the working end of the tool from a perspective of an image reference frame;

a display coupled to the camera for displaying the image of the working end of the tool; and

a processor programmed to render information of the tool over the image of the working end of the tool which was captured by the camera and is being displayed on the display by;

determining a pose of the tool in the image reference frame;

registering the information of the tool with a designated place on the working end of the tool in the image reference frame by using the pose of the tool in the image reference frame; and

rendering the information of the tool over the designated place on the working end of the tool in the image of the working end of the tool being displayed on the display after registering the information of the tool with the designated place on the working end of the tool, so that the entire information of the tool appears as a decal that is rendered over the working end of the tool in the image of the working end of the tool being displayed on the display by bending the decal to conform in shape and be applied to an outer surface of the tool at the designated place.

2. The robotic system of claim 1, wherein the processor is programmed to render the information of the tool over the designated place on the working end of the tool in the image of the working end of the tool by rendering an indication that the tool is energized on the decal.

3. The robotic system of claim 2, wherein the processor is programmed to render the indication that the tool is energized by rendering an icon on the decal, wherein presence of the icon indicates the tool is energized.

4. The robotic system of claim 2, wherein the processor is programmed to render the indication that the tool is energized by rendering a color on the decal, wherein the color indicates the tool is energized.

5. The robotic system of claim 2, wherein the processor is programmed to render the indication that the tool is energized by rendering a different brightness level on the decal, wherein the brightness level indicates the tool is energized.

6. The robotic system of claim 1, wherein the working end of the tool includes an end effector, a wrist, and a distal end of a shaft, and the decal is rendered over one of the end effector, the wrist, and the distal end of the shaft.

7. The robotic system of claim 1,

wherein the processor is programmed to render the information of the tool over the designated place on the working end of the tool in the image of the working end of the tool being displayed on the display by rendering an indication of an identity of the manipulator on the decal.

8. The robotic system of claim 7;

wherein the processor is programmed to render the indication of the identity of the manipulator by rendering a manipulator identification code on the decal, wherein the manipulator identification code uniquely identifies the manipulator.

9. The robotic system of claim 1, further comprising:

an input device operatively coupled to the manipulator, the input device being manipulatable by an operator;

wherein the processor is programmed to render information of the tool over the designated place on the working end of the tool in the image of the working end of the tool being displayed on the display by rendering an indication of an identity of at least one of the input device and the operator on the decal.

10. The robotic system of claim 1, wherein the information of the tool indicates whether or not the tool is locked in position at the time.

11. The robotic system of claim 1, wherein the information of the tool includes at least one of static and dynamic information.

12. The robotic system of claim 1, wherein the processor is programmed to determine the pose of the tool relative to the image reference frame by:

determining the pose of the tool in a reference frame of the tool and translating the pose of the tool from the reference frame of the tool to the pose of the tool in the image reference frame.

13. The robotic system of claim 12, wherein the processor is programmed to determine the pose of the tool relative to the reference frame of the tool by using kinematics information of the manipulator.

14. The robotic system of claim 1, wherein the display is a stereo viewer and the image of the tool being shown on the display is a three-dimensional image of the tool.

15. A method, implemented in a robotic system, the robotic system comprising:

a tool having a working end;

the working end adapted to physically interact with an object;

a manipulator coupled to the tool for robotic manipulation of the tool;

a camera spatially disposed relative to the tool so that the camera captures an image of the working end of the tool from a perspective of an image reference frame; and

a display coupled to the camera for displaying the image of the working end of the tool;

the method comprising;

rendering information of the tool over the image of the working end of the tool which was captured by the camera and is being displayed on the display by;

determining a pose of the tool in the image reference frame;

registering the information of the tool with a designated place on the working end of the tool in the image reference frame by using the pose of the tool in the image reference frame; and

rendering the information of the tool over the designated place on the working end of the tool in the image of the working end of the tool being displayed on the display after registering the information of the tool with the designated place on the working end of the tool, so that the entire information of the tool appears as a decal that is rendered over the working end of the tool in the image of the working end of the tool being displayed on the display by bending the decal to conform in shape and be applied to an outer shape of the tool at the designated place.

16. The method of claim 15, wherein the rendering of the information of the tool over the designated place on the working end of the tool in the image of the working end of the tool being displayed on the display comprises:

rendering an indication that the tool is energized on the decal.

17. The method of claim 16, wherein the rendering of the indication that the tool is energized comprises:

rendering an icon on the decal, wherein presence of the icon indicates the tool is energized.

18. The method of claim 16, wherein the rendering of the indication that the tool is energized comprises:

rendering a color on the decal, wherein the color indicates the tool is energized.

19. The method of claim 16, wherein the rendering of the indication that the tool is energized comprises:

rendering a different brightness level on the decal, wherein the brightness level indicates the tool is energized.

20. The method of claim 15, wherein the working end of the tool includes an end effector, a wrist, and a distal end of a shaft, and the decal is rendered over one of the end effector, the wrist, and the distal end of the shaft.

21. The method of claim 15, wherein the rendering of the information of the tool over the designated place on the working end of the tool in the image of the working end of the tool being displayed on the display comprises:

rendering an indication of an identity of the manipulator on the decal.

22. The method of claim 21, wherein the rendering of the indication of the identity of the manipulator comprises:

rendering a manipulator identification code on the decal, wherein the manipulator identification code uniquely identities the manipulator.

23. The method of claim 15, wherein the rendering of the information of the tool over the designated place on the working end of the tool in the image of the working end of the tool being displayed on the display comprises:

rendering an indication of an identity of at least one of an input device and an operator on the decal, wherein the input device is operatively coupled to the manipulator and the input device is manipulatable by the operator.

24. The method of claim 15, wherein the rendering of the information of the tool over the designated place on the working end of the tool in the image of the working end of the tool being displayed on the display comprises:

rendering an indication on the decal of whether or not the tool is locked in position at the time.

25. The method of claim 15, wherein the information of the tool includes at least one of static and dynamic information.

26. The method of claim 15, wherein the determining of the pose of the tool in the image reference frame comprises:

determining the pose of the tool relative to a reference frame of the tool; and

translating the pose of the tool from the reference frame of the tool to the pose of the tool in the image reference frame.

27. The method of claim 26, wherein the determining of the pose of the tool relative to the reference frame of the tool employs kinematics information of the manipulator.

28. The method of claim 15, wherein the display is a stereo viewer and the image of the tool being shown on the display is a three-dimensional image of the tool.