Rendering tool information as graphic overlays on displayed images of tools

• US 10,282,881 B2
• Filed: 04/05/2018
• Issued: 05/07/2019
• Est. Priority Date: 03/31/2009
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Itkowitz, Brandon D., DiMaio, Simon P., Halabe, Daniel J., Hasser, Christopher J., Hoffman, Brian D., Larkin, David Q., Mohr, Catherine J., Mohr, Paul W., Zhao, Tao, Zhao, Wenyi
• Primary Examiner(s): Luan, Scott T
• Application Number:
  US15/946,408
• Publication Number:
  US 20180225855A1
• Time in Patent Office: 397 Days
• CPC Class Codes: A61B 2034/2061 using shape-sensors, e.g. fiber shape sensors with Bragg gratings
• CPC Class Codes: A61B 2090/371 with simultaneous use of two cameras
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/25 User interfaces for surgical systems
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 90/36 Image-producing devices or illumination devices not otherwise provided for
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: B25J 9/1666 Avoiding collision or forbidden zones
• CPC Class Codes: B25J 9/1671 characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems
• CPC Class Codes: B25J 9/1692 Calibration of manipulator
• CPC Class Codes: G05B 19/4202 preparation of the programme medium using a drawing, a model
• CPC Class Codes: G05B 2219/36432 By putting some constraints on some DOF, move within limited volumes, areas, planes, limits motion in x, y or z planes, virtual reality constraints
• CPC Class Codes: G05B 2219/39083 Robot interference, between two robot arms
• CPC Class Codes: G05B 2219/39096 Self-collision, internal collison, collision between links of one robot
• CPC Class Codes: G05B 2219/39449 Pendant, pda displaying camera images overlayed with graphics, augmented reality
• CPC Class Codes: G05B 2219/40607 Fixed camera to observe workspace, object, workpiece, global
• CPC Class Codes: G05B 2219/45117 Medical, radio surgery manipulator
• CPC Class Codes: G05B 2219/45123 Electrogoniometer, neuronavigator, medical robot used by surgeon to operate
• CPC Class Codes: G06T 11/40 Filling a planar surface by adding surface attributes, e.g. colour or texture
• CPC Class Codes: G06T 11/60 : Editing figures and text; Combining figures or text
• CPC Class Codes: Y02P 90/02 : Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]

Abstract

An operator telerobotically controls tools to perform a procedure on an object at a work site while viewing real-time images of the work site on a display. Tool information is provided in the operator'"'"'s current gaze area on the display by rendering the tool information over the tool so as not to obscure objects being worked on at the time by the tool nor to require eyes of the user to refocus when looking at the tool information and the image of the tool on a stereo viewer.

Claims

1. A system comprising:

a first input device;

a first tool having a first working end, the first tool operatively coupled to the first input device;

an image capture device adapted to generate a captured image from a perspective of an image reference frame, the captured image including an image of the first working end;

a display; and

a processor programmed to;

cause the captured image to be displayed on the display; and

cause information of the first input device to be rendered on the image of the first working end in the captured image being displayed on the display by;

determining a pose of the first working end in the image reference frame;

registering the information of the first input device with a first designated place on the first working end in the image reference frame by using the pose of the first working end in the image reference frame; and

rendering the information of the first input device over the first designated place on the first working end on the image of the first working end in the captured image being displayed on the display, after registering the information of the first input device with the first designated place on the first working end, so that the entire information of the first input device appears as a first decal that is rendered over the image of the first working end in the captured image being displayed on the display by bending the first decal to conform in shape and be applied to an outer surface of the first working end at the first designated place.

2. The system of claim 1,

wherein the first input device is being operated by a user;

and

wherein the information of the first input device comprises information identifying the user.

3. The system of claim 1,

wherein the first input device is being operated by an appendage of a user;

and

wherein the information of the first input device comprises information identifying the appendage of the user.

4. The system of claim 1,

wherein the first working end of the first tool includes an end effector, a wrist, and a distal end of a shaft, and the first decal is rendered over one of the end effector, the wrist, and the distal end of the shaft.

5. The system of claim 1, further comprising:

a second input device;

a second tool having a second working end, the second tool operatively coupled to the second input device;

wherein the captured image includes an image of the second working end; and

wherein the processor is further programmed to;

cause information of the second input device to be rendered on the image of the second working end in the captured image being displayed on the display by;

determining a pose of the second working end in the image reference frame;

registering the information of the second input device with a second designated place on the second working end in the image reference frame by using the pose of the second working end in the image reference frame; and

rendering the information of the second input device over the second designated place on the second working end on the image of the second working end in the captured image being displayed on the display, after registering the information of the second input device with the second designated place on the second working end, so that the entire information of the second input device appears as a second decal that is rendered over the image of the second working end in the captured image being displayed on the display by bending the second decal to conform in shape and be applied to an outer surface of the second working end at the second designated place.

6. The system of claim 5,

wherein the first input device is being operated by a first user;

wherein the second input device is being operated by a second user;

wherein the information of the first input device comprises information identifying the first user; and

wherein the information of the second input device comprises information identifying the second user.

7. The system of claim 5,

wherein the first input device is being operated by a first appendage of a user;

wherein the second input device is being operated by a second appendage of the user; and

wherein the information of the first input device comprises information identifying the first appendage; and

wherein the information of the second input device comprises information identifying the second appendage.

8. The system of claim 1,

wherein the processor is further programmed to determine the pose of the first working end relative to the image reference frame by:

determining the pose of the first working end in a reference frame of the first tool and translating the pose of the first working end from the reference frame of the first tool to the pose of the first working end in the image reference frame.

9. The system of claim 8, further comprising:

a first manipulator adapted to manipulate the first tool;

wherein the processor is further programmed to determine the pose of the first working end relative to the reference frame of the first tool by using kinematics information of the first manipulator.

10. The system of claim 1,

wherein the image capture device comprises a stereoscopic camera, so that the captured image includes a stereo view of the first working end;

and

wherein the display is a stereo viewer.

11. A method comprising:

a processor causing a captured image to be displayed on a display, the captured image generated from a perspective of an image reference frame by an image capture device, the captured image including an image of a first working end of a first tool, the first tool operatively coupled to a first input device; and

the processor causing information of the first input device to be rendered on the image of the first working end in the captured image being displayed on the display by;

determining a pose of the first working end in the image reference frame;

registering the information of the first input device with a first designated place on the first working end in the image reference frame by using the pose of the first working end in the image reference frame; and

rendering the information of the first input device over the first designated place on the first working end on the image of the first working end in the captured image being displayed on the display, after registering the information of the first input device with the first designated place on the first working end, so that the entire information of the first input device appears as a first decal that is rendered over the image of the first working end in the captured image being displayed on the display by bending the first decal to conform in shape and be applied to an outer surface of the first working end at the first designated place.

12. The method of claim 11,

wherein the first input device is being operated by a user;

and

wherein the information of the first input device comprises information identifying the user.

13. The method of claim 11,

wherein the first input device is being operated by an appendage of a user;

and

wherein the information of the first input device comprises information indicating the appendage of the user.

14. The method of claim 11,

wherein the first working end of the first tool includes an end effector, a wrist, and a distal end of a shaft, and the first decal is rendered over one of the end effector, the wrist, and the distal end of the shaft.

15. The method of claim 11,

wherein the captured image includes an image of a second working end of a second tool, the second tool operatively coupled to a second input device;

and

further comprising;

the processor causing information of the second input device to be rendered on the image of the second working end in the captured image being displayed on the display by;

determining a pose of the second working end in the image reference frame;

registering the information of the second input device with a second designated place on the second working end in the image reference frame by using the pose of the second working end in the image reference frame; and

rendering the information of the second input device over the second designated place on the second working end on the image of the second working end in the captured image being displayed on the display, after registering the information of the second input device with the second designated place on the second working end, so that the entire information of the second input device appears as a second decal that is rendered over the image of the second working end in the captured image being displayed on the display by bending the second decal to conform in shape and be applied to an outer surface of the second working end at the second designated place.

16. The method of claim 15,

wherein the first input device is being operated by a first user;

wherein the second input device is being operated by a second user;

wherein the information of the first input device comprises information identifying the first user; and

wherein the information of the second input device comprises information identifying the second user.

17. The method of claim 15,

wherein the first input device is being operated by a first appendage of a user;

wherein the second input device is being operated by a second appendage of the user; and

wherein the information of the first input device comprises information identifying the first appendage; and

wherein the information of the second input device comprises information identifying the second appendage.

18. The method of claim 11,

wherein the processor determines the pose of the first working end relative to the image reference frame by:

determining the pose of the first working end in a reference frame of the first tool and translating the pose of the first working end from the reference frame of the first tool to the pose of the first working end in the image reference frame.

19. The method of claim 18,

wherein the processor determines the pose of the first working end relative to the reference frame of the first tool by using kinematics information of a first manipulator that is adapted to manipulate the first tool.

20. The method of claim 11,

wherein the image capture device comprises a stereoscopic camera, so that the captured image includes a stereo view of the first working end;

and

wherein the display is a stereo viewer.