ROBOTIC SYSTEM PROVIDING USER SELECTABLE ACTIONS ASSOCIATED WITH GAZE TRACKING

• US 20140282196A1
• Filed: 03/14/2014
• Published: 09/18/2014
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Zhao, Wenyi, Hasser, Christopher J., Itkowitz, Brandon D., Lilagan, Paul E., Scott, David D., Mohr, Paul W., DiMaio, Simon P., Zhao, Tao, Robinson, David W.
• Granted Patent: US 11,747,895 B2
• Time in Patent Office: Days
• US Class Current: 715/771
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object

Abstract

A robotic system provides user selectable actions associated with gaze tracking according to user interface types. User initiated correction and/or recalibration of the gaze tracking may be performed during the processing of individual of the user selectable actions.

Claims

1. A medical robotic system comprising:

a viewer for displaying an image of a work site;

a gaze tracker for tracking a gaze point of a user on the viewer;

a processor programmed to perform a plurality of user selectable actions associated with the gaze point.

2. The medical robotic system of claim 1, wherein the processor is programmed to perform individual of the plurality of user selectable actions according to whether the gaze point is associated with one of an object, a box, a cursor, and a calibration function.

3. The medical robotic system of claim 2, wherein the gaze point is associated with the object, and wherein the object comprises one of a tool, an anatomical structure, an icon, and text.

4. The medical robotic system of claim 3, wherein the object comprises the tool, and wherein individual of the plurality of user selectable actions is indicated by a user command comprising one of:

a command to display a menu of available action commands;

a command to identify the tool;

a command to identify an input device associated with the tool;

a command to disassociate the tool from an input device and associate another tool to the input device; and

a command to automatically track movement of the tool to maintain an image of the tool within a field of view of an image capturing device capturing images of the work site.

5. The medical robotic system of claim 3, wherein the object comprises the anatomical structure, and wherein individual of the plurality of user selectable actions is indicated by a user command comprising one of:

a command to display a menu of available action commands;

a command to display text associated with the anatomical structure;

a command to identify the anatomical structure;

a command to capture an image of the anatomical structure, store the captured image, and generate a selectable icon associated with the stored image;

a command to display icons indicating stored images of the anatomical structure;

a command to display a stored image by superimposing the stored image over a currently displayed image of the anatomical structure; and

a command to display selectable icons of stored images of the anatomical structure.

6. The medical robotic system of claim 3, wherein the object comprises the icon, and wherein individual of the plurality of user selectable actions comprises performing an action indicated by the icon.

7. The medical robotic system of claim 6, wherein the icon comprises one of:

an icon indicating a replacement tool;

an icon indicating an input device to be associated with a tool;

an icon associated with an imaging modality to indicate an image is to be captured using the imaging modality; and

an icon indicating a stored image to be displayed on the viewer.

8. The medical robotic system of claim 3, wherein the object comprises text, and wherein individual of the plurality of user selectable actions comprises performing an action indicated by the text.

9. The medical robotic system of claim 8, wherein the text is displayed in a menu of available ones of the plurality of user selectable actions.

10. The medical robotic system of claim 8, wherein the text is displayed proximate to one of the tool and the anatomical structure, and wherein the text is associated with the tool when displayed proximate to the tool and the text is associated with the anatomical structure when displayed proximate to the anatomical structure.

11. The medical robotic system of claim 3, wherein the processor is programmed to:

highlight a closest object to the tracked gaze point as determined by the gaze tracker on the viewer after the user has gazed on the viewer for a programmed period of time.

12. The medical robotic system of claim 11, wherein the processor is programmed to adjust a gaze point calibration of the gaze tracker if an indication for such adjustment is received from the user after highlighting the closest object to the tracked gaze point on the viewer.

13. The medical robotic system of claim 2, wherein the gaze point is associated with the box, and wherein individual of the plurality of user selectable actions comprises one of:

adjusting an image characteristic of a region of interest defined by the box;

enlarging a portion of an image indicated by the box;

displaying an image of the box on an auxiliary viewer which is also displaying the image of the worksite; and

creating a virtual constraint whose boundary is defined by the box.

14. The medical robotic system of claim 13, wherein the processor is programmed to implement the virtual constraint by preventing a tool from entering the boundary defined by the box.

15. The medical robotic system of claim 13, wherein the processor is programmed to implement the virtual constraint by preventing a tool from exiting the boundary defined by the box.

16. The medical robotic system of claim 13, wherein the processor is programmed to:

cause a box to be drawn around the tracked gaze point as determined by the gaze tracker on the viewer after the user has gazed on the viewer for a programmed period of time.

17. The medical robotic system of claim 16, wherein the processor is programmed to adjust a gaze point calibration of the gaze tracker if an indication for such adjustment is received from the user after causing the box to be drawn around the tracked gaze point on the viewer.

18. The medical robotic system of claim 16, wherein the processor is programmed to adjust one of the size and location of the box drawn around the tracked gaze point if an indication for such adjustment is received from the user after causing the box to be drawn around the tracked gaze point on the viewer.

19. The medical robotic system of claim 2, wherein the gaze point is associated with the cursor, and wherein individual of the plurality of user selectable actions is indicated by a user command comprising one of:

a command to display a cursor at the gaze point on the viewer;

a command to store a current position of the gaze point in memory; and

a command to perform an action related to a trajectory of the gaze point over the viewer.

20. The medical robotic system of claim 19, wherein the processor is programmed to:

cause a cursor to be automatically displayed at the tracked gaze point as determined by the gaze tracker on the viewer after the user has gazed on the viewer for a programmed period of time.

21. The medical robotic system of claim 20, wherein the processor is programmed to adjust a gaze point calibration of the gaze tracker if an indication for such adjustment is received from the user after causing the cursor to be drawn at the tracked gaze point on the viewer.

22. The medical robotic system of claim 2, wherein the gaze point is associated with the calibration function, and wherein individual of the plurality of user selectable actions comprises calibrating the tracked gaze point as determined by the gaze tracker with an actual gaze point of the user on the viewer.

23. The medical robotic system of claim 22, wherein the processor is programmed to calibrate the tracked gaze point as determined by the gaze tracker with the actual gaze point of the user on the viewer by sequentially displaying markers of known position on the viewer and adjusting for errors between the known position and the tracked gaze point as determined by the gaze tracker.

24. The medical robotic system of claim 1, wherein the processor is programmed to identify a user selection from among the plurality of user selectable actions by processing a voice command entered by the user into a microphone.

25. The medical robotic system of claim 1, wherein the processor is programmed to identify a user selection from among the plurality of user selectable actions by processing a command entered by the user into a user interface input.

26. A method for performing a plurality of user selectable actions associated with gaze tracking, the method comprising:

determining a gaze point of a user on a viewer by using gaze point calibration data;

providing an indication of the determined gaze point on the viewer; and

performing a user selected action associated with the gaze point upon receiving an indication from the user that the determined gaze point does not require adjustment.

27. The method of claim 26, further comprising:

interactively adjusting the gaze point calibration data to correct for errors between the determined gaze point and an actual gaze point of the user on the viewer upon receiving an indication from the user that the determined gaze point requires adjustment.