Gaze-assisted computer interface

• US 8,793,620 B2
• Filed: 04/21/2011
• Issued: 07/29/2014
• Est. Priority Date: 04/21/2011
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Stafford, Jeffrey Roger
• Primary Examiner(s): Tan, Alvin
• Assistant Examiner(s): Xia, Xuyang
• Application Number:
  US13/092,115
• Publication Number:
  US 20120272179A1
• Time in Patent Office: 1,195 Days
• Field of Search: 715/700, 715/781
• US Class Current: 715/862
• CPC Class Codes: G06F 2203/04801 Cursor retrieval aid, i.e. visual aspect modification, blinking, colour changes, enlargement or other visual cues, for helping user do find the cursor in graphical user interfaces
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/03543 Mice or pucks G06F3/03541 takes precedence
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/0486 Drag-and-drop
• CPC Class Codes: G06F 3/1446 display composed of modules, e.g. video walls
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G09G 2320/0693 Calibration of display systems
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 5/08 Cursor circuits
• CPC Class Codes: G09G 5/14 Display of multiple viewports

Abstract

Methods, systems, and computer programs for interfacing a user with a Graphical User Interface (GUI) are provided. One method includes an operation for identifying the point of gaze (POG) of the user. The initiation of a physical action by the user, to move a position of a cursor on a display, is detected, where the cursor defines a focus area associated with a computer program executing the GUI. Further, the method includes an operation for determining if the distance between the current position of the cursor and the POG is greater than a threshold distance. The cursor is moved from the current position to a region proximate to the POG in response to the determination of the POG and to the detection of the initiation of the physical action.

Claims

1. A method for interfacing a user with a computer program executing in a computing device, the method comprising:

identifying a point of gaze (POG) of the user on a display;

detecting initiation of a physical action by the user to move a position of a cursor on the display, the cursor defining a focus area associated with the computer program;

determining if a distance between a current position of the cursor and the POG is greater than a threshold distance; and

moving the cursor from the current position to a region proximate to the POG when the distance is greater than the threshold distance, wherein moving the cursor further includes;

changing a cursor displacement speed of a mouse that controls the cursor based on the distance between the current position of the cursor and the POG, wherein the cursor moves slower as the distance between the current position of the cursor and the POG becomes smaller.

2. The method as recited in claim 1, wherein the physical action is one of moving a mouse, pressing a key, pressing a button, touching a screen, touching a touchpad, making a gesture, or speaking.

3. The method as recited in claim 1, wherein determining the distance further includes:

determining a value of the distance measured in screen pixels between the current position and the POG.

4. The method as recited in claim 1, wherein determining the distance further includes:

determining if the current position and the POG are situated in different windows in a GUI.

5. The method as recited in claim 1, wherein the cursor is one of a mouse cursor, a text cursor, a field selected cursor, or a page cursor.

6. The method as recited in claim 1, wherein moving the cursor further includes:

detecting a plurality of entry fields in a web page; and

placing the cursor on an entry field from the plurality of entry fields that is closest to the POG when the user presses a keyboard key.

7. The method as recited in claim 1, further including:

tracking a position of a head of the user.

8. The method as recited in claim 1, further including:

detecting an input from the user that triggers immediate gaze-assisted navigation; and

moving the cursor to the POG after detecting the input.

9. The method as recited in claim 1, further including:

magnifying, after moving the cursor, an area centered around the cursor for presentation on the display; and

determining a first window on the display where the POG is detected, the window belonging to a graphical user interface (GUI) that supports display of a plurality of windows on the display.

10. The method as recited in claim 9, further including:

providing a transparent appearance on the display to windows of the GUI other than the first window

11. A system with a graphical user interface (GUI), the system comprising:

a processor executing a computer program that provides the GUI, the GUI being rendered in one or more displays;

an input device operable to move a position of a cursor on the one or more displays when a physical action is initiated by a user on the input device; and

a gaze detection module operable to identify a point of gaze (POG) of the user, wherein the computer program determines if a distance between a current position of the cursor and the POG is greater than a threshold distance, and wherein the computer program moves the cursor from the current position to a region proximate to the POG when the distance is greater than the threshold distance, wherein moving the cursor further includes changing a cursor displacement speed of the input device based on the distance between the current position of the cursor and the POG, wherein the cursor moves slower as the distance between the current position of the cursor and the POG becomes smaller, wherein moving the cursor further includes detecting a plurality of entry fields in a web page and moving a page cursor to an entry field that is closest to the POG in response to a user input, wherein the page cursor indicates which entry field is active.

12. The system as recited in claim 11, wherein the gaze detection module uses one of analysis of corneal reflection, or analysis of an user image taken with a camera.

13. The system as recited in claim 12, wherein the gaze detection module analyzes a plurality of user images to determine a location of one of more physical features of the user.

14. The system as recited in claim 11, wherein the GUI provides a hint to the user when the POG is placed on an object in the GUI.

15. The system as recited in claim 11, further including:

a voice recognition module is configured to execute a voice command based on the POG.

16. A method for interfacing a user with a computer program executing in a computing device, the method comprising:

identifying a point of gaze (POG) of the user on a display;

detecting initiation of a physical action by the user to move a position of a cursor on the display, the cursor defining a focus area associated with the computer program;

determining if a distance between a current position of the cursor and the POG is greater than a threshold distance;

moving the cursor from the current position to a region proximate to the POG when the distance is greater than the threshold distance, wherein a cursor displacement speed of a mouse that controls the cursor is changed based on the distance between the current position of the cursor and the POG, and the cursor moves slower as the distance between the current position of the cursor and the POG becomes smaller; and

magnifying, after moving the cursor, an area centered around the cursor for presentation on the display.

17. The method as recited in claim 16, further including:

determining a first window on the display where the POG is detected, the window belonging to a graphical user interface (GUI) that supports display of a plurality of windows on the display.

18. The method as recited in claim 17, further including:

providing a transparent appearance on the display to windows of the GUI other than the first window.

19. The method as recited in claim 17, further including:

making the first window as having a window focus of the GUI, wherein the window focus enables user interaction with a program associated with the window focus while disabling interaction with windows not having the window focus.

20. The method as recited in claim 17, further including:

rearranging, on the display, windows that are open based on the POG.