GAZE-CONTROLLED USER INTERFACE WITH MULTIMODAL INPUT

• US 20140354539A1
• Filed: 05/29/2014
• Published: 12/04/2014
• Est. Priority Date: 05/30/2013
• Status: Active Grant

• Current Assignee: Tobii Technology AB
• Original Assignee: Tobii Technology AB
• Inventors: SKOG, Mrten, ELVESJ, John Mikael, OLSSON, Anders
• Granted Patent: US 10,372,203 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

A personal computer system provides a gaze-controlled graphical user interface having a bidirectional and a unidirectional interaction mode. In the bidirectional interaction mode, a display shows one or more graphical controls in motion, each being associated with an input operation to an operating system. A gaze tracking system provides gaze point data of a viewer, and a matching module attempts to match a relative gaze movement against a relative movement of one of the graphical controls. The system includes a selector which is preferably controllable by a modality other than gaze. The system initiates a transition from the unidirectional interaction mode to the bidirectional interaction mode in response to an input received at the selector. The display then shows graphical controls in motion in a neighbourhood of the current gaze point, as determined based on current gaze data.

Claims

1. A personal computer system providing a gaze-controlled graphical user interface, the personal computer system comprising:

an operating system for executing an application;

a graphics module for generating a display signal to be supplied to a visual display, the graphics module being operable in;

a bidirectional interaction mode, in which the visual display shows one or more graphical controls in motion, and

a unidirectional interaction mode without said graphical controls in motion;

a gaze tracking system operable to provide gaze point data of a viewer of the visual display;

a matching module operable to determine a match between a relative movement of any of said graphical controls and a relative gaze movement derived from said gaze point data in the bidirectional mode,

wherein the personal computer system further comprises a selector, and the personal computer system is configured to;

i) initiate a transition from the unidirectional interaction mode to the bidirectional interaction mode in response to an input received at the selector;

ii) determine a current gaze point based on gaze point data; and

iii) enter the bidirectional interaction mode, wherein graphical controls in motion are present at least in a neighbourhood of the current gaze point.

2. The system of claim 1, wherein:

each of the graphical controls in motion, which are shown by the visual display in the bidirectional interaction mode, is associated with at least one predefined input operation to the operating system or to an application executing on the operating system; and

the matching module is adapted to indicate said at least one predefined input operation associated with a graphical control in motion, for which it has determined a match, to the operating system or to the application executing thereon.

3. The system of claim 1, wherein the neighbourhood of the gaze point is selected substantially smaller than a screen area of the visual display, the system being configured to

iv) enter the bidirectional interaction mode, wherein graphical controls in motion are present only in a neighbourhood of the current gaze point.

4. The system of claim 1, further comprising a timeout counter configured to measure a current dwell time in the bidirectional interaction mode, the system being adapted to initiate a transition back to the unidirectional interaction mode on a condition that the matching module has not determined a match and the dwell time exceeds a predefined threshold.

5. The system of claim 1, wherein the graphics module causes the visual display to show a stationary equivalent of at least one of the graphical controls in the unidirectional interaction mode.

6. The system of claim 1, wherein the graphics module generates a display signal in the unidirectional interaction mode in which at least one of the graphical controls is absent.

7. The system of claim 1, wherein said neighbourhood of the current gaze point comprises a plurality of graphical controls and undergoes a time-variable surface deformation.

8. The system of claim 1, wherein:

the gaze tracking system is adapted to supply gaze point data, which includes a gaze point of the viewer, by means of a configurable mapping having as input pre-processed eye image data; and

the gaze tracking system is operable to calibrate said configurable mapping by equating simultaneous values of a gaze point of the viewer and a location of a matching graphical control in motion or, if applicable, by equating simultaneous values of a gaze point of the viewer and a location of a stationary equivalent of a matching graphical control.

9. The system of claim 1, wherein:

the gaze tracking system is adapted to supply gaze point data, which includes a gaze point of the viewer; and

the matching module, the operating system or an application executing on the operating system is operable to output a calibration quality index based on a deviation between simultaneous values of a gaze point of the viewer and a location of a matching graphical control in motion.

10. The system of claim 9, wherein the size of said neighbourhood of the current gaze point is variable in dependence of the calibration quality index, wherein a relatively larger neighbourhood is provided in response to a calibration quality index value indicating a relatively larger deviation.

11. The system of claim 1, wherein the matching between the relative movement of the graphical controls and the relative gaze movement is determined based on a motion property which is one in the group comprising:

speed,

acceleration,

direction,

curvature.

12. The system of claim 1, wherein the selector is controllable by a modality in the group comprising:

voice,

body gesture,

body proximity,

touch activation.

13. The system of claim 1, wherein the visual display shows two or more graphical controls in motion in the bidirectional interaction mode, said graphical controls being associated with distinct predefined input operations to the operating system or the application executing thereon.

14. The system of claim 13, wherein the matching between the relative movement of the graphical controls and the relative gaze movement is determined based on a motion property which is one in the group comprising:

speed,

acceleration,

direction,

curvature, and

wherein graphical controls associated with different predefined input operations differ in respect of one or more of said motion properties.

17. The system of claim 2, wherein the neighbourhood of the gaze point is selected substantially smaller than a screen area of the visual display, the system being configured to

iv) enter the bidirectional interaction mode, wherein graphical controls in motion are present only in a neighbourhood of the current gaze point.

18. The system of claim 2, further comprising a timeout counter configured to measure a current dwell time in the bidirectional interaction mode, the system being adapted to initiate a transition back to the unidirectional interaction mode on a condition that the matching module has not determined a match and the dwell time exceeds a predefined threshold.

19. The system of claim 3, further comprising a timeout counter configured to measure a current dwell time in the bidirectional interaction mode, the system being adapted to initiate a transition back to the unidirectional interaction mode on a condition that the matching module has not determined a match and the dwell time exceeds a predefined threshold.

20. The system of claim 2, wherein the graphics module causes the visual display to show a stationary equivalent of at least one of the graphical controls in the unidirectional interaction mode.

15. A method of controlling an operating system in a personal computer system associated with a visual display, comprising:

displaying an image without graphical controls in motion;

receiving a mode transition command from a viewer of the visual display;

determining a current gaze point of the viewer based on gaze point data provided by a gaze tracking system;

displaying an image including one or more graphical controls in motion; and

in response to a positive determination of a match, indicating a predefined input operation associated with the graphical control to the operating system or to the application executing thereon,

wherein said graphical controls in motion are present at least in a neighbourhood of the current gaze point.

16. A computer program product comprising a non-transitory computer-readable medium with instructions for causing a programmable computer to perform the method of claim 15.