Gaze-controlled user interface with multimodal input
First Claim
1. A personal computer system providing a gaze-controlled graphical user interface, the personal computer system comprising:
- a gaze tracking system operable to provide gaze point data of a viewer of a visual display; and
a processor configured to generate a display signal to be supplied to the visual display, the visual display associated with a reference coordinate system that is stationary with respect to the visual display;
wherein the personal computer system is configured to;
determine a gaze point on the visual display based on the gaze point data, the gaze point determined with respect to the reference coordinate system,operate in a bidirectional interaction mode, in which the visual display shows multiple graphical controls in motion with respect to the reference coordinate system, wherein each one of the multiple graphical controls in motion, when selected using gaze input, causes a respective predefined input operation to occur,operate in a unidirectional interaction mode, in which the visual display shows the multiple graphical controls in constant locations with respect to the reference coordinate system, when selected using a non-gaze input causes the respective predefined input operation to occur;
transition from the unidirectional interaction mode to the bidirectional interaction mode, wherein the transition causes the multiple graphical controls to be in motion;
in response to transitioning to the bidirectional interaction mode;
present at least in a neighborhood of the gaze point a first graphical control in motion and a second graphical control in motion, wherein the neighborhood is substantially centered on the gaze point and is smaller than an entirety of the visual display,determine a first path including a first relative motion of the first graphical control and a second path including a second relative motion of the second graphical control, wherein the second relative motion is distinguishable from the first relative motion,determine a gaze motion based on relative movements of the gaze point,compare the gaze motion to each of the first path and the second path, andin response to determining a match between the gaze motion and either the first path or the second, indicate the respective predefined input operation associated with the matching one of the first graphical control or the second graphical control, andtransition from the bidirectional interaction mode to the unidirectional interaction mode when one or more criteria are met, causing the multiple graphical controls to be in constant locations.
2 Assignments
0 Petitions
Accused Products
Abstract
A personal computer system provides a gaze-controlled graphical user interface having a bidirectional and a unidirectional interaction mode. In the bidirectional interaction mode, a display shows one or more graphical controls in motion, each being associated with an input operation to an operating system. A gaze tracking system provides gaze point data of a viewer, and a matching module attempts to match a relative gaze movement against a relative movement of one of the graphical controls. The system includes a selector which is preferably controllable by a modality other than gaze. The system initiates a transition from the unidirectional interaction mode to the bidirectional interaction mode in response to an input received at the selector. The display then shows graphical controls in motion in a neighborhood of the current gaze point, as determined based on current gaze data.
22 Citations
20 Claims
-
1. A personal computer system providing a gaze-controlled graphical user interface, the personal computer system comprising:
-
a gaze tracking system operable to provide gaze point data of a viewer of a visual display; and a processor configured to generate a display signal to be supplied to the visual display, the visual display associated with a reference coordinate system that is stationary with respect to the visual display; wherein the personal computer system is configured to; determine a gaze point on the visual display based on the gaze point data, the gaze point determined with respect to the reference coordinate system, operate in a bidirectional interaction mode, in which the visual display shows multiple graphical controls in motion with respect to the reference coordinate system, wherein each one of the multiple graphical controls in motion, when selected using gaze input, causes a respective predefined input operation to occur, operate in a unidirectional interaction mode, in which the visual display shows the multiple graphical controls in constant locations with respect to the reference coordinate system, when selected using a non-gaze input causes the respective predefined input operation to occur; transition from the unidirectional interaction mode to the bidirectional interaction mode, wherein the transition causes the multiple graphical controls to be in motion; in response to transitioning to the bidirectional interaction mode; present at least in a neighborhood of the gaze point a first graphical control in motion and a second graphical control in motion, wherein the neighborhood is substantially centered on the gaze point and is smaller than an entirety of the visual display, determine a first path including a first relative motion of the first graphical control and a second path including a second relative motion of the second graphical control, wherein the second relative motion is distinguishable from the first relative motion, determine a gaze motion based on relative movements of the gaze point, compare the gaze motion to each of the first path and the second path, and in response to determining a match between the gaze motion and either the first path or the second, indicate the respective predefined input operation associated with the matching one of the first graphical control or the second graphical control, and transition from the bidirectional interaction mode to the unidirectional interaction mode when one or more criteria are met, causing the multiple graphical controls to be in constant locations. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13)
-
-
14. A method of controlling an operating system in a personal computer system associated with a visual display, the method including operations comprising:
-
displaying on the visual display an image of a user interface, the image displayed with respect to a reference coordinate system that is stationary with respect to the visual display; in a unidirectional interaction mode, in which the visual display shows multiple graphical controls in constant locations with respect to the reference coordinate system, wherein selection of each one of the multiple graphical controls not in motion using a non-gaze input causes a respective predefined input operation to occur; receiving a mode transition command from a viewer of the visual display, wherein the mode transition command triggers a transition from the unidirectional mode to a bidirectional interaction mode in which the transition causes the multiple graphical controls to be in motion with respect to the reference coordinate system; in the bidirectional interaction mode; determining a gaze point of the viewer relative to the visual display based on gaze point data provided by a gaze tracking system, the gaze point determined with respect to the reference coordinate system; determining a first path including a first relative motion of a first graphical control and a second path including a second relative motion of a second graphical control, wherein the second relative motion is distinguishable from the first relative motion, wherein the first graphical control and the second graphical control are present at least in a neighborhood of the gaze point on the visual display, and wherein the neighborhood is substantially centered on the gaze point and is smaller than an entirety of the visual display; determining a gaze motion based on relative movements of the gaze point; comparing the gaze motion to each of the first path and the second path; and in response to determining a match between the gaze motion and either the first path or the second path, indicating the respective predefined input operation associated with the matching one of the first graphical control or the second graphical control by sending a corresponding instruction to the operating system or to an application executing thereon; and transitioning from the bidirectional interaction mode to the unidirectional interaction mode when one or more criteria are met, causing the multiple graphical controls to be in constant locations. - View Dependent Claims (15, 16, 17, 18, 19, 20)
-
Specification