SYSTEM FOR GAZE INTERACTION

• US 20130169560A1
• Filed: 10/05/2012
• Published: 07/04/2013
• Est. Priority Date: 01/04/2012
• Status: Active Grant

• Current Assignee: Tobii Technology AB
• Original Assignee: Tobii Technology AB
• Inventors: Kaplan, Anders, Olsson, Anders, Skog, Mrten, Cederlund, Markus, Gavelin, Robert, Vennstrm, Anders
• Granted Patent: US 10,013,053 B2
• Time in Patent Office: Days
• US Class Current: 345/173
• CPC Class Codes: B60K 2360/1468 Touch gesture
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60K 35/23 Head-up displays [HUD] optical aspects of head-up displays G02B27/01
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/1613 for portable computers cooling arrangements therefor G06F1/203; constructional details or arrangements for pocket calculators, electronic agendas or books G06F15/0216; constructional details of portable telephone sets: with several bodies H04M1/0202
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

A method and system for assisting a user when interacting with a graphical user interface combines gaze based input with gesture based user commands. A user of a computer system without a traditional touch-screen can interact with graphical user interfaces in a touch-screen like manner using a combination of gaze based input and gesture based user commands. A solution for touch-screen like interaction uses gaze input and gesture based input as a complement or an alternative to touch-screen interactions with a computer device having a touch-screen. Combined gaze and gesture based interaction with graphical user interfaces can be used to achieve a touchscreen like environment in computer systems without a traditional touchscreen or in computer systems having a touchscreen arranged ergonomically unfavourable for the user or a touchscreen arranged such that it is more comfortable for the user to use gesture and gaze for the interaction than the touchscreen.

Claims

1. A control module for generating gesture based commands during user interaction with an information presentation area, wherein said control module is configured to:

acquire user input from input means adapted to detect user generated gestures and gaze data signals from a gaze tracking module; and

determine at least one user generated gesture based control command based on said user input;

determine a gaze point area on said information presentation area including the user'"'"'s gaze point based on at least the gaze data signals; and

execute at least one user action manipulating a view presented on said graphical information presentation area based on said determined gaze point area and at least one user generated gesture based control command, wherein said user action is executed with said determined gaze point area as a starting point.

2. The control module according to claim 1, further configured to:

determine at least one gesture based control command based on multiple simultaneous user input via said input means;

determine a gaze point area on said information presentation area including the user'"'"'s gaze point based on the gaze data signals; and

execute at least one user action manipulating a view presented on said graphical information presentation area based on said determined gaze point area and said at least one gesture based control command, wherein said user action is executed with said determined gaze point area as a starting point.

3. The control module according to claim 1,

wherein said control module comprises an input module configured to interpret signals representing at least one user generated gesture to provide at least one gesture based control command reflecting a user'"'"'s gesture, wherein said input module is configured to interpret the signals representing said at least one user generated gesture using gaze input signals and/or a predetermined set of possible gesture based control commands, each possible control command corresponding to a particular user gesture relative the input means.

4. The control module according to claim 1,

wherein at least one object is presented on said information presentation area, said object representing at least one graphical user interface component and configured to be manipulated based on the user-generated gesture based control commands, wherein said control module is configured to:

determine if the gaze point of the user is on an object or in an area surrounding that object based on the gaze data signals;

enable user activation of said object if the user'"'"'s gaze point is on or within an area surrounding that object synchronized with a user generated gesture based control command resulting from user input via said input means, wherein the activated object can be manipulated by user generated commands resulting from user input via said input means.

18. The control module according to claim 2,

wherein said control module comprises an input module configured to interpret signals representing at least one user generated gesture to provide at least one gesture based control command reflecting a user'"'"'s gesture, wherein said input module is configured to interpret the signals representing said at least one user generated gesture using gaze input signals and/or a predetermined set of possible gesture based control commands, each possible control command corresponding to a particular user gesture relative the input means.

19. The control module according to claim 2,

wherein at least one object is presented on said information presentation area, said object representing at least one graphical user interface component and configured to be manipulated based on the user-generated gesture based control commands, wherein said control module is configured to:

determine if the gaze point of the user is on an object or in an area surrounding that object based on the gaze data signals;

enable user activation of said object if the user'"'"'s gaze point is on or within an area surrounding that object synchronized with a user generated gesture based control command resulting from user input via said input means, wherein the activated object can be manipulated by user generated commands resulting from user input via said input means.

20. The control module according to claim 3,

wherein at least one object is presented on said information presentation area, said object representing at least one graphical user interface component and configured to be manipulated based on the user-generated gesture based control commands, wherein said control module is configured to:

determine if the gaze point of the user is on an object or in an area surrounding that object based on the gaze data signals;

enable user activation of said object if the user'"'"'s gaze point is on or within an area surrounding that object synchronized with a user generated gesture based control command resulting from user input via said input means, wherein the activated object can be manipulated by user generated commands resulting from user input via said input means.

5. A method for generating gesture based control commands during user interaction with an information presentation area associated with a computer device, said method comprising:

acquiring user input corresponding to user generated gestures and gaze data signals; and

determining at least one user generated gesture based control command based on said user input;

determining a gaze point area on said information presentation area including the user'"'"'s gaze point based on at least the gaze data signals; and

executing at least one user action manipulating a view presented on said information presentation area based on said determined gaze point area and at least one user generated gesture based control command, wherein said user action is executed with said determined gaze point area as a starting point.

6. The method according to claim 5, further comprising acquiring user generated gesture based control commands, said gesture based control commands being generated by touching a surface of input means including a touchpad or touch-screen or touching a surface of the touchpad.

7. The method according to claim 5, further comprising:

determining at least one gesture based control command based on multiple simultaneous user input via input means;

determining a gaze point area on said information presentation area including the user'"'"'s gaze point based on the gaze data signals; and

executing at least one user action manipulating a view presented on said graphical information presentation area based on said determined gaze point area and said at least one gesture based control command, wherein said user action is executed with said determined gaze point area as a starting point.

8. The method according to claim 7, further comprising interpreting the signals representing said at least one user generated gesture to provide at least one gesture based control command reflecting a user'"'"'s gesture, said signal being interpreted using gaze input signals and/or a predetermined set of possible gesture based control commands, each possible control command corresponding to a particular user gesture relative the input means.

9. The method according to claim 5, further comprising:

presenting at least one object on said information presentation, said object representing at least one graphical user interface component and configured to be manipulated based on the user-generated gesture based control commands;

determining if the gaze point of the user is on an object or in an area surrounding that object based on the gaze data signals;

enabling user activation of said object if the user'"'"'s gaze point is on or within an area surrounding that object synchronized with a user generated activation command resulting from user input via said input means, wherein the activated object can be manipulated by user generated commands resulting from user input via said input means.

10. The method according to claim 5,

further comprising acquiring gesture based control commands, wherein a user may generate a control command resulting in zooming out of an object presented on said information presentation area by a pinching movement of pressure on the surface of input means including a touchpad.

11. The method according to claim 5, further comprising acquiring gesture based control commands, wherein a user may generate a control command resulting in a rotation of an object presented on said information presentation area by a rotational movement on the surface of input means including a touchpad.

12. A wireless transmit/receive unit, WTRU, associated with an information presentation area and comprising input means adapted to detect user generated gestures and a gaze tracking module adapted to detect gaze data of a viewer of said information presentation area, said WTRU further comprising a control module configured to:

acquire user input from said input means and gaze data signals from said gaze tracking module;

determine at least one user generated gesture based control command based on said user input;

determine a gaze point area on said information presentation area including the user'"'"'s gaze point based on at least the gaze data signals; and

execute at least one user action manipulating a view presented on said information presentation area based on said determined gaze point area and at least one user generated gesture based control command, wherein said user action is executed with said determined gaze point area as a starting point.

13. A system for user interaction with an information presentation area, said system comprising:

input means adapted to detect user generated gestures;

a gaze tracking module adapted to detect gaze data of a viewer of said information presentation area;

a control module configured to;

acquire user input from said input means and gaze data signals from said gaze tracking module;

determine at least one user generated gesture based control command based on said user input;

determine a gaze point area on said information presentation area including the user'"'"'s gaze point based on at least the gaze data signals; and

execute at least one user action manipulating a view presented on said graphical information presentation area based on said determined gaze point area and at least one user generated gesture based control command, wherein said user action is executed with said determined gaze point area as a starting point.

14. A computer device associated with an information presentation area, said computer device comprising:

input means adapted to detect user generated gestures;

a gaze tracking module adapted to detect gaze data of a viewer of said information presentation area;

a control module configured to;

acquire user input from input means adapted to detect user generated gestures and gaze data signals from a gaze tracking module;

determine at least one user generated gesture based control command based on said user input;

determine a gaze point area on said information presentation area including the user'"'"'s gaze point based on at least the gaze data signals; and

execute at least one user action manipulating a view presented on said information presentation area based on said determined gaze point area and at least one user generated gesture based control command, wherein said user action is executed with said determined gaze point area as a starting point.

15. A handheld portable device including an information presentation area and comprising input means adapted to detect user generated gestures and a gaze tracking module adapted to detect gaze data of a viewer of said information presentation area, said device further comprising a control module configured to:

acquire user input from said input means and gaze data signals from said gaze tracking module;

determine at least one user generated gesture based control command based on said user input;

determine a gaze point area on said information presentation area including the user'"'"'s gaze point based on at least the gaze data signals; and

execute at least one user action manipulating a view presented on said information presentation area based on said determined gaze point area and at least one user generated gesture based control command, wherein said user action is executed with said determined gaze point area as a starting point.

16. A system for user interaction with a wearable head mounted information presentation area, said system comprising:

input means adapted to be worn on a wrist, a hand, or at least a finger,

said input means being configured to detect user generated gestures and adapted to wirelessly communicate with a control module;

a gaze tracking module adapted to detect gaze data of a viewer of said information presentation area; and

wherein said control module is configured to;

acquire user input from said input means and gaze data signals from said gaze tracking module;

determine at least one user generated gesture based control command based on said user input;

determine a gaze point area on said information presentation area including the user'"'"'s gaze point based on at least the gaze data signals; and

execute at least one user action manipulating a view presented on said graphical information presentation area based on said determined gaze point area and at least one user generated gesture based control command, wherein said user action is executed with said determined gaze point area as a starting point.

17. A system for user interaction with an information presentation area, said system comprising:

input means adapted to detect user generated gestures, said input means comprising at least one touchpad arranged on a steering device of a vehicle or adapted to be integrated in a steering device of a vehicle;

a gaze tracking module adapted to detect gaze data of a viewer of said information presentation area;

a control module configured to;

acquire user input from said input means and gaze data signals from said gaze tracking module;

determine at least one user generated gesture based control command based on said user input;

determine a gaze point area on said information presentation area including the user'"'"'s gaze point based on at least the gaze data signals; and

execute at least one user action manipulating a view presented on said graphical information presentation area based on said determined gaze point area and at least one user generated gesture based control command, wherein said user action is executed with said determined gaze point area as a starting point.