COMPUTER-IMPLEMENTED GAZE INTERACTION METHOD AND APPARATUS

• US 20170123491A1
• Filed: 03/16/2015
• Published: 05/04/2017
• Est. Priority Date: 03/17/2014
• Status: Abandoned Application

• Current Assignee: ITU Business Development
• Original Assignee: ITU Business Development
• Inventors: Hansen, Dan Witzner, Mardanbegi, Diako
• Application Number:
  US15/126,596
• Publication Number:
  US 20170123491A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: H04L 12/2814 Exchanging control software or macros for controlling appliance services in a home automation network arrangements for maintenance or administration involving configuration of the network and network elements H04L41/08

Abstract

A computer-implemented method of communicating via interaction with a user-interface based on a person'"'"'s gaze and gestures, comprising: computing an estimate of the person'"'"'s gaze comprising computing a point-of-regard on a display through which the person observes a scene in front of him; by means of a scene camera, capturing a first image of a scene in front of the person'"'"'s head (and at least partially visible on the display) and computing the location of an object coinciding with the person'"'"'s gaze; by means of the scene camera, capturing at least one further image of the scene in front of the person'"'"'s head, and monitoring whether the gaze dwells on the recognised object; and while gaze dwells on the recognised object: firstly, displaying a user interface element, with a spatial expanse, on the display face in a region adjacent to the point-of-regard; and secondly, during movement of the display, awaiting and detecting the event that the point-of-regard coincides with the spatial expanse of the displayed user interface element. The event may be processed by communicating a message.

Claims

1. A computer-implemented method of responding to a person'"'"'s gaze, comprising:

computing an estimate of the person'"'"'s gaze comprising computing a point-of-regard on a screen through which the person observes a scene;

identifying an object the person is looking at, if any, using the estimate of the person'"'"'s gaze and information about an object'"'"'s identity and location relative to the gaze;

detecting an event indicating that the person wants to interact with the object, and in response thereto displaying a predefined user interface element on the screen in a region adjacent to the point of regard;

verifying, during movement of the screen, that gaze is fixed on the object; and

then detecting the event that the point-of-regard coincides with the spatial expanse of a predefined displayed user interface element; and

processing the event that the point-of-regard coincides with the spatial expanse of a predefined displayed user interface element comprising performing an action.

3. A computer-implemented method according to claim 1, further comprising:

entering an interaction state as of the moment an object the person is looking at is identified;

while in the interaction state detecting whether the person'"'"'s gaze is away from the object; and

in the positive event thereof, exiting the interaction state to prevent performing the action.

4. A computer-implemented method according to claim 1, further comprising:

displaying multiple user interface elements, with respective spatial expanses, on the display on a location adjacent to the point-of-regard;

wherein the user interface elements are linked with a respective action;

determining which, if any, among the multiple user interface elements the point-of-regard coincides with; and

selecting an action linked with the determined user interface element.

5. A computer-implemented method according to claim 1, further comprising:

arranging the location and/or size of one or multiple user interface element(s) on the user interface plane in dependence of the distance between the location of the point-of-regard and bounds of the user interface plane.

6. A computer-implemented method according to claim 1, further comprising:

estimating a main direction or path of a moving object; and

arranging the location of one or multiple user interface element(s) on the display within at least one section thereof in order to prevent unintentional collision with the object moving in the main direction or along the path.

7. A computer-implemented method according to claim 1, further comprising:

transmitting the message to a remote station configured to communicate with the identified object and/or transmitting the message directly to a communications unit installed with the identified object.

8. A computer program product comprising program code means adapted to cause a data processing system to perform the steps of the method according to claim 1, when said program code means are executed on the data processing system.

9. A computer data signal embodied in a carrier wave and representing sequences of instructions which, when executed by a processor, cause the processor to perform the steps of the method according to claim 1.

2. (canceled)

10. A mobile device, such as a head-worn computing device, configured with a screen and to respond to a person'"'"'s gaze, comprising:

an eye-tracker configured to compute an estimate of the person'"'"'s gaze comprising computing a point-of-regard on a screen through which the person observes a scene;

a processor configured to identify an object the person is looking at, if any, using the estimate of the person'"'"'s gaze and information about an object'"'"'s identity and location relative to the gaze; and

a processor configured to;

detect the event that the person wants to interact with the object, and in response thereto displaying a predefined user interface element on the screen in a region adjacent to the point of regard;

verify, during movement of the screen, that gaze is and remains fixed on the object; and

then detect the event that the point-of-regard coincides with the spatial expanse of the predefined user interface element on the screen, and to process the event that the point-of-regard coincides with the spatial expanse of a predefined displayed user interface element comprising performing an action.

12. A mobile device according to claim 10 configured to

enter an interaction state as of the moment an object the person is looking at is identified;

and

while in the interaction state, detect whether the person'"'"'s gaze is away from the object; and

in the positive event thereof, exit the interaction state to prevent performing the action.

13. A mobile device according to claim 10 configured to:

display multiple user interface elements, with respective spatial expanses, on the screen on a location adjacent to the point-of-regard;

wherein the user interface elements are linked with a respective action;

determine which, if any, among the multiple user interface elements the point-of-regard coincides with; and

select an action linked with the determined user interface element.

14. A mobile device according to claim 10, configured to:

arrange the location and/or size of one or multiple user interface element(s) on the screen in dependence of the distance between the location of the point-of-regard and bounds of the user interface plane.

15. A mobile device according to claim 10, configured to:

estimate a main direction or path of a moving object; and

arrange the location of one or multiple user interface element(s) on the display within at least one section thereof in order to prevent unintentional collision with the object moving in the main direction or along the path.

16. A mobile device according to claim 10, configured to:

transmit the message to a remote station configured to communicate with the identified object and/or transmitting the message directly to a communications unit installed with the identified object.

11. (canceled)