Selection using eye gaze evaluation over time

• US 9,804,753 B2
• Filed: 03/20/2014
• Issued: 10/31/2017
• Est. Priority Date: 03/20/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Ramsby, Scott, Ambrus, Tony, Scavezze, Michael, Lee, Abby Lin, Mount, Brian, McIntyre, Ian Douglas, Burns, Aaron Mackay, McMackin, Russ, Doran, Katelyn Elizabeth, Schneider, Gerhard, Miller, Quentin Simon Charles
• Primary Examiner(s): SHAH, PRIYANK J
• Application Number:
  US14/221,147
• Publication Number:
  US 20150268821A1
• Time in Patent Office: 1,321 Days
• Field of Search: 345156, 345173, 345174, 345175, 345176, 345177, 345178, 345104, 345204
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06V 40/19 Sensors therefor

Abstract

Various embodiments relating to selection of a user interface object displayed on a graphical user interface based on eye gaze are disclosed. In one embodiment, a selection input may be received. A plurality of eye gaze samples at different times within a time window may be evaluated. The time window may be selected based on a time at which the selection input is detected. A user interface object may be selected based on the plurality of eye gaze samples.

Claims

1. On a computing device comprising an eye tracking system, a method for selecting a user interface object displayed on a graphical user interface based on eye gaze, the method comprising:

sampling eye gaze samples via the eye tracking system;

storing the eye gaze samples;

detecting a selection input, via a user input mechanism other than the eye tracking system, at a time when an eye gaze location intersects a first user interface object;

upon receipt of the selection input, evaluating a plurality of the eye gaze samples at different times within a time window selected based on the time at which the selection input is received to determine a combined location of the plurality of the eye gaze samples;

when the combined location of the plurality of the eye gaze samples is closer to a second user interface object than the first user interface object, then selecting the second user interface object as a result of the selection input;

evaluating a different plurality of the eye gaze samples in the time window to determine a combined location of the different plurality of the eye gaze samples; and

confirming the selecting of the second user interface object or selecting a different user interface object as a result of the selection input based upon the combined location of the different plurality of the eye gaze samples.

2. The method of claim 1, wherein the second user interface object is selected based on having a highest weighting according to a selection rule.

3. The method of claim 1, wherein the time window includes eye gaze samples taken before and after the time at which the selection input is received, and wherein the method further comprises selecting the first user interface object when the combined location of the plurality of the eye gaze samples is closer to the first user interface object than the second user interface object.

4. The method of claim 1, further comprising:

setting a center of the time window to a different time that is prior to the time at which the selection input is received.

5. The method of claim 1, further comprising:

evaluating the plurality of the eye gaze samples compared to a stability metric, and not including any eye gaze samples that do not meet the stability metric in evaluating the plurality of the eye gaze samples.

6. The method of claim 1, further comprising:

in response to selecting the second user interface object, providing visual feedback of the selecting of the second user interface object on the graphical user interface.

7. The method of claim 1, wherein the time window is a first time window, wherein selecting the different user interface object comprises

selecting the different user interface object if the different user interface object is nearest to the combined location of the different plurality of the eye gaze samples, and undoing the selecting of the second user interface object.

8. The method of claim 7, further comprising:

upon evaluation of the different plurality of eye gaze samples in the second time window, removing evaluated eye gaze samples in the first time window and the second time window from being eligible for future evaluations of eye gaze samples.

9. The method of claim 7, wherein the selection input is a first selection input, the method further comprising:

receiving a second selection input after the first selection input is received; and

if the second selection input is detected before the different plurality of the eye gaze samples are evaluated, then shortening the second time window.

10. A data-holding device, comprising an eye tracking system, holding instructions executable by a logic device to:

sample eye gaze samples via the eye tracking system;

store the eye gaze samples;

detect a selection input made via a user input mechanism other than the eye tracking system, at a time when an eye gaze location intersects a first user interface object;

upon receipt of the selection input, evaluate a plurality of the eye gaze samples at different eye gaze locations and at different times within a time window selected based on the time at which the selection input is detected to determine a combined location of the plurality of the eye gaze samples;

when the combined location of the plurality of the eye gaze samples is closer to a second user interface object than the first user interface object, then select the second user interface object as a result of the selection input and based on the second user interface object having a highest weighting according to a selection rule;

evaluate a different plurality of the eye gaze samples in the time window to determine a combined location of the different plurality of the eye gaze samples; and

confirm the selecting of the second user interface object or select a different user interface object as a result of the selection input based upon the combined location of the different plurality of gaze samples.

11. The data-holding device of claim 10, wherein the highest weighting is assigned to a user interface object that is a nearest user interface object to the combined location at which the plurality of the eye gaze samples intersect with a graphical user interface, the nearest user interface object being the first user interface object when the combined location is closer to the first user interface object than the second user interface object.

12. The data-holding device of claim 10, wherein the time window includes eye gaze samples taken before and after the time at which the selection input is detected.

13. The data-holding device of claim 10, further holding instructions executable by the logic device to:

set a center of the time window to a different time prior to the time at which the selection input is detected.

14. The data-holding device of claim 10, further holding instructions executable by the logic device to:

evaluate the plurality of the eye gaze samples compared to a stability metric, and not include any eye gaze samples that do not meet the stability metric in evaluating the plurality of the eye gaze samples.

15. The data-holding device of claim 10, further holding instructions executable by the logic device to:

in response to selecting the second user interface object, provide visual feedback of selection of the second user interface object on the graphical user interface.

16. The data-holding device of claim 10, wherein the time window is a first time window, and the data-holding device further holds instructions executable by the logic device to:

after the second user interface object is selected, evaluate a different plurality of the eye gaze samples in a second time window that is centered later than the first time window to determine the combined location of the different plurality of the eye gaze samples; and

if the different user interface object is nearest to the combined location of the different plurality of the eye gaze samples, then undo the selecting of the second user interface object and select the different user interface object.

17. The data-holding device of claim 16, further holding instructions executable by the logic device to:

upon evaluation of the different plurality of the eye gaze samples, remove evaluated eye gaze samples in the first time window and the second time window from being eligible for future evaluations of eye gaze samples.

18. The data-holding device of claim 16, wherein the selection input is a first selection input, and the data-holding device further holds instructions executable by the logic device to:

detect a second selection input after the time at which the first selection input is detected;

if the second selection input is detected before the different plurality of the eye gaze samples are evaluated, then shorten the second time window.

19. A head-mounted display device, comprising:

a display configured to display a graphical user interface including one or more user interface objects;

an eye tracking system configured to detect a location at which an eye gaze of a wearer of the head-mounted display device intersects with the display;

a logic device; and

a data-holding device holding instructions executable by the logic device to;

sample eye gaze samples via the eye tracking system;

store the eye gaze samples;

detect a selection input made via a user input mechanism other than the eye tracking system, at a time when an eye gaze location intersects a first user interface object,

upon receipt of the selection input, evaluate a plurality of the eye gaze samples at different eye gaze locations and at different times within a time window selected based on the time at which the selection input is detected to determine a combined location of the plurality of the eye gaze samples, and

when the combined location of the plurality of the eye gaze samples is closer to a second user interface object than the first user interface object, then select the second user interface object as a result of the selection input;

evaluate a different plurality of the eye gaze samples in the designated time window to determine a combined location of the different plurality of the eye gaze samples; and

confirm the selecting of the second user interface object or select a different user interface object as a result of the selection input based upon the combined location of the different plurality of eye gaze samples.

20. The head-mounted display device of claim 19, wherein the time window includes eye gaze samples taken before and after the time at which the selection input is detected, and wherein the first user interface object is selected when the combined location of the plurality of the gaze samples is closer to the first user interface object than the second user interface object.