Method to reliably detect correlations between gaze and stimuli

• US 10,528,131 B2
• Filed: 05/16/2018
• Issued: 01/07/2020
• Est. Priority Date: 05/16/2018
• Status: Active Grant

• Current Assignee: Tobii Technology AB
• Original Assignee: Tobii Technology AB
• Inventors: Davies, Alexander, Gordon, Maria, Stoltz, Per-Edvin
• Primary Examiner(s): Dicke, Chad M
• Application Number:
  US15/981,629
• Publication Number:
  US 20190354172A1
• Time in Patent Office: 601 Days
• Field of Search: None
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence

Abstract

A system and techniques for calibrating an eye tracking system are described. The system can update the calibration of personal calibration parameters continuously based on a user'"'"'s gaze on a user interface, following user interface stimulus events. The system improves continuous calibration techniques by determining an association between the user'"'"'s eye sequences and the stimulus events, and updates the personal calibration parameters accordingly. A record indicative of a user gaze, including eye sequences, such as eye movements or eye fixations, is maintained over a time period. A user interface stimulus event associated with the user interface and occurring within the time period is detected. An association is determined between the eye sequence and the user interface stimulus event. An interaction observation that includes the eye sequence and a location of the stimulus event is generated. Personal calibration parameters are updated based on the interaction observation.

Claims

1. A method for calibrating an eye tracking system, the method performed by the eye tracking system and comprising:

updating a record indicative of a user gaze on a user interface over a time period, the user interface associated with the eye tracking system, the record including an eye sequence;

detecting a user interface stimulus event occurring within the time period and associated with the user interface;

determining, based on the record, an association between the eye sequence and the user interface stimulus event;

generating, based on the association, a first interaction observation that includes at least the eye sequence and a location of the user interface stimulus event associated with the user interface;

updating personal calibration parameters for tracking the user gaze based on the first interaction observation, and

generating a personal calibration parameter update decision by;

accessing a set of at least four interaction observations comprising locations of stimulus events and eye sequences, the set including the first interaction observation and a second interaction observation, the second interaction observation comprising a second location of a second stimulus event and a second eye sequence;

selecting the second interaction observation from the set;

constructing trial personal calibration parameters based on the second interaction observation, such that a gaze target projected for the second eye sequence using the trial personal calibration parameters falls proximate to the second location;

projecting gaze targets for remaining interaction observations in the set based on the trial personal calibration parameters and the eye sequences of the remaining interaction observations;

detecting an inlier subset of the remaining interaction observations, such that a respective projected gaze for a respective inlier falls within a first threshold value from the location of the respective inlier;

detecting an outlier subset of the remaining interaction observations, such that a respective projected gaze for a respective outlier falls greater than the first threshold value from the location of the respective outlier;

computing a ratio of a number of inliers to a number of outliers; and

responsive to the ratio being greater than a second threshold value, generating a decision to update the personal calibration parameters; and

responsive to the ratio being within the second threshold value, discarding the trial personal calibration parameters and constructing second trial personal calibration parameters based on a second set of interaction observations.

2. The method of claim 1, wherein determining the association between the eye sequence and the user interface stimulus event comprises:

determining, based on a type of the user interface stimulus event, a user gaze time interval within the time period of the record;

identifying, from the record, that the eye sequence occurred during the user gaze time interval; and

determining the association based on the identifying of the eye sequence.

3. The method of claim 2, wherein determining the user gaze time interval comprises:

determining, based on a type of the application, an expected type of candidate user interface stimulus events; and

determining that the type of the user interface stimulus event is the expected type.

4. The method of claim 1, further comprising generating a personal calibration parameter update decision by performing an update check using the first interaction observation, the update check comprising one or more of:

Random Sample Consensus (RANSAC);

Maximum Likelihood Estimate Sample Consensus (MLESAC);

Maximum A Posteriori Sample Consensus (MAPSAC);

a KALMANSAC procedure; and

machine learning.

5. The method of claim 1, wherein updating the personal calibration parameters comprises:

determining a difference between the trial personal calibration parameters and the personal calibration parameters;

responsive to the difference being greater than a third threshold value, replacing the personal calibration parameters with the trial personal calibration parameters; and

responsive to the difference being within the third threshold value, keeping the personal calibration parameters.

6. The method of claim 1, wherein the user interface stimulus event includes one or more of:

a user mouse click;

a user screen tap;

a user selection;

a user keyboard input;

a user joystick or game pad input;

a user hand controller input;

a displayed cursor;

a displayed icon;

a displayed notification; and

a displayed animation.

7. The method of claim 1, wherein detecting the user interface stimulus event occurring within the time period further comprises:

accessing a list of expected events corresponding to an underlying application implemented on the user interface by a computing device separately from the eye tracking system; and

detecting that the user interface stimulus event matches an expected event in the list.

8. The method of claim 1, wherein:

the nature of the user interface stimulus event is dynamic;

the nature being dynamic denotes the user interface stimulus event has a motion or a displayed animation originating in the user interface; and

determining the association between the eye sequence and the user interface stimulus event further comprises identifying that a pattern of movement of the eye sequence corresponds to a similar pattern of the motion or the displayed animation.

9. The method of claim 1:

further comprising generating a personal calibration parameter update decision based on;

a plurality of interaction observations including the first interaction observation; and

a consistency among a plurality of eye model parameter fits for the plurality of interaction observations or a distance between respective stimulus events and eye sequence locations of the plurality of interaction observations; and

wherein updating the personal calibration parameters is based on the personal calibration parameter update decision.

10. A non-transitory computer-readable storage medium storing instructions, that upon execution on a computer system, cause the computer system to perform a method for calibrating an eye tracking system, the method comprising:

updating a record indicative of a user gaze on a user interface over a time period, the user interface associated with the eye tracking system, the record including an eye sequence;

detecting a user interface stimulus event occurring within the time period and associated with the user interface;

determining, based on the record, an association between the eye sequence and the user interface stimulus event;

generating, based on the association, a first interaction observation that includes at least the eye sequence and a location of the user interface stimulus event associated with the user interface;

updating personal calibration parameters for tracking the user gaze based on the first interaction observation, and

generating a personal calibration parameter update decision by;

accessing a set of at least four interaction observations comprising locations of stimulus events and eye sequences, the set including the first interaction observation and a second interaction observation, the second interaction observation comprising a second location of a second stimulus event and a second eye sequence;

selecting the second interaction observation from the set;

constructing trial personal calibration parameters based on the second interaction observation, such that a gaze target projected for the second eye sequence using the trial personal calibration parameters falls proximate to the second location;

projecting gaze targets for remaining interaction observations in the set based on the trial personal calibration parameters and the eye sequences of the remaining interaction observations;

detecting an inlier subset of the remaining interaction observations, such that a respective projected gaze for a respective inlier falls within a first threshold value from the location of the respective inlier;

detecting an outlier subset of the remaining interaction observations, such that a respective projected gaze for a respective outlier falls greater than the first threshold value from the location of the respective outlier;

computing a ratio of a number of inliers to a number of outliers; and

responsive to the ratio being greater than a second threshold value, generating a decision to update the personal calibration parameters; and

responsive to the ratio being within the second threshold value, discarding the trial personal calibration parameters and constructing second trial personal calibration parameters based on a second set of interaction observations.

11. The non-transitory computer-readable storage medium of claim 10, wherein determining the association between the eye sequence and the user interface stimulus event comprises:

determining, based on a type of the user interface stimulus event, a user gaze time interval within the time period of the record;

identifying, from the record, that the eye sequence occurred during the user gaze time interval; and

determining the association based on the identifying of the eye sequence.

12. The non-transitory computer-readable storage medium of claim 10, wherein the location of the associated stimulus event is two-dimensional or three-dimensional.

13. The non-transitory computer-readable storage medium of claim 10, wherein the record is stored in a circular buffer.

14. The non-transitory computer-readable storage medium of claim 13, wherein a size of the circular buffer depends on a type of application presenting the user interface stimulus event.

15. An eye tracking system, comprising:

a camera;

a processor; and

a memory storing computer-readable instructions that, upon execution by the processor, cause the eye tracking system to perform a method for calibrating the eye tracking system, the method comprising;

updating a record indicative of a user gaze on a user interface over a time period, the user interface associated with the eye tracking system, the record including an eye sequence;

detecting a user interface stimulus event occurring within the time period and associated with the user interface;

determining, based on the record, an association between the eye sequence and the user interface stimulus event;

generating, based on the association, a first interaction observation that includes at least the eye sequence and a location of the user interface stimulus event associated with the user interface;

updating personal calibration parameters for tracking the user gaze based on the first interaction observation, and

generating a personal calibration parameter update decision by;

accessing a set of at least four interaction observations comprising locations of stimulus events and eye sequences, the set including the first interaction observation and a second interaction observation, the second interaction observation comprising a second location of a second stimulus event and a second eye sequence;

selecting the second interaction observation from the set;

constructing trial personal calibration parameters based on the second interaction observation, such that a gaze target projected for the second eye sequence using the trial personal calibration parameters falls proximate to the second location;

projecting gaze targets for remaining interaction observations in the set based on the trial personal calibration parameters and the eye sequences of the remaining interaction observations;

detecting an inlier subset of the remaining interaction observations, such that a respective projected gaze for a respective inlier falls within a first threshold value from the location of the respective inlier;

detecting an outlier subset of the remaining interaction observations, such that a respective projected gaze for a respective outlier falls greater than the first threshold value from the location of the respective outlier;

computing a ratio of a number of inliers to a number of outliers; and

responsive to the ratio being greater than a second threshold value, generating a decision to update the personal calibration parameters; and

responsive to the ratio being within the second threshold value, discarding the trial personal calibration parameters and constructing second trial personal calibration parameters based on a second set of interaction observations.

16. The eye tracking system of claim 15, wherein determining the association between the eye sequence and the user interface stimulus event further comprises one or more of the following:

identifying, from the record, that the user gaze was stable during an eye fixation time interval within the time period of the record;

or

identifying, from the record, that a pattern of movement of the eye sequence corresponded to a similar movement pattern of the user interface stimulus event.

17. The eye tracking system of claim 15, wherein updating the personal calibration parameters comprises generating the personal calibration parameters without performing an initial calibration.

18. The eye tracking system of claim 15, wherein

detecting the user interface stimulus event further comprises one or more of:

copying the record of the user gaze to a storage immediately for later analysis;

copying the record of the user gaze to a storage after a predetermined time period for later analysis;

copying the record of the user gaze to a storage after a variable time period for later analysis;

or

suspending changes to the record of the user gaze.

19. The eye tracking system of claim 15, wherein determining the association between the eye sequence and the user interface stimulus event is based on one or more of:

a distance from the location of the stimulus event to a projected gaze for the eye sequence;

a duration of the eye sequence;

a precision of tracking the eye sequence;

or

a correlated movement within the eye sequence.