Targeting saccade landing prediction using visual history

• US 10,895,908 B2
• Filed: 11/26/2018
• Issued: 01/19/2021
• Est. Priority Date: 03/04/2013
• Status: Active Grant

• Current Assignee: Tobii Technology AB
• Original Assignee: Tobii Technology AB
• Inventors: Nijs, Daan
• Primary Examiner(s): Gray, Ryan M
• Application Number:
  US16/199,819
• Publication Number:
  US 20190094963A1
• Time in Patent Office: 785 Days
• Field of Search: None
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/039 Accessories therefor, e.g. mouse pads
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G09G 2320/0261 in the context of movement of objects on the screen or movement of the observer relative to the screen
• CPC Class Codes: G09G 2320/103 Detection of image changes, e.g. determination of an index representative of the image change
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 5/37 Details of the operation on graphic patterns G09G5/38 takes precedence
• CPC Class Codes: G09G 5/373 for modifying the size of the graphic pattern

Abstract

According to the invention, techniques for refining a ballistic prediction are described. In an example, an eye tracking system may record images over time of content presented on a display. Saccade data may be received and used as a trigger to retrieve particular ones of the recoded images. The eye tracking system may compare the images to identify a change in the content. The location of this change may correspond to a sub-area of the display. The output of the ballistic prediction may include a landing point that represents an anticipated gaze point. This landing point may be adjusted such that a gaze point is now predicted to fall within the sub-area when the change is significant.

Claims

1. A method implemented by an eye tracking system, the method comprising:

recording, in storage, a first image and a second image of content presented at a display;

receiving saccade data indicating a user saccade;

accessing, from the storage and based on receiving the saccade data, the first image and the second image;

comparing the first image and the second image accessed from the storage to determine a change in the content;

predicting, based on the saccade data and the change in the content, a gaze point on the display,

receiving a prediction for the gaze point;

determining, based on the comparing, a location of the change, a second change in the content, and a second location of the second change;

determining a first distance between the location of the change and the user saccade and a second distance between the second location of the second change and the user saccade;

selecting the change based on a comparison of the first distance and the second distance; and

updating, based on the selecting, the prediction based on the location of the change such that the gaze point is predicted to be within a sub-area of the display corresponding to the location.

2. The method of claim 1, further comprising:

determining a location of the change based on the comparing of the first image and the second image, wherein the gaze point is predicted based on the location of the change.

3. The method of claim 2, wherein predicting the gaze point comprises:

receiving a prediction for the gaze point, wherein the prediction is based on the saccade data;

determining, based on the saccade data, that the location of the change is within a threshold distance from a direction of the user saccade; and

updating, based on the location being within the threshold distance, the prediction such that the gaze point is predicted to be within a sub-area of the display corresponding to the location.

4. The method of claim 1, wherein the first image and the second image correspond to presented frames of the content.

5. The method of claim 4, wherein comparing the first image and the second image comprises comparing properties of pixels in the first image and the second image, wherein the properties comprise at least one of:

color or luminosity.

6. The method of claim 5, wherein the change in the content is determined based on a comparison of a change in the properties of a subset of the pixels between the first image and the second image to a content change threshold, and wherein a location of the change is determined based on the subset of the pixels.

7. The method of claim 6, wherein the content change threshold is dynamically defined based on a time difference between the content change and a start time of the user saccade.

8. The method of claim 4, wherein each of the first image and the second image has a lower image resolution than the presented frames of the content.

9. The method of claim 1, further comprising:

receiving a prediction for the gaze point;

determining, based on the comparing, a location of the change, a second change in the content, and a second location of the second change;

determining that the location of the change and the second location of the second change are within a threshold distance of the user saccade;

selecting the change based on a speed of the user saccade being less than a speed threshold;

updating, based on the selecting, the prediction based on the location of the change such that the gaze point is predicted to be within a sub-area of the display corresponding to the location.

10. The method of claim 1, wherein the change is presented within a sub-area of the display, and wherein predicting the gaze point comprises:

receiving a prediction for the gaze point; and

updating the prediction based on the change such that the gaze point is predicted to be within the sub-area of the display.

11. The method of claim 10, wherein updating the prediction comprises positioning the gaze point to be within the sub-area of the display based on a speed of the user saccade.

12. The method of claim 10, wherein updating the prediction comprises positioning the gaze point to be within the sub-area of the display based on a saccade landing position indicated by the user saccade.

13. The method of claim 1, wherein the content is presented in frames, wherein the first image corresponds to a most recent frame, and wherein the second image corresponds to a preceding frame.

14. The method of claim 1, wherein the content is presented in frames, wherein the first image corresponds to a first frame, wherein the second image corresponds to second frame that precedes the first frame, and further comprising:

selecting the second image based on a start of the user saccade.

15. The method of claim 1, further comprising:

receiving, from a rendering engine, a location of a graphical object that is first presented in the content at a time associated with a start of the user saccade or that is moving with a presentation of the content at the time associated with the start of the user saccade, wherein the gaze point is further predicted based on the location of the graphical object.

16. One or more non-transitory computer-readable storage media comprising instructions, that upon execution on a system, cause the system to perform steps comprising:

recording, in storage, a first image and a second image of content presented at a display;

receiving saccade data indicating a user saccade;

accessing, from the storage and based on receiving the saccade data, the first image and the second image;

comparing the first image and the second image accessed from the storage to determine a change in the content;

predicting, based on the saccade data and the change in the content, a gaze point on the display;

receiving a prediction for the gaze point;

determining, based on the comparing, a location of the change, a second change in the content, and a second location of the second change;

determining a first distance between the location of the change and the user saccade and a second distance between the second location of the second change and the user saccade;

selecting the change based on a comparison of the first distance and the second distance;

updating, based on the selecting, the prediction based on the location of the change such that the gaze point is predicted to be within a sub-area of the display corresponding to the location.

17. The one or more non-transitory computer-readable storage media of claim 16, wherein the steps further comprise:

determining a location of the change based on the comparing of the first image and the second image, wherein the gaze point is predicted based on the location of the change.

18. A system comprising:

one or more processors; and

one or more non-transitory computer-readable storage media comprising instructions, that upon execution by the one or more processors, configure the system to;

record, in storage, a first image and a second image of content presented at a display;

receive saccade data indicating a user saccade;

access, from the storage and based on receiving the saccade data, the first image and the second image;

compare the first image and the second image accessed from the storage to determine a change in the content; and

predict, based on the saccade data and the change in the content, a gaze point on the display;

receive a prediction for the gaze point;

determine, based on the comparing, a location of the change, a second change in the content, and a second location of the second change;

determine a first distance between the location of the change and the user saccade and a second distance between the second location of the second change and the user saccade;

select the change based on a comparison of the first distance and the second distance;

update, based on the selecting, the prediction based on the location of the change such that the gaze point is predicted to be within a sub-area of the display corresponding to the location.

19. The system of claim 18, wherein predicting the gaze point comprises:

receiving a prediction for the gaze point, wherein the prediction is based on the saccade data;

determining, based on the saccade data, that a location of the change is within a threshold distance from a direction of the user saccade; and

updating, based on the location being within the threshold distance, the prediction such that the gaze point is predicted to be within a sub-area of the display corresponding to the location.