Method and apparatus for evaluating results of gaze detection

• US 9,639,745 B2
• Filed: 03/21/2013
• Issued: 05/02/2017
• Est. Priority Date: 03/22/2012
• Status: Active Grant

• Current Assignee: SensoMotoric Instruments GmbH (Apple Inc.)
• Original Assignee: SensoMotoric Instruments GmbH (Apple Inc.)
• Inventors: Williams, Denis, Hoffmann, Jan
• Primary Examiner(s): Vu, Kim
• Assistant Examiner(s): DELANEY, MOLLY K
• Application Number:
  US14/387,026
• Publication Number:
  US 20150063635A1
• Time in Patent Office: 1,503 Days
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

The invention relates to a method and an apparatus for evaluating results of gaze detection, wherein these results are present or are obtained in the form of information which defines, for each of a multiplicity of successive times, a viewing direction detected at this time and a focal point identified thereby in a scene image assigned to this time. For this purpose, the invention provides for the following steps to be carried out: —a temporal change in the viewing direction and/or the focal point is evaluated in order to identify different viewing events which differ from one another by different speeds of an eye movement, wherein saccades and fixations and/or pursuit movements are detected as different types of viewing events and the identified viewing events are classified according to the type thereof, —a period of time spanned by the times is divided into intervals in such a manner that an interval corresponding to a duration of the particular viewing event is assigned to each of the identified viewing events, wherein at least some of these intervals each contain a sequence of a plurality of times, —precisely one of the times or a true subset of the times is selected in each case from each of the intervals assigned to a fixation or a pursuit movement, —and, for each of these selected times, the focal point identified in the scene image assigned to the particular time is mapped to a position corresponding to this focal point in a reference image.

Claims

1. A method for evaluating results of an eye tracking, these results being present or obtained as information which defines, for each of a large number of successive moments, respectively a gaze direction detected at this moment and a consequently identified visual point in a scene image assigned to this moment, the scene images being time-dependent such that the scene images assigned to the different moments are different and the method comprising the following steps:

evaluating a temporal change in the gaze direction and/or in the visual point and/or in an eye movement for identifying different visual events, at least two different types of visual events being recognized and the identified visual events being classified according to their type,

dividing a period of time comprising the large number of successive moments into intervals such that an interval, which corresponds to a duration of the respective visual event, is assigned to each of the identified visual events,

selecting a moment or proper subset of moments from each of the intervals which are assigned to the respective visual event of at least one selected type of visual event, and

for the selected moment or proper subset of moments in each of the intervals, determine a location of the visual point identified in the scene image, respective visual points corresponding to the respective visual events of the respective intervals, and map a respective determined location of the respective visual point for the respective selected moment or proper subset of moments in each of the intervals onto a position in a reference image corresponding to the respective determined location of the respective visual point of the interval in the scene image.

2. The method according to claim 1, wherein, during evaluation, visual events in which information intake is effected, and visual events in which no information intake is effected, are recognized as different types of visual events, the selection respectively of precisely one of the moments or of a subset of the moments being effected for each of the intervals which are assigned to a visual event with information intake.

3. The method according to claim 1, wherein the different visual events are differentiated from each other by different speeds and/or amplitudes of an eye movement and/or in that saccades are recognized as visual events in which no information intake is effected, and/or in that fixations and/or consequent movements are recognized as visual events in which information intake is effected.

4. The method according to claim 1, wherein, for each of the intervals which are assigned to a visual event of a selected type, a position in the reference image together with information about the duration of the respective visual event is stored as imaging target, this position being chosen

as the position onto which the visual point, which is defined for the moment selected from this interval, is imaged provided that precisely one of the moments is selected from this interval, or

as one of the positions or an average of the positions onto which the visual points, which are defined for the moments selected from this interval, are imaged provided that a plurality of the moments is selected from this interval.

5. The method according to claim 1, wherein the scene images at the mentioned moments are recorded with a camera, the gaze directions being detected by means of an eye tracker which determines the visual points in the scene images as a function of the thus obtained information about the gaze directions, the camera being thereby worn preferably by the head of a test subject whose gaze directions are detected by the eye tracker.

6. The method according to claim 1, wherein the reference image is an image in a scene visible in the scene images which corresponds to a fixed perspective, or is an abstract reproduction or coding of characteristic features which can be detected in the scene images.

7. The method according to claim 1, wherein, for each of the selected moments, respectively the scene image assigned to this moment is output for manual implementation or for visual checking of the location of the visual point into the reference image.

8. The method according to claim 1, wherein the location of the visual points onto the positions is determined automatically by

identifying a plurality of characteristic features and locations of these features in the reference image by means of an algorithm for feature detection,

identifying these features and determining locations of these features in the respective scene image by applying the same or a corresponding algorithm to the scene image,

determining a transformation which images the locations of the features in the scene image onto the corresponding locations in the reference image, and

applying this transformation to the visual point identified in the respective scene image.

9. A device for evaluating results of an eye tracking, these results being present or being able to be read in or produced as information which defines, for each of a large number of successive moments, respectively a gaze direction detected at this moment and a consequently identified visual point in a scene image assigned to this moment, the scene images being time-dependent so that the scene images assigned to the different moments are different, and the device being equipped for implementing the following steps:

evaluating a temporal change in the gaze direction and/or in the visual point and/or an eye movement for identifying different visual events and recognizing at least two different types of visual events and also classifying the identified visual events according to their type,

dividing a period of time comprising the large number of successive moments into intervals such that an interval, which corresponds to a duration of the respective visual event, is assigned to each of the identified visual events,

outputting the information defining the visual point in the scene image for a proper subset of selected moments from each of the intervals which are assigned to the respective visual event of at least one selected type, and

for each of the selected moments in each of the intervals, determining a location of the visual point identified in the scene image, respective visual points corresponding to the respective visual events of the respective intervals, and mapping a respective determined location of the respective visual point for the respective proper subset of selected moments in each of the intervals onto a position in a reference image and also storing information assigned to this position as an attribute, about the duration of the visual event or interval to which this proper subset of selected moment belongs.

10. The device according to claim 9, wherein it is equipped to recognize visual events in which information intake is effected, and visual events in which no information intake is effected as different types of visual events.

11. The device according to claim 9, wherein it is equipped, for dividing the period of time into the intervals, to assign respectively an attribute, which indicates to which of the intervals or visual events this moment belongs, to each of the moments.

12. The device according to claim 9, wherein the device is equipped to identify the different visual events as a function of different speeds and/or amplitudes of an eye movement and/or to recognize saccades as visual events in which no information intake is effected and/or to recognize fixations and/or consequent movements as visual events in which information intake is effected.

13. The device according to claim 9, wherein it is equipped to image the visual points defined for the selected moments automatically onto positions in the reference image corresponding to these image points and to store the thus determined positions as the images of the visual points.

14. The device according to claim 13, wherein it is equipped, for automatic imaging of the visual points onto the corresponding positions in the reference image, to implement the following steps:

identifying a plurality of characteristic features and locations of these features in the reference image by means of an algorithm for feature detection,

identifying these features and determining locations of these features in the respective scene image by applying the same or a corresponding algorithm to the scene image,

determining a transformation which images the locations of the features in the scene image onto the thus identified locations of the respective features in the reference image, and

applying this transformation to the visual point identified in the respective scene image.

15. The device according to claim 9, wherein it is equipped, for each of the selected moments, to store the corresponding position in the reference image together with the attribute assigned to these positions respectively as a function of an input which defines the position and links this to the moment or to the scene image assigned to this moment.

16. The device according to claim 9, wherein it is equipped, for an input defining the selected moments as such or for an automatic selection of these moments, by applying a specification defining these moments as a function of the intervals.

17. The device according to claim 9, wherein it has furthermore an eye tracker for detecting the gaze directions with a camera for recording the scene images and/or a camera for recording the reference image.

18. The device according to claim 9, further comprising an eye tracker for detecting the gaze directions, the eye tracker including a camera for recording the scene images and a mounting carrying said camera, the mounting being configured for being fixed on or to a head of a test subject.

19. A method for evaluating results of an eye tracking, these results being present or obtained as information which defines, for each of a large number of successive moments, respectively a gaze direction of a test subject detected at this moment and a thereby identified visual point in a scene image assigned to this moment, the scene images being time-dependent, wherein the scene images assigned to the different moments differ from each other due to a head movement of the test subject, the method comprising the following steps:

evaluating a temporal change in the gaze direction and/or in the visual point and/or in an eye movement for identifying different visual events, at least two different types of visual events being recognized and the identified visual events being classified according to their type,

dividing a period of time comprising the large number of successive moments into intervals such that an interval, which corresponds to a duration of the respective visual event, is assigned to each of the identified visual events,

selecting a moment or proper subset of moments from each of the intervals which are assigned to the respective visual event of at least one selected type of visual event, and

for the selected moment or proper subset of moments in each of the intervals, determine a location of the visual point identified in the scene image, respective visual points corresponding to the respective visual events of the respective intervals, and map a respective determined location of the respective visual point for the respective selected moment or proper subset of moments in each of the intervals onto a position in a reference image corresponding to the respective determined location of the respective visual point of the interval in the scene image.