Method for detecting, evaluating, and analyzing look sequences

• US 6,997,556 B2
• Filed: 09/30/2002
• Issued: 02/14/2006
• Est. Priority Date: 10/01/2001
• Status: Active Grant

• Current Assignee: Viewpoint Sicherheitsforschung- Blickforschung GmbH
• Original Assignee: Ernst Pfleger
• Inventors: Pfleger, Ernst
• Primary Examiner(s): Mantis-Mercader, Eleni
• Assistant Examiner(s): SANDERS JR, JOHN R
• Application Number:
  US10/262,608
• Publication Number:
  US 20030086061A1
• Time in Patent Office: 1,233 Days
• Field of Search: 351/209, 351/210, 351/237, 345 7- 9, 382/103, 382/107, 382/100, 382/106, 600/558, 702/150
• US Class Current: 351/246
• CPC Class Codes: A61B 3/0025 characterised by electronic signal processing, e.g. eye models
• CPC Class Codes: A61B 3/0033 characterised by user input arrangements
• CPC Class Codes: A61B 3/0058 for multiple images
• CPC Class Codes: A61B 3/113 for determining or recording eye movement

Abstract

In a method for detecting, evaluating, and analyzing look sequences of a test person using a look detection system, the field of view of the test person is detected by a first camera, which is directed forwardly and firmly held on the person'"'"'s head, and recorded in a field of view video. The movement of the pupils is detected by a second camera, also firmly held on the person'"'"'s head and recorded in an eye video. The pupil coordinates for each frame of the eye video are determined by automatically measuring the contrast of the pupils relative to the surrounding areas with an image recognition program, and the centroid of the dark area, which corresponds to the pupil center having the pupil coordinates is established. A correlation function between pupil coordinates in the eye video and the coordinates of the corresponding visual point is determined in the field of view video. Subsequently, the coordinates of the corresponding visual point in the field of view video are extrapolated from the pupil coordinates for each frame in the eye video.

Claims

1. A method for detecting, evaluating, and analyzing look sequences of a test person in an environment using a look detection system comprising the steps of:

detecting a field of view of the test person with a first test camera, said camera pointing forward and being firmly held on the head of the test person so as to provide the test person an unimpeded view of the environment, said field of view being recorded in frames of a field of view video,

detecting a pupil image of at least one of the pupils in the eyes of the test person with a second camera, said second camera being also firmly held on the head of the test person, said pupil image being recorded in frames of an eye video;

recording each frame of the eye video at the same time as a corresponding frame of the field of view video is recorded;

automatically determining pupil coordinates for a center of said pupil image in each frame of the eye video;

instructing the test person to look at a respective given object in the environment at at least one respective given time, a respective given frame of the eye video corresponding to each respective given time when the test person looks at said respective given object, said respective given object being located at a respective given visual point in the field of view shown in each corresponding frame of the field of view video;

determining location coordinates for said respective given visual point in each corresponding frame of the field of view video;

establishing a correlation function between the pupil coordinates in given frames of the eye video and the location coordinates of said respective given visual point in the field of view shown in each corresponding frame of the field of view video; and

extrapolating from pupil coordinates determined from a selected frame of the eye video the location coordinates of the respective selected visual point in the corresponding frame of the field of view video using said correlation function, the location coordinates of said respective selected visual point indicating where in the environment the test person was looking at a time other than a given time.

2. The method of claim 1, wherein the pupil coordinates are manually corrected after the pupil coordinates have been determined automatically.

4. The method of claim 1, wherein location coordinates for the visual where the respective given object is located in a corresponding frame of the field of view video are automatically determined using an image recognition method.

5. The method of claim 1, wherein location coordinates of the visual point where the respective given object is located in a corresponding frame of the field of view video are determined by a mouse click.

6. The method of claim 1, wherein the at least one look sequence is obtained by rotating the head of the test person to multiple head positions while the test person looks at a respective given object.

7. The method of claim 1, wherein the at least one look sequence includes between 25 and 100 given object positions.

8. The method of claim 1, further comprising the step of generating a result video from the field of view video and the location coordinates of the visual point in corresponding frames of the field of view video, said location coordinates having been extrapolated using the established correlation function, said result video indicating the visual point by a clearly visible marking.

9. The method of claim 1, further comprising the steps of

providing respective definitions for a plurality of categories for objects shown in the field of view video, and

determining which category definition is applicable to an object located at the respective visual point in a corresponding frame of the field of view video.

10. The method of claim 9, further comprising the steps of:

defining a diagram having a time axis and a category axis; and

providing in the diagram a mark for the respective visual point in each corresponding frame at a respective location on the diagram corresponding to the recording time of the corresponding frame on the time axis and the category applicable to an object shown at the respective visual point in said corresponding frame of the field of view video on the category axis, thereby producing a compressed sequential representation of the look sequence.

11. The method of claim 9, further comprising the steps of:

identifying in which respective corresponding frames a given category definition is applicable to the respective visual point, determining a dwell time of the visual point in said given category by adding the duration time of each of said identified respective corresponding frames;

providing a field of view image corresponding to at least a portion of a field of view shown in said respective corresponding frames; and

providing a respective graphic element in said field of view image where an object in each respective category is located, said graphic element having at least one parameter that is proportional to the dwell time so that a compressed representation of a dwell time of the visual point in a specific category is produced.

12. The method of claim 9, further comprising the steps of

providing a field of view image corresponding to at least a portion of a field of view shown in said respective corresponding frames of a look sequence;

and

indicating the sequence of visual points recorded in the look sequence with respective arrows, so that a compressed representation of the look sequence is produced.

13. The method of claim 1 further comprising the steps of:

recording a view of an environment; and

displaying said recorded view of said environment back to the test person so that said recorded view provides a portion of the environment of the test person.

15. The method of claim 1, wherein the at least one given object is a control point mark, said method further comprising the step of placing a control point mark in the environment.

17. The method of claim 6, wherein said at least one look sequence includes between 25 and 100 head positions.

18. The method of claim 9 further comprising the step of assigning a visual point on a frame of the field of view video to a null category.

3. A method for detecting, evaluating, and analyzing look sequences of a test person in an environment using a look detection system, comprising the steps of:

detecting a field of view of the test person with a first test camera, said first camera pointing forward and being firmly held on the head of the test person, said field of view being recorded in frames of a field of view video;

detecting a pupil image of at least one of the pupils in the eyes of the test person with a second camera, said second camera being also firmly held on the head of the test person, said pupil image being recorded in frames of an eye video;

recording each frame of the eye video at the same time as a corresponding frame of the field of view video;

automatically determining pupil coordinates for a centroid of said pupil image in each frame of the eye video;

instructing the test person to look at a respective given object in the environment at at least one respective given time, a respective given frame of the eye video corresponding to each respective given time when the test person looks at said respective given object, said respective given object being located at a respective given visual point in the field of view shown in each corresponding frame of the field of view video;

recording at least one look sequence, said look sequence providing a plurality of respective given frames of an eye video and corresponding frames of a field of view video;

determining location coordinates for said respective given visual point in each corresponding frame of the field of view video;

associating the pupil coordinates in each given frame of the eye video with location coordinates of the respective visual point where the respective given object is located in the corresponding frame of the field of view video in a look sequence;

storing the associated pupil coordinates and location coordinates as a data set;

establishing a correlation function between said pupil coordinates in said given frames of the eye video and said location coordinates of the respective visual point in said corresponding frames from said data set; and

extrapolating from pupil coordinates determined from a selected frame of the eye video the location coordinates of the respective selected visual point in the corresponding frame of the field of view video using said correlation function, the location coordinates of said respective selected visual point indicating where in the environment the test person was looking at a time in the look sequence other than a given time.

14. The method of claim 3, wherein the pupil coordinates and the coordinates of the corresponding visual point are correlated from said data sets through quadratic regression.

16. The method of claim 3, further comprising the step of detecting non-linearity in said data sets.