Detection of gaze point assisted by optical reference signal

• US 9,495,589 B2
• Filed: 01/26/2009
• Issued: 11/15/2016
• Est. Priority Date: 01/26/2009
• Status: Active Grant

• Current Assignee: Tobii Technology AB
• Original Assignee: Tobii Technology AB
• Inventors: Holtz Elvesj, John Mikael, Strombom, Johan, Skogo, Marten, Nystedt, Per, Gustafsson, Simon, Blixt, Peter
• Primary Examiner(s): Patel, Jay
• Assistant Examiner(s): Habib, Irfan
• Application Number:
  US13/145,636
• Publication Number:
  US 20110279666A1
• Time in Patent Office: 2,850 Days
• Field of Search: 348/78
• US Class Current: 1/1
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

A gaze-point detection system includes at least one infrared (IR) signal source to be placed in a test scene as a reference point, a pair of eye glasses to be worn by a person, and a data processing and storage unit for calculating a gaze point of the person wearing the pair of eye glasses. The pair of eye glasses includes an image sensor, an eye-tracking unit and a camera. The image sensor detects IR signals from the at least one IR signal source and generates an IR signal source tracking signal. The eye-tracking unit determines adapted to determine the gaze direction of the person and generates an eye-tracking signal, and the camera acquires a test scene picture. The data processing and storage unit communicates with the pair of eye glasses and calculates the gaze point relative to the test scene picture.

Claims

1. A pair of eye glasses comprising:

a frame;

an image reference sensor carried by said frame and configured to

receive incident infrared (IR) signals from at least one IR signal source arranged in a region of space, and external to the pair of eye glasses

calculate at least one angle between an optical axis of said image reference sensor and a line of sight between said image reference sensor and said IR signal source, and

generate at least one IR signal source tracking signal that includes the at least one calculated angle and corresponds to a location of the at least one IR signal source relative to the pair of eye glasses;

an eye-tracking unit carried by said frame and comprising

at least one IR-light-emitting device configured to emit a light pulse toward an eye of a person wearing the pair of eye glasses,

a detector configured to receive a light pulse reflected from the eye and to repeatedly acquire an image of the eye,

said at least one IR-light-emitting device and said detector being arranged non-coaxially, and

an evaluation unit connected to said detector and configured to determine a gaze direction of the eye toward the region of space based on information extracted from the acquired image of the eye including location of a reflection on a cornea of the eye from the emitted light pulse and position of a pupil of the eye, and to generate an eye-tracking signal corresponding to the detected gaze direction; and

a projection carried by said frame and communicatively connected to a data processing and storage unit and configured to output the IR signal source tracking signal and the eye-tracking signal thereto, with the data processing and storage unit configured to calculate a gaze point of the person based on the at least one IR signal source tracking signal and the eye-tracking signal.

2. The pair of eye glasses according to claim 1, further comprising a camera carried by said frame and configured to acquire at least one picture of the region of space.

3. The pair of eye glasses according to claim 2, for use in a system comprising the data processing and storage unit, which is further configured to receive the at least one picture, said data processing and storage unit configured to calculate the gaze point of the person with respect to the at least one picture based on the at least one IR signal source tracking signal, the eye-tracking signal and the at least one picture.

4. The pair of eye glasses according to claim 3, wherein said image reference sensor and said eye-tracking unit are communicatively coupled to a support unit comprising at least one of the following:

a voltage source, a data buffer, a data pre-processor, and a transceiver.

5. The pair of eye glasses according to claim 1, wherein said image reference sensor comprises a sensor surface and an optical aperture configured to receive incident IR signals, and wherein said image reference sensor is configured to calculate the at least one angle between an incident IR signal and the optical axis of the image reference sensor by determining a position of the incident IR signal on said sensor surface.

6. The pair of eye glasses according to claim 1, further comprising a first polarization filter to be positioned in front of the eye and configured to pass through polarized light in a first direction, and a second polarization filter positioned in front of said detector and configured to pass through polarized light in a second direction that is essentially orthogonal to the first direction.

7. The pair of eye glasses according to claim 1, wherein said image reference sensor comprises:

a first optical line sensor having a scanning direction that deviates slightly from a vertical direction when the pair of eye glasses are worn; and

a second optical line sensor having a scanning direction that is substantially perpendicular to that of said first optical line sensor;

with scanning directions of said first and second optical line sensors both being substantially parallel to a frontal plane of the person wearing the pair of eye glasses.

8. A system for determining a gaze point of a person relative to a picture of a region of space, the system comprising:

at least one infrared (IR) signal source located in the region of space and configured to emit an IR signal;

a pair of eye glasses to be worn by a person, said pair of eye glasses comprising

a frame;

an image reference sensor carried by said frame and configured to receive incident IR signals from said at least one IR signal source arranged in the region of space and external to said pair of eye glasses, to calculate at least one angle between an optical axis of said image reference sensor and a line of sight between said sensor and said IR signal source, and to generate at least one IR signal source tracking signal that includes the calculated angle and corresponds to a location of said IR signal source relative to said pair of eye glasses, and

an eye-tracking unit carried by said frame and comprising

at least one IR-light-emitting device configured to emit a light pulse directed toward an eye of the person wearing said pair of glasses,

a detector configured to receive a light pulse reflected from the eye and to repeatedly acquire an image of the eye, and

an evaluation unit connected to said detector and configured to determine a gaze direction of the eye toward the region of space based on information extracted from the acquired image including location of a reflection on a cornea of the eye from the emitted light pulses and position of a pupil of the eye, and

said at least one IR-light-emitting device and said detector being arranged non-coaxially; and

a data processing and storage unit configured to communicate with said pair of eye glasses to obtain the at least one IR signal source tracking signal and the eye-tracking signal, and to calculate the gaze point of the person with respect to the picture of the region of space based on the at least one IR signal source tracking signal, the eye-tracking signal and the picture.

9. The system according to claim 8, further comprising a scene camera configured to acquire at least one picture of the region of space.

10. The system according to claim 8, wherein said at least one pair of eye glasses further comprises a camera carried by said frame and configured to acquire at least one picture of the region of space.

11. The system according to claim 10, wherein said data processing and storage unit is configured to match a picture acquired by said camera to information from the at least one IR signal source tracking signal to create a combined image of the acquired picture and a position relative thereto of said at least one IR signal source.

12. The system according to claim 8, wherein said at least one IR signal source comprises at least four IR signal sources;

and wherein said data processing and storage unit is configured to further match the picture to information on positions of said at least four IR signal sources, and to store information of the positions of said at least four IR signal sources.

13. The system according to claim 8, wherein said data processing and storage unit is further configured to extract and continuously track image features in the acquired picture to detect a change of position of said pair of eye glasses.

14. The system according to claim 8, wherein the IR signal comprises a modulated IR light;

and wherein said data processing and storage unit is configured to determine an identity of a specific IR signal source based on the at least one IR signal source tracking signal.

15. The system according to claim 14, wherein the modulated IR light is modulated based on at least one of time, wavelength and polarization.

16. The system according to claim 8, wherein said image reference sensor comprises a sensor surface configured to receive the incident IR signals, and wherein said image reference sensor is configured to calculate at least one angle between an incident IR signal and the optical axis of the image reference sensor by determining a position of the incident IR signal on the sensor surface.

17. The system according to claim 8, wherein said pair of eye glasses further comprises a first polarization filter to be positioned in front of the eye and configured to pass through polarized light in a first direction, and a second polarization filter positioned in front of said detector and configured to pass through polarized light in a second direction that is essentially orthogonal to the first direction.

18. The system according to claim 8, further comprising a support unit communicatively coupled to said pair of eye glasses and comprising at least one of the following:

a voltage source, a data buffer, a data pre-processor, and a transceiver configured to communicate with said data processing and storage unit and to communicate with said pair of eye glasses.

19. The system according to claim 8, wherein said image sensor comprises:

a first optical line sensor having a scanning direction that deviates slightly from a vertical direction when said pair of eye glasses are worn; and

a second optical line sensor having a scanning direction that is substantially perpendicular to that of said first optical line sensor;

with scanning directions of said first and second optical line sensors both being substantially parallel to a frontal plane of the person when said pair of eye glasses are worn.

20. The system according to claim 8, wherein said at least one IR signal source is operable in an energy-saving mode that is interruptible by receipt of a wireless signal.

21. The system according to claim 8, wherein additional gaze points are determined for additional people relative to the picture of the region of space;

wherein said pair of eye glasses is configured to generate an eye-tracking signal and an IR signal source tracking signal for each person; and

wherein said data processing and storage unit is configured to calculate a gaze point of each person with respect to the picture acquired by a camera in said pair of eye glasses.

22. A method for determining a gaze point of a person relative to a picture of a region of space, and comprising:

(a) emitting an infrared (IR) signal from at least one external IR signal source in the region of space;

(b) receiving at a receiver the at least one IR signal;

(c) determining an angle between an optical axis of said receiver and a line of sight between said receiver and said IR signal source;

(d) acquiring at a camera a picture of the region of space containing the at least one IR signal source;

(e) determining positions of the at least one IR signal source relative to the picture;

(f) determining a gaze point of the person relative to the positions of the at least one IR external signal source in the region of space based the determined angle and on information extracted from images of an eye of the person including location of a reflection on a cornea of the eye from light pulses, emitted from an IR-light-emitting device, and position of a pupil of the eye, wherein said images are acquired by a detector that is non-coaxial with the IR-light-emitting device; and

(g) outputting a combined image of the region of space and the gaze point.

23. The method according to claim 22, wherein the gaze points are determined for a plurality of persons relative to the picture of the region of space, and further comprising performing steps (a) and (d);

and for each person performing steps (b), (c), (e) and (f); and

wherein outputting the combined image of the region of space and the gaze points is based on the plurality of persons.