Gaze point detection method and gaze point detection device

• US 9,135,708 B2
• Filed: 08/09/2011
• Issued: 09/15/2015
• Est. Priority Date: 08/09/2010
• Status: Active Grant

• Current Assignee: National University Corporation Shizuoka University
• Original Assignee: National University Corporation Shizuoka University
• Inventors: Ebisawa, Yoshinobu
• Primary Examiner(s): Park, Chan
• Assistant Examiner(s): TRAN, MAI H
• Application Number:
  US13/814,800
• Publication Number:
  US 20130188834A1
• Time in Patent Office: 1,498 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 40/19 Sensors therefor

Abstract

A gaze point detection device 1 comprises two stereo cameras 2a, 2b for acquiring a face image of a subject A, light sources 3a, 3b disposed on the outside of apertures 9a, 9b, a control circuit 4, 5, 6, and an image processor 7. The image processor 7 calculates a vector r from a corneal reflection point to a pupil on a plane perpendicular to reference lines of the stereo cameras 2a, 2b, computes an angle θ of lines of sight of the subject A with respect to the respective reference lines by using a function f₁, corrects the function f₁ such that directions of lines of sight are closer to each other, and calculates the line of sight directions to detect a gaze point Q on a display screen.

Claims

1. A gaze point detection method comprising:

a face image production step of producing a face image of a subject as a bright pupil image and a dark pupil image by using at least two cameras and a light source disposed on the outside of apertures of the at least two cameras;

a vector calculation step of calculating a vector r from a corneal reflection point of the subject to a pupil thereof on a plane perpendicular to a reference line connecting the camera and the pupil according to the face image obtained by each of the at least two cameras;

a line of sight direction calculation step of calculating respective angles θ

of lines of sight of the subject with respect to the reference lines of the at least two cameras according to the vectors r corresponding to the at least two cameras from the following expression (1);

θ

=f(|r|) 

 

(1)

by using a function f;

a parameter correction step of correcting the function f such that at least two gaze points on a predetermined plane determined by the directions of lines of sight calculated so as to correspond to the at least two cameras are closer to each other; and

a gaze point detection step of calculating the directions of lines of sight according to the expression (1) by using the function f corrected by the parameter correction step and determining an intersection of the lines of sight on the predetermined plane, so as to detect a gaze point of the subject on the predetermined plane,

wherein the parameter correction step determines intersections of the lines of sight with respective virtual gaze point planes including positions of at least two cameras and being perpendicular to the reference lines thereof, projects the intersections onto a predetermined projective virtual gaze point plane, and then corrects the function f such that the intersections are closer to each other on the projective virtual gaze point plane.

2. A gaze point detection method according to claim 1, further comprising a pupil coordinate detection step of detecting three-dimensional coordinates of the pupil of the subject according to respective face images obtained by the at least two cameras;

wherein the vector calculation step calculates the vector r on a plane, perpendicular to the reference line of the camera, including the three-dimensional coordinates of the pupil.

3. A gaze point detection method according to claim 1, wherein the line of sight direction calculation step calculates the angle θ

of the line of sight of the subject according to the following expression (2);

θ

=k×

|r| 

 

(2)

by using a parameter k; and

wherein the parameter correction step corrects the parameter k.

4. A gaze point detection method according to claim 3, wherein the parameter correction step changes the gaze point of the subject, so as to compute the parameter k when the vector r is changed variously, thereby determining the parameter k as a function of the magnitude |r| of vector r;

and

wherein the gaze point detection step calculates the direction of the line of sight by using the function and the expression (2).

5. A gaze point detection method according to claim 1, wherein the gaze point detection step detects the gaze point occurring when the subject is caused to gaze at one specified point on the predetermined plane and corrects an intersection of the line of sight with the predetermined plane according to a deviation between the specified point and the gaze point.

6. A gaze point detection method according to claim 1, wherein the gaze point detection step detects the respective gaze points occurring when the subject is caused to gaze at two or more specified points on the predetermined plane;

and

wherein the parameter correction step further corrects the function f according to a deviation between the specified point and the gaze point.

7. A gaze point detection method according to claim 1, wherein the vector calculation step detects a pupil position of the subject by taking a difference between the bright and dark pupil images.

8. A gaze point detection device for detecting a gaze point of a subject on a predetermined plane according to a face image of the subject, the device comprising:

at least two cameras for acquiring the face image of the subject;

a light source disposed on the outside of apertures of the at least two cameras;

a control circuit for controlling the cameras and light source; and

an image processing unit for processing image signals issued from the at least two cameras;

wherein the control circuit performs such control as to produce the face image of the subject as a bright pupil image and a dark pupil image;

wherein the image processing unit calculates a vector r from a corneal reflection point of the subject to a pupil thereof on a plane perpendicular to a reference line connecting the camera and the pupil according to the face image obtained by each of the at least two cameras,

calculates respective angles θ

of lines of sight of the subject with respect to the reference lines of the at least two cameras according to the vectors r corresponding to the at least two cameras from the following expression (1);

θ

=f(|r|) 

 

(1)

by using a function f,

corrects the function f such that at least two gaze points on a predetermined plane determined by the directions of lines of sight calculated so as to correspond to the at least two cameras are closer to each other, and

calculates the directions of lines of sight according to the expression (1) by using the corrected function f and determines an intersection of the lines of sight on the predetermined plane, so as to detect a gaze point of the subject on the predetermined plan;

wherein the image processing unit determines intersections of the lines of sight with respective virtual gaze point planes including positions of the at least two cameras and being perpendicular to the reference lines thereof, projects the intersections onto a predetermined projective virtual gaze point plane, and then corrects the function f such that the intersections are closer to each other on the projective virtual gaze point plane.