Eye gaze tracking method and apparatus and computer-readable recording medium

• US 9,182,819 B2
• Filed: 11/17/2014
• Issued: 11/10/2015
• Est. Priority Date: 04/10/2014
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Strupczewski, Adam, Naruniec, Jacek, Mucha, Kamil, Czuprynski, Blazej
• Primary Examiner(s): BHATNAGAR, ANAND P
• Application Number:
  US14/543,213
• Publication Number:
  US 20150293588A1
• Time in Patent Office: 358 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction

Abstract

A method and an apparatus of tracking an eye gaze to determine where a gaze point of a user is located on a display unit of a device, based on a facial pose and a position of an iris center, are provided. The method includes detecting a facial feature in a captured initial image, three-dimensionally modeling the detected facial feature, tracking the three-dimensionally modeled facial feature in consecutively captured images, detecting an iris center in the consecutively captured images, acquiring an eye gaze vector based on the tracked three-dimensionally modeled facial feature and the detected iris center, and acquiring a gaze point on a display unit based on the eye gaze vector.

Claims

1. A method of tracking an eye gaze, the method comprising:

detecting a facial feature in a captured initial image;

three-dimensionally modeling the detected facial feature;

tracking the three-dimensionally modeled facial feature in consecutively captured images;

detecting an iris center in the consecutively captured images;

acquiring an eye gaze vector based on the tracked three-dimensionally modeled facial feature and the detected iris center; and

acquiring a gaze point on a display unit based on the eye gaze vector.

2. The method of claim 1, wherein the detecting of the facial feature comprises displaying a message requesting a user to gaze at a certain point of the display unit.

3. The method of claim 1, wherein the three-dimensional modeling of the detected facial feature comprises:

calculating an eyeball center based on the detected facial feature; and

three-dimensionally modeling the facial feature including the calculated eyeball center.

4. The method of claim 3, wherein the calculating of the eyeball center is performed based on statistics of a facial size and an eyeball diameter.

5. The method of claim 1, wherein the tracking of the three-dimensionally modeled facial feature comprises:

determining a facial pose based on the three-dimensionally modeled facial feature; and

tracking the facial pose.

6. The method of claim 1, wherein the detecting of the iris center comprises detecting the iris center by using only an eye region of the three-dimensionally modeled facial feature.

7. The method of claim 1, wherein the detecting of the iris center comprises calculating a position of the iris center on the three-dimensionally modeled facial feature.

8. The method of claim 1, wherein the detecting of the iris center comprises:

detecting a position of an iris of a left eye and a position of an iris of a right eye; and

determining a weight by estimating a quality of the detected position of the iris of the left eye and a quality of the detected position of the iris of the right eye.

9. The method of claim 8, wherein,

the acquiring of the eye gaze vector comprises determining an eye gaze vector of the left eye and an eye gaze vector of the right eye, and

the acquiring of the gaze point on the display unit comprises applying the weight based on the estimated quality of the detected position of the iris of the left eye and the estimated quality of the detected position of the iris of the right eye to the eye gaze vector of the left eye and the eye gaze vector of the right eye, respectively.

10. The method of claim 1, wherein the three-dimensional modeling of the detected facial feature comprises modeling the facial feature as a three-dimensional grid by using an Active Appearance Model (AAM).

11. The method of claim 1, wherein the detecting of the facial feature is performed by using an AAM.

23. A non-transitory computer-readable recording medium configured to store computer-readable instructions that, when executed by a computer, perform the method of claim 1.

12. An apparatus of tracking an eye gaze, the apparatus comprising:

a facial feature detecting unit configured to detect a facial feature in a captured initial image;

a facial feature tracking unit configured to three-dimensionally model the detected facial feature and to track the three-dimensionally modeled facial feature in consecutively captured images;

an iris center detecting unit configured to detect an iris center in the consecutively captured images; and

an eye gaze tracking unit configured to acquire an eye gaze vector based on the tracked three-dimensionally modeled facial feature and the detected iris center and to acquire a gaze point on a display unit based on the eye gaze vector.

13. The apparatus of claim 12, wherein the facial feature detecting unit is further configured to display a message requesting a user to gaze at a certain point of the display unit when detecting the facial feature.

14. The apparatus of claim 12, wherein the facial feature tracking unit is further configured to calculate an eyeball center based on the detected facial feature and to three-dimensionally model the facial feature including the calculated eyeball center.

15. The apparatus of claim 14, wherein the facial feature tracking unit is further configured to calculate the eyeball center based on statistics of a facial size and an eyeball diameter.

16. The apparatus of claim 12, wherein the facial feature tracking unit is further configured to determine a facial pose based on the three-dimensionally modeled facial feature and to track the facial pose.

17. The apparatus of claim 12, wherein the iris center detecting unit is further configured to detect the iris center by using only an eye region of the three-dimensionally modeled facial feature.

18. The apparatus of claim 12, wherein the iris center detecting unit is further configured to calculate a position of the iris center on the three-dimensionally modeled facial feature.

19. The apparatus of claim 12, wherein the iris center detecting unit is further configured to detect a position of an iris of a left eye and a position of an iris of a right eye and to determine a weight by estimating a quality of the detected position of the iris of the left eye and a quality of the detected position of the iris of the right eye.

20. The apparatus of claim 19, wherein the eye gaze tracking unit is further configured to determine an eye gaze vector of the left eye and an eye gaze vector of the right eye and to acquire the gaze point on the display unit by applying the weight based on the estimated quality of the detected position of the iris of the left eye and the estimated quality of the detected position of the iris of the right eye to the eye gaze vector of the left eye and the eye gaze vector of the right eye, respectively.

21. The apparatus of claim 12, wherein the facial feature tracking unit is further configured to model the facial feature as a three-dimensional grid by using an Active Appearance Model (AAM).

22. The apparatus of claim 12, wherein the facial feature detecting unit is further configured to detect the facial feature by using an AAM.