Eye gaze tracking method and apparatus and computer-readable recording medium

• US 9,798,384 B2
• Filed: 10/06/2015
• Issued: 10/24/2017
• Est. Priority Date: 04/10/2014
• Status: Expired due to Fees

• Current Assignee: Samsung Electronics Co. Ltd. (Samsung Group)
• Original Assignee: Samsung Electronics Co. Ltd. (Samsung Group)
• Inventors: Strupczewski, Adam, Naruniec, Jacek, Mucha, Kamil, Czuprynski, Blazej
• Primary Examiner(s): BHATNAGAR, ANAND P
• Application Number:
  US14/876,226
• Publication Number:
  US 20160026246A1
• Time in Patent Office: 749 Days
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction

Abstract

A method and an apparatus of tracking an eye gaze to determine where a gaze point of a user is located on a display unit of a device, based on a facial pose and a position of an iris center, are provided. The method includes detecting a facial feature in a captured initial image, three-dimensionally modeling the detected facial feature, tracking the three-dimensionally modeled facial feature in consecutively captured images, detecting an iris center in the consecutively captured images, acquiring an eye gaze vector based on the tracked three-dimensionally modeled facial feature and the detected iris center, and acquiring a gaze point on a display unit based on the eye gaze vector.

Claims

1. A method of tracking an eye gaze, the method comprising:

detecting a facial feature in a captured image;

detecting an iris of a left eye and an iris of a right eye;

determining weights by estimating a quality of the detected iris of the left eye and a quality of the detected iris of the right eye;

detecting iris centers of the left eye and the right eye in the captured image based on the detected iris of the left eye and the detected iris of the right eye; and

acquiring a gaze point based on the facial feature, the detected iris centers, and the weights.

2. The method of claim 1, wherein the detecting of the facial feature comprises displaying a message requesting a user to gaze at a certain point of a display unit.

3. The method of claim 1, wherein the detecting of the facial feature comprises:

calculating an eyeball center based on the detected facial feature; and

three-dimensionally modeling the facial feature including the calculated eyeball center.

4. The method of claim 3, wherein the detecting of the iris centers comprises detecting at least one of the iris centers by using only an eye region of the three-dimensionally modeled facial feature.

5. The method of claim 3, wherein the detecting of the iris centers comprises calculating a position of at least one of the iris centers on the three-dimensionally modeled facial feature.

6. The method of claim 3, wherein the three-dimensional modeling of the detected facial feature comprises modeling the facial feature as a three-dimensional grid by using an Active Appearance Model (AAM).

7. The method of claim 3, wherein the calculating of the eyeball center is performed based on statistics of a facial size and an eyeball diameter.

8. The method of claim 3, further comprising:

tracking the three-dimensionally modeled facial feature in consecutively captured images,

wherein the acquiring of the gaze point is further based on the tracked three-dimensionally modeled facial feature.

9. The method of claim 8, wherein the tracking of the three-dimensionally modeled facial feature comprises:

determining a facial pose based on the three-dimensionally modeled facial feature; and

tracking the facial pose.

10. The method of claim 8, wherein the acquiring of the gaze point further comprises:

acquiring an eye gaze vector of the left eye and an eye gaze vector of the right eye based on the tracked three-dimensionally modeled facial feature and the detected iris centers;

applying the weights based on the estimated quality of the detected iris of the left eye and the estimated quality of the detected iris of the right eye to the eye gaze vector of the left eye and the eye gaze vector of the right eye, respectively; and

acquiring the gaze point on a display unit based on the weights applied to the eye gaze vector of the left eye and the weights applied to the eye gaze vector of the right eye.

11. The method of claim 8, wherein the detecting of the iris centers comprises detecting the iris centers in the consecutively captured images.

12. The method of claim 11, wherein the acquiring of the gaze point comprises:

acquiring an eye gaze vector based on the tracked three-dimensionally modeled facial feature and the detected iris centers; and

acquiring the gaze point on a display unit based on the eye gaze vector.

13. The method of claim 1, wherein the detecting of the facial feature is performed by using an AAM.

14. A non-transitory computer-readable recording medium configured to store computer-readable instructions that, when executed by a computer, perform the method of claim 1.

15. The method of claim 1, wherein the facial feature includes at least one of an eye region, a nose region, a mouth region, and an eyeball center.

16. An apparatus for tracking an eye gaze, the apparatus comprising:

a camera configured to capture an image; and

a processor configured to;

detect a facial feature in the captured image,

detect an iris of a left eye and an iris of a right eye,

determine weights by estimating a quality of the detected iris of the left eye and a quality of the detected iris of the right eye,

detect iris centers of the left eye and the right eye in the captured image, based on the detected iris of the left eye and the detected iris of the right eye, and

acquire a gaze point based on the facial feature, the detected iris centers, and the weights.

17. The apparatus of claim 16, wherein the processor is further configured to display a message requesting a user to gaze at a certain point of a display unit when detecting the facial feature.

18. The apparatus of claim 16, wherein the processor is further configured to calculate an eyeball center based on the detected facial feature and to three-dimensionally model the facial feature including the calculated eyeball center.

19. The apparatus of claim 18, wherein the processor is further configured to detect at least one of the iris centers by using only an eye region of the three-dimensionally modeled facial feature.

20. The apparatus of claim 18, wherein the processor is further configured to calculate a position of at least one of the iris centers on the three-dimensionally modeled facial feature.

21. The apparatus of claim 18, wherein the processor is further configured to model the facial feature as a three-dimensional grid by using an Active Appearance Model (AAM).

22. The apparatus of claim 18, wherein the processor is further configured to calculate the eyeball center based on statistics of a facial size and an eyeball diameter.

23. The apparatus of claim 18, wherein the processor is further configured to:

track the three-dimensionally modeled facial feature in consecutively captured images, and

acquire the gaze point further based on the tracked three-dimensionally modeled facial feature.

24. The apparatus of claim 23, wherein the processor is further configured to determine a facial pose based on the three-dimensionally modeled facial feature and to track the facial pose.

26. The apparatus of claim 23, wherein the processor is further configured to detect the iris centers in the consecutively captured images.

27. The apparatus of claim 26, wherein the processor is further configured to:

acquire an eye gaze vector based on the tracked three-dimensionally modeled facial feature and the detected iris centers; and

acquire the gaze point on a display unit based on the eye gaze vector.

28. The apparatus of claim 16, wherein the processor is further configured to detect the facial feature by using an AAM.

29. The apparatus of claim 16, wherein the facial feature includes at least one of an eye region, a nose region, a mouth region, and an eyeball center.

25. The apparatus of claimer 23, wherein the processor is further configured to:

determine an eye gaze vector of the left eye and an eye gaze vector of the right eye based on the tracked three-dimensionally modeled facial feature and the detected iris centers,

apply the weights based on the estimated quality of the detected iris of the left eye and the estimated quality of the detected iris of the right eye to the eye gaze vector of the left eye and the eye gaze vector of the right eye, respectively, and

acquire the gaze point on a display unit based on the weights applied to the eye gaze vector of the left eye and the weights applied to the eye gaze vector of the right eye.