Systems and methods of light modulation in eye tracking devices

• US 9,836,639 B2
• Filed: 01/09/2015
• Issued: 12/05/2017
• Est. Priority Date: 01/10/2014
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Sztuk, Sebastian, Tall, Martin Henrik, San Agustin Lopez, Javier
• Primary Examiner(s): Owens, Tsion B
• Application Number:
  US14/593,796
• Publication Number:
  US 20150199559A1
• Time in Patent Office: 1,061 Days
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/651 for reducing power consumption by affecting camera operations, e.g. sleep mode, hibernation mode or power off of selective parts of the camera
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/684 performed by controlling the image sensor readout, e.g. by controlling the integration time
• CPC Class Codes: H04N 23/73 by influencing the exposure time

Abstract

An image of a user'"'"'s eyes and face may be analyzed using computer-vision algorithms. A computing device may use the image to determine the location of the user'"'"'s eyes and estimate the direction in which the user is looking. The eye tracking technology may be used in a wide range of lighting conditions and with many different and varying light levels. When a user is near a light source, an automatic exposure feature in the camera may result in the user'"'"'s face and eyes appearing too dark in the image, possibly reducing the likelihood of face and eye detection. Adjusting attributes such as the camera exposure time and the intensity and illumination interval of the light sources based on motion and light levels may improve detection of a user'"'"'s features.

Claims

1. A method comprising:

capturing a first image using a camera of an eye tracking device;

detecting a face region of the first image, the face region depicting the face and eye features of a user;

based on at least a portion of the face region, measuring a parameter of the first image, wherein the parameter is a value representing a visual property of the first image;

based on the value of the parameter of the first image being outside of a target range of parameter values, adjusting an attribute associated with the eye tracking device to improve detection of the eye features for use by an eye tracking algorithm, wherein the attribute is a value representing a setting of the eye tracking device having an effect on images captured by the camera, and wherein adjusting the attribute comprises determining a value for the attribute that places the value of the parameter at a center of the target range;

after adjusting the attribute associated with the eye tracking device, capturing a second image using the camera of the eye tracking device; and

performing the eye tracking algorithm on the second image to determine, using the eye tracking device, a gaze direction of the user based on detected eye features in the second image.

2. The method of claim 1, wherein measuring the parameter of the first image comprises measuring a contrast of the first image.

3. The method of claim 1, wherein measuring the parameter of the first image comprises measuring a brightness of the first image.

4. The method of claim 1, wherein measuring the parameter of the first image comprises attempting to detect facial features of the user.

5. The method of claim 1, wherein the attribute is a camera exposure time of the camera.

6. The method of claim 1, wherein the attribute is a camera gain of the camera, the camera gain specifying a light amplification factor of the camera.

7. The method of claim 1, wherein the attribute is a camera f-number of the camera, the camera f-number specifying a ratio between a focal length and an entrance pupil diameter of the camera.

8. The method of claim 1, wherein the attribute is a camera frame rate of the camera.

9. The method of claim 1, wherein the attribute is a light intensity of a light source of the eye tracking device.

19. The method of claim 1, wherein the parameter is an average brightness of the face region of the first image, wherein the attribute is a camera exposure time of the camera, and wherein determining a value for the attribute that places the value of the attribute at a center of the target range comprises:

computing a scale factor between a center value of a target average brightness range and the average brightness of the face region in the first image; and

determining a value for a camera exposure time for the second image by scaling a camera exposure time of the first image by the scale factor.

20. The method of claim 1, wherein the parameter is a contrast between a pupil of the user depicted in the first image and an iris of the user depicted in the first image, wherein the attribute is a light intensity of one or more light sources of the eye tracking device, and wherein determining a value for the attribute that places the value of the attribute at a center of the target range comprises:

responsive to determining that the contrast between the pupil and the iris in the first image is lower than a target contrast range, increasing the light intensity of one or more light sources.

10. A computing device comprising:

a camera module configured to capture images;

an attribute adjustment module configured to;

receive a first image captured by the camera module;

detect a face region of the first image, the face region depicting the face and eye features of a user,

based on at least a portion of the face region, measure a parameter of the first image, wherein the parameter is a value representing a visual property of the first image, and

based on the value of the parameter of the first image being outside of a target range of parameter values, adjust an attribute associated with the camera module to improve detection of the eye features for use by an eye tracking algorithm, wherein the attribute is a value representing a setting of the eye tracking device having an effect on images captured by the camera, and wherein adjusting the attribute comprises determining a value for the attribute that places the value of the parameter at a center of the target range; and

an eye tracking module configured to;

receive a second image captured by the camera module, the second image captured after adjusting the attribute associated with the camera module, and

perform the eye tracking algorithm on the second image to determine a gaze direction of the user based on detected eye features in the second image.

11. The computing device of claim 10, further comprising a light source module configured to emit light while the images are captured.

12. The computing device of claim 10, wherein measuring the parameter of the first image comprises measuring a contrast of the first image.

13. The computing device of claim 10, wherein measuring the parameter of the first image comprises measuring a brightness of the first image.

14. The computing device of claim 10, wherein measuring the parameter of the first image comprises attempting to detect facial features of the user.

15. The computing device of claim 10, wherein the attribute is a camera exposure time of the camera module.

16. The computing device of claim 10, wherein the attribute is a camera gain of the camera module, the camera gain specifying a light amplification factor of the camera module.

17. The computing device of claim 10, wherein the attribute is a camera f-number of the camera module, the camera f-number specifying a ratio between a focal length and an entrance pupil diameter of the camera module.

18. A non-transitory machine-readable storage medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:

capturing a first image using a camera of an eye tracking device;

detecting a face region of the first image, the face region depicting the face and eye features of a user;

based on at least a portion of the face region, measuring a parameter of the first image, wherein the parameter is a value representing a visual property of the image;

based on the value of the parameter of the first image being outside of a target range of parameter values, adjusting an attribute associated with the eye tracking device to improve detection of the eye features for use by an eye tracking algorithm, wherein the attribute is a value representing a setting of the eye tracking device having an effect on images captured by the camera, and wherein adjusting the attribute comprises determining a value for the attribute that places the value of the parameter at a center of the target range;

after adjusting the attribute associated with the eye tracking device, capturing a second image using the camera of the eye tracking device; and

performing the eye tracking algorithm on the second image to determine, using the eye tracking device, a gaze direction of the user based on detected eye features in the second image.