Systems and methods of eye tracking control on mobile device

• US 9,612,656 B2
• Filed: 11/25/2013
• Issued: 04/04/2017
• Est. Priority Date: 11/27/2012
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Tall, Martin Henrik, Sztuk, Sebastian, Lopez, Javier San Agustin
• Primary Examiner(s): ABDIN, SHAHEDA A
• Application Number:
  US14/089,609
• Publication Number:
  US 20140145935A1
• Time in Patent Office: 1,226 Days
• CPC Class Codes: G02B 5/208 for use with infrared or ultraviolet radiation, e.g. for separating visible light from infrared and/or ultraviolet radiation
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction

Abstract

Methods and systems to facilitate eye tracking control on mobile devices are provided. An image of a portion of a user is received at an eye tracking device, where the image includes reflections caused by light emitted on the user from one or more light sources located within the eye tracking device. One or more eye features associated with an eye of the user is detected using the reflections. Point of regard information is determined using the one or more eye features, where the point of regard information indicates a location on a display of a computing device coupled to the eye tracking device at which the user was looking when the image of the portion of the user was taken. The point of regard information is sent to an application capable of performing a subsequent operation using the point of regard information.

Claims

1. A method comprising:

controlling, at an eye tracking device, one or more light sources located within the eye tracking device to synchronize emission of light from the one or more light sources with a timing of image capture by a camera with a depth of field approximately in the range 40 cm to 80 cm, wherein the controlling further includes filtering the light from the light sources using an infrared pass filter so that only light in the 800-900 nm range enters the camera, and wherein the camera has a field of view that matches the field of emission of the one or more light sources;

receiving, at the eye tracking device, an image of a portion of a user captured by the camera, the image including reflections caused by light emitted on the user from the one or more light sources located within the eye tracking device;

detecting one or more eye features associated with an eye of the user using the reflections;

determining point of regard information using the one or more eye features, the point of regard information indicating a location on a display of a computing device coupled to the eye tracking device at which the user was looking when the image of the portion of the user was taken; and

sending the point of regard information to an application capable of performing a subsequent operation using the point of regard information.

2. The method of claim 1, wherein the camera is located within the eye tracking device.

3. The method of claim 1, wherein the subsequent operation includes moving an object displayed on the display based on the point of regard information.

4. The method of claim 1, wherein the camera includes an infrared filter that blocks visible light.

5. The method of claim 1, wherein the eye tracking device is a docking device to which the computing device is docked.

6. The method of claim 1, wherein the one or more light sources have a field of emission of approximately 30-40 degrees.

7. A non-transitory machine-readable storage medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:

controlling one or more light sources located within an eye tracking device to synchronize emission of light from the one or more light sources with a timing of image capture by a camera with a depth of field approximately in the range 40 cm to 80 cm, wherein the controlling further includes filtering the light from the light sources using an infrared pass filter so that only light in the 800-900 nm range enters the camera, and wherein the camera has a field of view that matches the field of emission of the one or more light sources;

receiving an image of a portion of a user captured by the camera, the image including reflections caused by light emitted on the user from the one or more light sources located within an eye tracking device;

detecting one or more eye features associated with an eye of the user using the reflections;

determining point of regard information using the one or more eye features, the point of regard information indicating a location on a display of a computing device coupled to the eye tracking device at which the user was looking when the image of the portion of the user was taken; and

sending the point of regard information to an application capable of performing a subsequent operation using the point of regard information.

8. The non-transitory machine-readable storage medium of claim 7, wherein the camera is located within the eye tracking device.

9. The non-transitory machine-readable storage medium of claim 7, wherein the subsequent operation includes moving an object displayed on the display based on the point of regard information.

10. An eye tracking device comprising:

one or more light sources configured to emit light, wherein the one or more sources further comprise an infrared pass at filters that to be within an 800-900 nm range; and

one or more hardware processors in communication with the one or more light sources, the one or more hardware processors configured to;

control the one or more light sources to synchronize emission of light from the one or more light sources with a timing of image capture by a camera with a depth of field approximately in the range 40 cm to 80 cm, wherein the camera has a field of view that matches the field of emission of the one or more light sources;

receive an image of a portion of a user captured by the camera, the image including reflections caused by light emitted on the user from the one or more light sources;

detect one or more eye features associated with an eye of the user including detecting the one or more eye features using the reflections;

determine point of regard information using the one or more eye features, the point of regard information indicating a location on a display of a computing device coupled to the eye tracking device at which the user was looking when the image of the portion of the user was taken; and

send the point of regard information to an application capable of performing a subsequent operation using the point of regard information.

11. The eye tracking device of claim 10, wherein the camera is located within the eye tracking device.

12. The eye tracking device of claim 10, wherein the subsequent operation includes moving an object displayed on the display based on the point of regard information.

13. The eye tracking device of claim 10, wherein the one or more light sources have a field of emission of approximately 30-40 degrees.

14. The eye tracking device of claim 10, wherein one or more light sources are turned off when the user is not detected for a predetermined amount of time and are turned on when the user is detected again, the predetermined amount of time being five seconds or greater.

15. The eye tracking device of claim 10, wherein one or more cameras are turned off when the user is not detected for a predetermined amount of time and are turned on when the user is detected again, the predetermined amount of time being five seconds or greater.

16. A system comprising:

a mobile computing device; and

an eye tracking device coupled to and in communication with the mobile computing device, the eye tracking device being configured to;

control one or more light sources located within the eve tracking device to synchronize emission of light from the one or more light sources with a timing of image capture by a camera with a depth of held approximately in the range 30 cm to 60 cm, wherein to control the one or more light sources the eve tracking device is further configured to filter the light from the one or more light sources using an infrared pass filter so that only the 800-900 nm range enters the camera and wherein the camera has a field of view that matches the field of emission of the one or more light sources;

receive an image of a portion of a user captured by the camera, the image including reflections caused by light emitted on the user from the one or more light sources located within the eye tracking device;

detect one or more eye features associated with an eye of the user using the reflections;

determine point of regard information using the one or more eye features, the point of regard information indicating a location on a display of the mobile computing device at which the user was looking when the image of the portion of the user was taken; and

send the point of regard information to an application on the mobile computing device capable of performing a subsequent operation using the point of regard information.

17. The system of claim 16, wherein the camera is located within the eye tracking device.

18. The system of claim 16, wherein the subsequent operation includes moving an object displayed on the display based on the point of regard information.