Systems and methods of eye tracking control

• US 10,459,520 B2
• Filed: 10/24/2017
• Issued: 10/29/2019
• Est. Priority Date: 01/21/2013
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Facebook Technologies, LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: San Agustin Lopez, Javier, Sztuk, Sebastian, Tall, Martin Henrik
• Primary Examiner(s): Zhou, Hong
• Application Number:
  US15/792,315
• Publication Number:
  US 20180046248A1
• Time in Patent Office: 735 Days
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures

Abstract

Methods and systems to facilitate eye tracking control are provided. A user input is received at a computing device. Point of regard information associated with a user of the computing device is determined while the user input is being received. The point of regard information indicates a location on a display of the computing device at which the user is looking. An operation associated with a display object identified based on the point of regard information is performed when receipt of the user input is determined to have terminated.

Claims

1. A method comprising:

receiving a user input at a computing device;

while receiving the user input;

determining information about at least one point of regard indicating at least one location on a display of the computing device at which a user is looking; and

displaying, in response to the user input, a plurality of display objects on the display, each display object positioned at a different location within the display and associated with a respective performable operation;

determining that the user input has terminated; and

in response to termination of the user input;

determining a point of regard at a time when the user input has terminated;

identifying a display object of the plurality of display objects based on the determined point of regard; and

performing an operation associated with the identified display object.

2. The method of claim 1, further comprising:

capturing image data related to the user;

detecting one or more eye features of the user using the captured image data; and

determining the information about the at least one point of regard, based in part on the one or more eye features.

3. The method of claim 2, wherein the one or more eye features include information about at least one of:

a location of an eye of the user, a location of a pupil of the eye, and a location of a corneal reflection related to the eye.

4. The method of claim 1, further comprising:

activating, in response to the user input, an eye tracking functionality at the computing device;

calculating one or more parameters specific to the user, upon activating the eye tracking functionality; and

determining the information about the at least one point of regard, based in part on the one or more parameters.

5. The method of claim 4, wherein the one or more parameters include at least one of:

a vertical offset between an optical axis and a visual axis of an eye of the user, and a horizontal offset between the optical axis and the visual axis.

6. The method of claim 4, further comprising:

calculating the one or more parameters based on matching a first path determined by a moving target displayed on the display with a second path determined by movement information associated with at least one eye feature of the user.

7. The method of claim 6, wherein the at least one eye feature includes at least one of:

a pupil of an eye of the user, an iris of the eye, an estimated point of regard of the user, an optical axis of the eye, a visual axis of the eye, and one or more corneal reflections produced by one or more infrared light sources.

8. The method of claim 1, further comprising:

illuminating a portion of a face of the user including at least one eye of the user;

capturing infrared light reflected from the portion of the face; and

determining the information about the at least one point of regard, based in part on the captured infrared light.

9. The method of claim 8, further comprising:

capturing the infrared light by blocking one or more components of light reflected from the portion of the face having wavelengths below 800 nm and passing one or more other components of the reflected light having wavelengths above 800 nm.

10. The method of claim 1, further comprising:

calculating a user'"'"'s point of regard at a predetermined rate until the user input is detected.

11. The method of claim 1, wherein each display object includes an eye-controllable item controlled by the point of regard.

12. The method of claim 1, further comprising:

stop displaying the plurality of display objects on the display, after the operation was performed.

13. A computing device, comprising:

a memory having instructions embodied thereon;

one or more processors configured by the instructions to perform operations comprising;

receiving a user input at the computing device;

while receiving the user input;

determining information about at least one point of regard indicating at least one location on a display of the computing device at which a user is looking; and

displaying, in response to the user input, a plurality of display objects on the display, each display object positioned at a different location within the display and associated with a respective performable operation;

determining that the user input has terminated; and

in response to termination of the user input;

determining a point of regard at a time when the user input has terminated; and

identifying a display object of the plurality of display objects based on the determined point of regard; and

performing an operation associated with the identified display object.

14. The computing device of claim 13, further comprising:

one or more illumination sources configured to illuminate a portion of a face of the user including at least one eye of the user;

an infrared pass filter configured to pass infrared light reflected from the portion of the face; and

a camera coupled to the infrared pass filter configured to capture the infrared light, wherein

the one or more processors are configured to determine the information about the at least one point of regard, based in part on the captured infrared light.

15. The computing device of claim 14, wherein the infrared pass filter is further configured to:

block one or more components of light reflected from the portion of the face having wavelengths below 800 nm; and

pass one or more other components of the reflected light having wavelengths above 800 nm.

16. The computing device of claim 14, wherein the infrared pass filter is further configured to:

pass one or more components of light reflected from the portion of the face having wavelengths between 800 nm and 900 nm.

17. The computing device of claim 13, further comprising:

one or more cameras configured to capture image data related to the user, wherein the one or more processors are further configured to

detect one or more eye features of the user using the captured image data, and

determine the information about the at least one point of regard, based in part on the one or more eye features.

18. The computing device of claim 13, wherein the one or more processors are further configured to:

activate, in response to the user input, an eye tracking functionality at the computing device;

calculate one or more parameters specific to the user, upon activating the eye tracking functionality; and

determine the information about the at least one point of regard, based in part on the one or more parameters.

19. The computing device of claim 18, wherein the one or more processors are further configured to:

calculate the one or more parameters based on matching a first path determined by a moving target displayed on the display with a second path determined by movement information associated with at least one eye feature of the user.

20. A non-transitory machine-readable storage medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:

receiving a user input at a computing device;

while receiving the user input;

determining, by the one or more processors, information about at least one point of regard indicating at least one location on a display of the computing device at which a user is looking; and

displaying, in response to the user input, a plurality of display objects on the display, each display object positioned at a different location within the display and associated with a respective performable operation;

determining that the user input has terminated; and

in response to termination of the user input;

determining a point of regard at a time when the user input has terminated;

identifying a display object of the plurality of display objects based on the determined point of regard; and

performing an operation associated with the identified display object.