Determining gaze target based on facial features

• US 10,545,574 B2
• Filed: 03/03/2017
• Issued: 01/28/2020
• Est. Priority Date: 03/01/2013
• Status: Active Grant

• Current Assignee: Tobii Technology AB
• Original Assignee: Tobii Technology AB
• Inventors: George-Svahn, Erland, Henderek, David Figgins, Lannsjo, Rebecka, Skogo, Marten, Elvesjo, John
• Primary Examiner(s): Iluyomade, Ifedayo B
• Application Number:
  US15/449,058
• Publication Number:
  US 20170177079A1
• Time in Patent Office: 1,061 Days
• Field of Search: None
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/02 Input arrangements using manually operated switches, e.g. using keyboards or dials
• CPC Class Codes: G06F 3/0227 Cooperation and interconnection of the input arrangement with other functional units of a computer G06F3/023 - G06F3/037 take precedence
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning

Abstract

A computer system can be controlled with non-contact inputs, such as eye-tracking devices. A visual indicator can be presented on a display to indicate the location where a computer function will take place (e.g., a common cursor). The visual indicator can be moved to a gaze target in response to continued detection of an action (e.g., touchpad touch) by a user for a predetermined period of time. A delay between the action and the movement of the visual indicator can allow a user time to “abort” movement of the visual indicator. Additionally, once the visual indicator has moved, the visual indicator can be controlled with additional precision as the user moves the gaze while continuing the action (e.g., continued holding of the touchpad).

Claims

1. A computing device, comprising:

a display device for presenting a display;

an image sensor configured to capture images of at least a portion of a user of the computing device; and

at least one processor;

wherein the at least one processor is configured to determine a first facial feature based on at least one captured image, the first facial feature comprising an eye lid position of an eye of the user, and the first facial feature being designated for control of the display in a vertical direction;

determine a second facial feature based on the at least one captured image, the second facial feature being designated for control of the display in a horizontal direction, the second facial feature comprising a head pose;

determine an orientation value, for the head pose of the user based on the at least one captured image, the orientation value indicating an offset of the head pose from a direct forward line of sight, wherein the orientation value is less than or equal to a threshold orientation value of at least seven degrees;

in the event the orientation value does not exceed a threshold orientation value, determine a gaze target of the user on the display based on the first facial feature and the second facial feature by at least;

determining a vertical component of the gaze target based on an openness of the eye lid position wherein;

a first eye lid position at an upper location on the eye of the user corresponds with the gaze target being adjacent an upper portion of the display; and

a second eye lid position at a lower location on the eye of the user corresponds with the gaze target being adjacent a bottom portion of the display; and

determining a horizontal component of the gaze target based on the second facial feature; and

scroll content displayed on the display based on the gaze target being within a predefined scrod zone on the display.

2. The computing device of claim 1, wherein the at least one processor is further configured to determine the horizontal component based on the offset of the head pose.

3. The computing device of claim 1, wherein the at least one processor is further configured to determine changes in the horizontal component based on the second facial feature.

4. The computing device of claim 1, further comprising an infrared light source.

5. The computing device of claim 4, wherein:

the image sensor is further configured to detect the infrared light emitted by the infrared light source, and

the at least one processor is further configured to determine the gaze target based on a reflection, from at least one eye, of the infrared light emitted by the infrared light source.

10. The computing device of claim 1, wherein:

the at least one processor is further configured to determine that the user is reading elements on the display based on detecting saccades of the eye; and

scrolling the content displayed on the display is further based on determining that the user is reading the elements on the display.

11. The computing device of claim 1, wherein the at least one processor is further configured to:

determine a first facial feature based on at least one captured image, the first facial feature comprising an eye lid position of an eye of the user, and the first facial feature being designated for control of the display in a vertical direction; and

in the event the orientation value does not exceed a threshold orientation value, determine a gaze target of the user on the display based on the first facial feature and the second facial feature by at least;

determining a horizontal component of the gaze target based on the second facial feature; and

determining a vertical component of the gaze target based on an openness of the eye lid position wherein;

a first eye lid position at an upper location on the eye of the user corresponds with the gaze target being adjacent an upper portion of the display; and

a second eye lid position at a lower location on the eye of the user corresponds with the gaze target being adjacent a bottom portion of the display; and

scroll content displayed on the display based on the gaze target being within a predefined scroll zone on the display.

12. The computing device of claim 11, wherein the at least one processor is further configured to determine changes in the vertical component based on the first facial feature.

13. The computing device of claim 11, wherein:

the openness of the eye lid position comprises a first value when the gaze target is adjacent to a top portion of the display; and

the openness of the eye lid position comprises a second value when the gaze target is adjacent to a bottom portion of the display, the second value being smaller than the first value.

14. The computing device of claim 11, wherein determining the vertical component of the gaze target is based solely on the openness of the eye lid position.

6. A method of providing user input to a computer device, the method comprising:

determining a first facial feature based on at least one captured image, the first facial feature comprising an eye lid position of an eye of a user, and the first facial feature being designated for control of a display in a vertical direction;

determining a second facial feature based on the at least one captured image, the second facial feature being designated for control of the display in a horizontal direction, the second facial feature comprising a head pose;

determine an orientation value for the head pose of the user based on the at least one captured image, the orientation value indicating an offset of the head pose from a direct forward line of sight, wherein the orientation value is less than or equal to a threshold orientation value of at least seven degrees;

in the event the orientation value does not exceed a threshold orientation value, determining a gaze target of the user on the display based on the first facial feature and the second facial feature by at least;

determining a vertical component of the gaze target based solely on an openness of the eye lid position wherein;

a first eye lid position at an upper location on the eye of the user corresponds with the gaze target being adjacent an upper portion of the display; and

a second eye lid position at a lower location on the eye of the user corresponds with the gaze target being adjacent a bottom portion of the display; and

determining a horizontal component of the gaze target based on the second facial feature; and

scrolling content displayed on the display based on the gaze target being within a predefined scroll zone on the display.

7. The method of claim 6, further comprising determining changes in the horizontal component based on the second facial feature.

8. The method of claim 6, further comprising determining, based on the second facial feature, that the gaze target is within the predefined scroll zone on a display device.

15. The method of providing user input to a computer device of claim 6, further comprising:

determining a first facial feature based on at least one captured image, the first facial feature comprising an eye lid position of an eye of the user, and the first facial feature being designated for control of the display in a vertical direction; and

in the event the orientation value does not exceed a threshold orientation value, determine a gaze target of the user on the display based on the first facial feature and the second facial feature by at least;

determining a horizontal component of the gaze target based on the second facial feature; and

determining a vertical component of the gaze target based on an openness of the eye lid position wherein;

a first eye lid position at an upper location on the eye of the user corresponds with the gaze target being adjacent an upper portion of the display; and

a second eye lid position at a lower location on the eye of the user corresponds with the gaze target being adjacent a bottom portion of the display; and

scroll content displayed on the display based on the gaze target being within a predefined scroll zone on the display.

16. The method of claim 15, further comprising determining changes in the vertical component based on the first facial feature.

9. A non-transitory computer-readable storage medium storing program instructions. wherein the program instructions, when executed by one or more processors, perform at least the steps of:

determining a first facial feature based on at least one captured image, the first facial feature comprising an eye lid position of an eye of a user, and the first facial feature being designated for control of a display in a vertical direction;

determining a second facial feature based on the at least one captured image, the second facial feature being designated for control of the display in a horizontal direction, the second facial feature comprising a head pose;

determine an orientation value for the head pose of the user based on the at least one captured image, the orientation value indicating an offset of the head pose from a direct forward line of sight;

in the event the orientation value is less than or equal to a threshold orientation value of at least seven degrees, determining a gaze target of the user on a display of a display device based on the first facial feature and the second facial feature by at least;

determining a vertical component of the gaze target based solely on an openness of the eye lid position; and

determining a horizontal component of the gaze target based on the second facial feature; and

scrolling content displayed on the display based on the gaze target being within a predefined scroll zone on the display.

17. The non-transitory computer-readable storage medium storing program instructions of claim 9, further comprising the steps of:

determining a first facial feature based on at least one captured image, the first facial feature comprising an eye lid position of an eye of the user, and the first facial feature being designated for control of the display in a vertical direction; and

in the event the orientation value does not exceed a threshold orientation value, determine a gaze target of the user on the display based on the first facial feature and the second facial feature by at least;

determining a horizontal component of the gaze target based on the second facial feature; and

determining a vertical component of the gaze target based on an openness of the eye lid position wherein;

a first eye lid position at an upper location on the eye of the user corresponds with the gaze target being adjacent an upper portion of the display; and

a second eye lid position at a lower location on the eye of the user corresponds with the gaze target being adjacent a bottom portion of the display; and

scroll content displayed on the display based on the gaze target being within a predefined scroll zone on the display.