Facial image processing system

• US 20030169907A1
• Filed: 01/24/2003
• Published: 09/11/2003
• Est. Priority Date: 07/24/2000
• Status: Active Grant

• Current Assignee: Seeing Machines LTD
• Original Assignee: Seeing Machines LTD
• Inventors: Edwards, Timothy, Heinzmann, Jochen, Zelinsky, Alex, Rougeaux, Sebastien
• Granted Patent: US 7,043,056 B2
• Time in Patent Office: Days
• US Class Current: 382/118
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A method of determining an eye gaze direction of an observer is disclosed comprising the steps of: (a) capturing at least one image of the observer and determining a head pose angle of the observer; (b) utilizing the head pose angle to locate an expected eye position of the observer, and (c) analyzing the expected eye position to locate at least one eye of the observer and observing the location of the eye to determine the gaze direction.

Claims

1. A method of determining a head-pose measurement of a person, said head pose measurement including a rotational displacement and translation displacement, said method including the steps of:

(a) capturing images of the face and head of said person from at least two viewpoints;

(b) computing an initial estimate of head-pose;

(c) computing the three dimensional location of two or more facial features; and

(d) determining a rotational displacement and translation displacement which best match a head-model to the computed location of said two or more facial features.

2. A method of determining a head-pose measurement of a person as claimed in claim 1 wherein step (b) includes the sub-step of, localising the face within two or more of the captured images.

3. A method of determining a head-pose measurement of a person as claimed in claim 1 or claim 2 wherein step (c) includes the substeps of:

identifying feature search regions, based on said initial estimate of head pose, within each captured image that may contain an identifiable facial feature;

measuring the position of two or more facial features within two or more respective feature search regions in each of said captured images;

15. A method as claimed in any previous claim wherein said method further includes determining the three dimensional position of a centre of rotation of an observer'"'"'s eye.

16. A method as claimed in any previous claim wherein said method further determines eye closure and blinking.

18. A method as claimed in claim 17 wherein said head-pose measurement is determined using the method of any one of claims 1 to 3.

19. A method of analysis of a person performing a task wherein the method of any one of the previous claims is used to analyse an action of said subject.

20. A method as claimed in any one of claims 1 to 17 wherein said method utilises a measure feature location to determine if individual features on an observer'"'"'s face are flexible or fixed.

21. A method of animating an image, wherein the movement of a subject is tracked using a method according to any one of the previous claims, and wherein said image is animated in response to said movement.

23. A method for measuring operator fatigue wherein head-pose is measured using the method of any one of claims 1 to 3.

27. An apparatus when implementing a method of any previous claim.

4. A method of passively determining the direction of eye gaze of a person, said method including the steps of:

(a) capturing images of the face and head of said person from at least two viewpoints;

(b) localising the face within two or more of the captured images;

(c) computing an initial estimate of head-pose;

(d) identifying feature search regions, based on said initial estimate of head pose, within each captured image that may contain an identifiable facial feature;

(e) measuring the position of two or more facial features within two or more respective feature search regions in each of said captured images;

(f) computing the three-dimensional location of each of said two or more facial features;

(g) determining a head pose measurement, said head pose measurement including a rotational displacement and translation displacement which best match a head-model to the computed location of said two or more facial features;

(h) using said head pose measurement to determine the location the centre of one eyeball;

(i) using said head pose measurement to find an eye region corresponding with said eyeball, within each captured image, and determining the location of the centre of the iris of said eye;

(j) using the determined position of said centre of said eyeball and the determined location of the centre of the iris of said eye, to calculate an eye-gaze direction for said eye.

5. A method of passively determining the direction of eye gaze of a person, wherein the direction of eye gaze of said person is calculated by averaging the direction of eye gaze of said person'"'"'s two eyes, wherein the direction of eye gaze for each eye is determined using the method of claim 4.

7. A method of detecting a blink of a person'"'"'s eye, said method including the following steps:

calculating eye closure value using a method as claimed in claim 4;

forming a temporal sequence of eye closure values; and

comparing said temporal sequence of eye closure values with a blink template, wherein said blink template comprises a sequence of eye closure values corresponding to blink, in order to determine a correlation value between said blink template and said model temporal sequence of eye closure value, wherein a blink is determined to have occurred if said correlation value is greater than a predetermined threshold.

26. A method for measuring operator fatigue wherein a blink rate is determined using the method of any one of claims 7 to 9.

6. A method of determining the eye closure amount of an eye located on a person'"'"'s face, said eye having an eyeball, a top eyelid, and a bottom eyelid, said top and bottom eyelids meeting at a pair of eye corners;

wherein eye closure amount is defined as the ratio of eye corner separation to eyelid distance, said method including the steps of;

(a) capturing images of the face and head of said person from at least two viewpoints;

(b) determining a head pose measurement, from said captured images;

(c) using said head pose measurement to determine an eye region within said captured image;

(d) determining the location of said eye corners within said eye region;

(e) fitting at least two oriented edge detectors to said top eyelid and to said bottom eyelid, such that one of said oriented edge detectors of said top eyelid intersects with one oriented edge detectors of said bottom eyelid at each eye corner;

(f) approximating the position of said top eyelid, and said bottom eyelid, by fitting a parabola to said at least two oriented edge detectors corresponding to each eyelid, such that each parabola is tangential to the at least two oriented edge detectors of said parabola'"'"'s respective eyelid;

(g) determining said eye corner separation by calculating the distance between said eye corners;

(h) determining the maximum eyelid distance by calculating the maximum distance between said parabolas on the portion of said parabolas between said eye corners, in a direction perpendicular to a line joining the eye corners;

(i) calculating the eye closure amount by taking the ratio of the determined eye corner separation to maximum eyelid distance.

8. A method of detecting a blink of a person'"'"'s eye using a plurality of independent measurements, wherein each of said plurality of measurements produces a time varying output value, and at least one of said plurality of measurements is a measurement of eye closure value which has been calculated using a method as claimed in claim 6, said method including the step of:

combining the output value of two or more of said plurality of independent measurements in a weighted manner to calculate a blink detection value, wherein a blink is determined to have occurred if said blink detection value is greater than a predetermined threshold blink detection value.

9. A method of detecting a blink of a person'"'"'s eye as claimed in claim 8 wherein at least one of said plurality of independent measurements is selected from:

an eye region template correlation coefficient;

an optical flow measurement in said eye region;

eye region image rate of change;

colour segmentation within the eye region; and

iris template correlation coefficient

25. A method for measuring operator fatigue wherein eye closure amount is measured using the method of claim 6.

10. A method of determining an eye gaze direction of an observer comprising the steps of:

(a) capturing at least one image of the observer and determining a head pose angle of said observer;

(b) utilising said head pose angle to locate an expected eye position of said observer; and

(c) analysing said expected eye position to locate at least one eye of said observer, and observing the location of the eye to determine an eye gaze direction.

11. A method as claimed in claim 10 wherein said step (a) includes capturing an image in stereo and processing said image in stereo.

12. A method as claimed in any previous claim 10 or claim 11 wherein said step (a) incudes the steps of:

(i) determining a set of distinguishable feature locations on an observer'"'"'s face;

(ii) sorting candidate feature locations of a current head pose to determine a likely head pose.

13. A method as claimed in any one of claims 10 to claim 12 wherein said method degrades gracefully in the presence of occlusion of the observer'"'"'s face.

14. A method as claimed in claim 13 wherein said occlusion includes the wearing of spectacles or sunglasses by a user.

24. A method for measuring operator fatigue wherein eye gaze is measured using the method of any one of claims 4 or 5 or 10 to 14.

17. A method of detecting a flexible feature on a person'"'"'s face, said method including the step of

comparing the variance of said feature'"'"'s position from an expected position of said facial feature, based on a head-pose measurement, with a threshold variance to determine whether the feature is flexible.

22. A method for measuring operator fatigue including the step of:

tracking any one or more of;

head-pose;

eye gaze;

eye closure amount;

or blink rate of the operator.

28. A method of detecting eye, gaze substantially as hereinbefore described with reference to the accompanying drawings.

29. A method of detecting head-pose substantially as hereinbefore described with reference to the accompanying drawings.

30. A method of blink detection substantially as hereinbefore described with reference to the accompanying drawings.

31. A method of determining eye closure amount substantially as hereinbefore described with reference to the accompanying drawings.