Facial image processing system

• US 7,043,056 B2
• Filed: 01/24/2003
• Issued: 05/09/2006
• Est. Priority Date: 07/24/2000
• Status: Active Grant

• Current Assignee: Seeing Machines LTD
• Original Assignee: Seeing Machines Pty Limited
• Inventors: Edwards, Timothy, Heinzmann, Jochen, Zelinsky, Alex, Rougeaux, Sebastien
• Primary Examiner(s): Johns, Andrew W.
• Application Number:
  US10/350,835
• Publication Number:
  US 20030169907A1
• Time in Patent Office: 1,201 Days
• Field of Search: 382/103, 382/104, 382/117, 382/118, 382/154, 382/199, 348/77, 348/78, 340/575, 340/576, 340/573.7
• US Class Current: 382/103
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A method of determining an eye gaze direction of an observer is disclosed comprising the steps of: (a) capturing at least one image of the observer and determining a head pose angle of the observer; (b) utilizing the head pose angle to locate an expected eye position of the observer, and (c) analyzing the expected eye position to locate at least one eye of the observer and observing the location of the eye to determine the gaze direction.

Claims

1. A method of determining a head-pose measurement of a person, said head pose measurement including a rotational displacement and translation displacement, said method including the steps of:

(a) capturing images of the face and head of said person from at least two viewpoints;

(b) computing an initial estimate of head-pose;

(c) computing the three-dimensional location of two or more facial features; and

(d) determining a rotational displacement and translation displacement which best match a head-model to the computed location of said two or more facial features.

2. A method of determining a head-pose measurement of a person as claimed in claim 1 wherein step (b) includes the sub-step of, localizing the face within two or more of the captured images.

3. A method of determining a head-pose measurement of a person as claimed in claim 1 wherein step (c) includes the sub-steps of:

identifying feature search regions, based on said initial estimate of within each captured image that may contain an identifiable facial feature;

measuring the position of two or more facial features within two or more respective feature search regions in each of said captured images.

4. A method for measuring operator fatigue wherein head-pose is measured using the method of claim 3.

5. A method of animating an image, wherein a head-pose measurement of a person is tracked using a method according to claim 1, and wherein said image is animated in response to said head-pose measurement.

6. A method of passively determining the direction of eye gaze of a person, said method including the steps of:

(a) capturing images of the face and head of said person from at least two viewpoints;

(b) localizing the face within two or more of the captured images;

(c) computing an initial estimate of head-pose;

(d) identifying feature search regions, based on said initial estimate of head pose, within each captured image that may contain an identifiable facial feature;

(e) measuring the position of two or more facial features within two or more respective feature search regions in each of said captured images;

(f) computing the three-dimensional location of each of said two or more facial features;

(g) determining a head pose measurement, said head pose measurement including a rotational displacement and translation displacement which best match a head-model to the computed location of said two or more facial features;

(h) using said head pose measurement to determine the location the centre of one eyeball;

(i) using said head pose measurement to find an eye region corresponding with said eyeball, within each captured image, and determining the location of the centre of the iris of said eye;

(j) using the determined position of said centre of said eyeball and the determined location of the centre of the iris of said eye, to calculate an eye-gaze direction for said eye.

7. A method of passively determining the direction of eye gaze of a person, wherein the direction of eye gaze of said person is calculated by averaging the direction of eye gaze of said person'"'"'s two eyes, wherein the direction of eye gaze for each eye is determined using the method of claim 6.

8. A method of determining the eye closure amount of an eye located on a person'"'"'s face, said eye having an eyeball, a top eyelid, and a bottom eyelid, said top and bottom eyelids meeting at a pair of eye corners;

wherein eye closure amount is defined as the ratio of eye corner separation to eyelid distance, said method including the steps of;

(a) capturing images of the face and head of said person from at least two viewpoints;

(b) determining a head pose measurement, from said captured images;

(c) using said head pose measurement to determine an eye region within said captured image;

(d) determining the location of said eye corners within said eye region;

(e) fitting at least two oriented edge detectors to said top eyelid and to said bottom eyelid, such that one of said oriented edge detectors of said top eyelid intersects with one oriented edge detectors of said bottom eyelid at each eye corner;

(f) approximating the position of said top eyelid, and said bottom eyelid, by fitting a parabola to said at least two oriented edge detectors corresponding to each eyelid, such that each parabola is tangential to the at least two oriented edge detectors of said parabola'"'"'s respective eyelid;

(g) determining said eye corner separation by calculating the distance between said eye corners;

(h) determining the maximum eyelid distance by calculating the maximum distance between said parabolas on the portion of said parabolas between said eye corners, in a direction perpendicular to a line joining the eye corners;

(i) calculating the eye closure amount by taking the ratio of the determined eye corner separation to maximum eyelid distance.

9. A method of detecting a blink of a person'"'"'s eye, said method including the following steps:

calculating eye closure value using a method as claimed in claim 8;

forming a temporal sequence of eye closure values; and

comparing said temporal sequence of eye closure values with a blink template, wherein said blink template comprises a sequence of eye closure values corresponding to blink, in order to determine a correlation value between said blink template and said model temporal sequence of eye closure value, wherein a blink is determined to have occurred if said correlation value is greater than a predetermined threshold.

10. A method of detecting a blink of a person'"'"'s eye using a plurality of independent measurements, wherein each of said plurality of measurements produces a time varying output value, and at least one of said plurality of measurements is a measurement of eye closure value which has been calculated using a method as claimed in claim 8 said method including the step of:

combining the output value of two or more of said plurality of independent measurements in a weighted manner to calculate a blink detection value, wherein a blink is determined to have occurred if said blink detection value is greater than a predetermined threshold blink detection value.

11. A method of detecting a blink of a person'"'"'s eye as claimed in claim 10 wherein at least one of said plurality of independent measurements is selected from:

an eye region template correlation coefficient;

an optical flow measurement in said eye region;

eye region image rate of change;

color segmentation within the eye region; and

iris template correlation coefficient.

12. A method for measuring operator fatigue wherein a blink rate is determined using the method of claim 11.

13. An apparatus when implementing a method of claim 12.

14. A method for measuring operator fatigue wherein eye closure amount is measured using the method of claim 8.

15. A method of determining an eye gaze direction of an observer comprising the steps of:

(a) capturing at least one image of the observer and determining a head pose angle of said observer;

(b) utilizing said head pose angle to locate an expected eye position of said observer; and

(c) analyzing said expected eye position to locate at least one eye of said observer and observing the location of the eye to determine an eye gaze direction.

16. A method as claimed in claim 15 wherein said step (a) includes capturing an image in stereo and processing said image in stereo.

17. A method as claimed in claim 16 wherein said step (a) includes the steps of:

(i) determining a set of distinguishable feature locations on an observer'"'"'s face;

(ii) sorting candidate feature locations of a current head pose to determine a likely head pose.

18. A method as claimed in claim 17 wherein said method degrades gracefully in the presence of occlusion of the observer'"'"'s face.

19. A method as claimed in claim 18 wherein said occlusion includes the wearing of spectacles or sunglasses by a user.

20. A method as claimed in claim 19 wherein said method further includes determining the three dimensional position of a center of rotation of an observer'"'"'s eye.

21. A method as claimed in claim 20 wherein said method further determines eye closure and blinking.

22. A method for measuring operator fatigue wherein eye gaze is measured using the method of claim 19.

23. A method of detecting a flexible feature on a person'"'"'s face, said method including the step of

comparing the variance of said feature'"'"'s position from an expected position of said facial feature, based on a head-pose measurement, with a threshold variance to determine whether the feature is flexible.

24. A method as claimed in claim 23 wherein a head-pose measurement is determined using the method of:

(a) capturing images of the face and head of said person from at least two viewpoints;

(b) computing an initial estimate of head pose;

(c) computing the three dimensional location of two or more facial features; and

(d) determining a rotational displacement and translation displacement which best matches a head-model to the computed location of said two or more facial features.

25. A method of analysis of a person performing a task wherein the method of claim 24 is used to analyze an action of said person.

26. A method as claimed in claim 23 wherein said method utilizes a measure feature location to determine if individual features on an observer'"'"'s face are flexible or fixed.