System method for automatically detecting neutral expressionless faces in digital images

• US 20030133599A1
• Filed: 01/17/2002
• Published: 07/17/2003
• Est. Priority Date: 01/17/2002
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: International Business Machines Corporation
• Inventors: Tian, Ying-Li, Bolle, Rudolf M.
• Granted Patent: US 6,879,709 B2
• Time in Patent Office: Days
• US Class Current: 382/118
• CPC Class Codes: G06V 40/175 Static expression

Abstract

A system and method for automatic detecting neutral expressionless faces in digital images and video is described. First a face detector is used to detect the pose and position of a face and find the facial components. Second, the detected face is normalized to a standard size face. Then a set of geometrical facial features and three histograms in zones of mouth are extracted. Finally, by feeding these features to a classifier, the system detects if there is the neutral expressionless face or not.

Claims

1. A system for automatically detecting neutral, expressionless face images in digital images, the system comprising:

an image acquisition unit;

an face detector receiving input from the image acquisition unit, detecting one or more face subimages of one or more faces in the image;

a characteristic point detector that receives input from the face detector and that estimates one or more characteristic facial features as characteristic points in each detected face subimage, one of the characteristic points being a first and a second mouth corner on a mouth of the face;

a facial feature detector that detects one or more contours of one or more facial components;

a facial feature analyzer that determines a mouth shape of the mouth from the contour of the mouth and creates a representation of the mouth shape, the mouth being one of the facial components; and

a face classification unit that classifies the representation into one of a neutral class and a non-neutral class.

2. A system, as in claim 1, where the mouth shape is determined by a width of the mouth, the first distance from the first mouth corner to a reference in the coordinate systems, and the second distance from the second mouth corner to the reference in the coordinate systems.

3. A system, as in claim 1, where one or more of the characteristic facial features is an eye openness measure of an eye of the facial image and the representation includes the eye openness measure.

4. A system, as in claim 1, where one or more characteristic facial features is derived from any one of an eye region and an eyebrow region in the face subimage.

5. A system, as in claim 1, where the contours are extracted by an edge detector.

6. A system, as in claim 1, where the characteristic facial features are estimated by template matching techniques.

7. A system, as in claim 1, where th e facial components include any one or more of the following:

a mouth, an eye, an eyebrow, a nose, an ear, and a wrinkle.

8. A system, as in claim 1, where a mouth position of the mouth is determined with respect to a geometric reference.

9. A system, as in claim 1, where a mouth position of the mouth is determined with respect to an image coordinate system.

10. A system, as in claim 1, where the characteristic points define one or more characteristic distances to one or more respective geometric references.

11. A system, as in claim 10, where one or more of the characteristic distances is normalized to another of the characteristic distances.

12. A system, as in claim 1, where the characteristic points further include a location of each eye on the face, being eye characteristic points and one of the characteristic distances is a distance between the eye characteristic points, being an eye separation distance.

13. A system, as in claim 12, where one or more of the other characteristic distances is normalized by the eye separation distance.

14. A system, as in claim 12, where the eye characteristic points are located on the eye pupil.

15. A system, as in claim 10, where the facial feature analyzer divides the face sub-image into two or more zones.

16. A system, as in claim 15, where facial feature analyzer determines one or more component shapes from one or more respective component contours in one or more of the zones.

17. A system, as in claim 16, where the component is the mouth and the shape of the mouth is extracted from three of the zones containing the mouth.

18. A system, as in claim 17, where the shape of the mouth is determined by histograming the contours of the mouth in each of the three zones containing the mouth.

19. A system as in claim 18 where the mouth shape is defined by three shape histograms each corresponding to one of the three zones containing the mouth obtained by quantizing the contour of the mouth into four directions.

20. A system, as in claim 17, where a further component is one or more of an eye and an eyebrow that are extracted from three of the zones containing the eyes.

21. A system, as in claim 10, where one or more of the characteristic distances is the normalized width of the mouth, the width of the mouth being the distance between the first and second mouth corners.

22. A system, as in claim 3, where the eye openness measure of the eyes is measured by a normalized height of the eyes.

23. A system, as in claim 12, where a mouth-eye distance between the first mouth corner and a first eye is normalized by the eye separation distance, and the normalized mouth-eye distance is an additional feature used by the face classification unit.

24. A system, as in claim 12, where a brow-eye distance between a first eyebrow and a first eye is normalized by the eye separation distance, and the normalized brow-eye distance is an additional feature used by the face classification unit.

25. A system, as in claim 12, where a brow distance between the first eyebrow and the second eyebrow is normalized by the eye separation distance, and the normalized brow distance is an additional feature used by the face classification unit.

26. A system, as in claim 1, where one or more normalized characteristic distances is used by the face classification unit classify if the face is neutral.

27. A system, as in claim 26, where the face classification unit is one or more of the following classifiers:

neural networks, nearest neighbor methods, Bayesian inference methods, linear discriminant methods, perceptrons, support vector machines, and related variants.

28. A system, as in claim 26, where the face classification unit is neural network (NN).

29. A system, as in claim 1, where the face classification unit is used to enroll one or more subjects in a face-based authentication system with a neutral face.

30. A system, as in claim 1, where the face classification unit is used to authenticate one or more subjects in a face-based authentication system with a neutral face.

31. A system, as in claim 1, used as a user interface that reacts to one or more human emotion states.

32. A system, as in claim 1, used as a surveillance device.

33. A system, as in claim 32, used in one or more of the following:

a financial institution, a bank a retail store, a credit card check, a check in, an air port, an air plane boarding check, a building access system, and a object access authorization system.

34. A system, as in claim 1, used as an information retrieval device.

35. A system, as in claim 34, where the information retrieval device includes any one or more of the following:

a video retrieval of a frame containing a face image with a specified expression, an image of an image containing a face image with a specified expression.

36. A method for automatically detecting neutral, expressionless face images in digital images, the method comprising the steps of:

acquiring an image;

detecting one or more face subimages of one or more faces in the image;

estimating one or more characteristic facial features as characteristic points in each detected face subimage, one of the characteristic points being a first and a second mouth corner on a mouth of the face;

detecting one or more contours of one or more facial components;

determining a mouth shape of the mouth from the contour of the mouth;

creating a representation of the mouth shape, the mouth being one of the facial components; and

classifying the representation into one of a neutral class and a non-neutral class.

37. A system for automatically detecting neutral, expressionless face images in digital images, the method comprising:

means for acquiring an image;

means for detecting one or more face subimages of one or more faces in the image;

means for estimating one or more characteristic facial features as characteristic points in each detected face subimage, one of the characteristic points being a first and a second mouth corner on a mouth of the face;

means for detecting one or more contours of one or more facial components;

means for determining a mouth shape of the mouth from the contour of the mouth;

means for creating a representation of the mouth shape, the mouth being one of the facial components; and

means for classifying the representation into one of a neutral class and a non-neutral class.

38. A computer program product having a computer program for automatically detecting neutral, expressionless face images in digital images, the computer program comprising the steps of:

acquiring an image;

detecting one or more face subimages of one or more faces in the image;

estimating one or more characteristic facial features as characteristic points in each detected face subimage, one of the characteristic points being a first and a second mouth corner on a mouth of the face;

detecting one or more contours of one or more facial components;

determining a mouth shape of the mouth from the contour of the mouth;

creating a representation of the mouth shape, the mouth being one of the facial components; and

classifying the representation into one of a neutral class and a non-neutral class.