System and method for multi-view face detection

• US 7,050,607 B2
• Filed: 03/04/2002
• Issued: 05/23/2006
• Est. Priority Date: 12/08/2001
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, HongJiang, Li, ZiQing, Zhang, ZhenQiu
• Primary Examiner(s): Chen, Wenpeng
• Assistant Examiner(s): SHAH, UTPAL D
• Application Number:
  US10/091,100
• Publication Number:
  US 20030108244A1
• Time in Patent Office: 1,541 Days
• Field of Search: 382/118, 382/205, 382/159, 382/181
• US Class Current: 382/118
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/24765 Rule-based classification
• CPC Class Codes: G06V 10/446 using Haar-like filters, e.g. using integral image techniques
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation

Abstract

A system and method for real-time multi-view (i.e. not just frontal view) face detection. The system and method uses a sequence of detectors of increasing complexity and face/non-face discriminating thresholds to quickly discard non-faces at the earliest stage possible, thus saving much computation compared to prior art systems. The detector-pyramid architecture for multi-view face detection uses a coarse-to-fine and simple-to-complex scheme. This architecture solves the problem of lengthy processing that precludes real-time face detection effectively and efficiently by discarding most of non-face sub-windows using the simplest possible features at the earliest possible stage. This leads to the first real-time multi-view face detection system which has the accuracy almost as good as the state-of-the-art system yet 270 times faster, allowing real-time performance.

Claims

1. A computer-implemented face detection process for detecting a person'"'"'s face in an input image and identifying a face pose range into which the face pose exhibited by the detected face falls, comprising using a computer to perform the following process actions:

creating a database comprising a plurality of training feature characterizations, each of which characterizes the face of a person at a known face pose or a non-face;

training a plurality of detectors arranged in a pyramidal architecture comprising three detector layers and wherein said first detector layer comprises a single full-view detector responsible for the full range of −

90 to 90 degrees of face pose, with 0 degrees being frontal view;

said second detector layer comprises a first, second and third detector, said first detector being capable of detecting face pose ranges of −

90 to 40 degrees, said second detector being capable of detecting face pose ranges of −

30 to 30 degrees, and said third detector being capable of detecting face pose range of 40 to 90 degrees;

said third detector layer comprising nine detectors, capable of detecting face pose ranges of −

90 to −

80 degrees, −

70 to −

60 degrees, −

50 to −

40 degrees, −

30 to −

20 degrees, −

10 to 10 degrees, 20 to 30 degrees, 40 to 50 degrees, 60 to 70 degrees, and 80 to 90 degrees, respectively, to determine whether a portion of an input image depicts a person'"'"'s face having a face pose falling within a face pose range associated with one of the detectors using the training feature characterizations; and

wherein

said detectors using a greater number of feature characterizations are arranged at the bottom of the pyramid, and wherein

said detectors arranged to detect finer ranges of face pose are arranged at the bottom of the pyramid;

inputting a portion of an input image into the plurality of detectors arranged in a pyramid architecture; and

interpreting the output of the plurality of detectors to determine whether the portion of the input image contains a face and if so to identify the pose associated with each detected face.

2. The process action of claim 1, wherein the process action of inputting a region from the input image comprises the action of:

partitioning said input image into sub-windows.

3. The process action of claim 2 wherein said partitioning of said input image into sub-windows comprises moving a search window of a prescribed size across the input image and prior to each shift extracting the pixels contained within the search window to create an input image region.

4. The process of claim 3, further comprising the process actions of:

employing a search window having a size approximately equal to the size of the smallest face it is anticipated will be depicted in the input image and which it is desired to detect;

after regions from every part of the input image it is desired to screen for faces have been extracted, reducing the size of the input image by a prescribed scale increment;

progressively shifting the search window across the reduced input image and prior to each shift extracting the pixels contained within the search window to create an input image region; and

repeating the reducing and shifting process actions until a prescribed reduction limit is reached.

5. The process of claim 3, wherein the search window size corresponds to the size of the training images.

6. The process of claim 3, wherein the search window size is the size of the smallest detectable face anticipated to be found in the input image.

7. The process of claim 6, wherein the search window size is 20 by 20 pixels.

8. The process of claim 3, wherein the initial search window size is increased by a scale factor in a step-wise fashion all the way up to the input image size;

and after each increase in scale partitioning the input image with the search sub-window size.

9. The process of claim 3 wherein the original sub-window size matches the entire image and this sub-window is then scaled down on an incremental basis.

10. The process of claim 1, wherein the process action of inputting a portion of an input image into the plurality of detectors arranged in a pyramid architecture comprises:

inputting a portion of the input image into a first detector layer;

if the portion of the input image is rejected by the detector at the top layer, it is classified as a non-face region it is not processed by detectors in later detector layers;

if the portion of the input image is processed by the detectors in the first detector layer, it is processed by the second layer, if a detector in the second layer classifies the input image portion as a non-face region it is not processed by detectors in the third layer;

if the portion of the input image is processed by the detectors in the second detector layer, it is processed by the third detector layer, which classifies the input image region into a face pose range corresponding to a detector trained to detect a given face pose range.

11. A computer-implemented face detection process for detecting a person'"'"'s face in an input image and identifying a face pose range into which the face pose exhibited by the detected face falls, comprising using a computer to perform the following process actions:

creating a database comprising a plurality of training feature characterizations, each of which characterizes the face of a person at a known face pose or a non-face;

training a plurality of detectors arranged in a pyramidal architecture to determine whether a portion of an input image depicts a person'"'"'s face having a face pose falling within a face pose range associated with one of the detectors using the training feature characterizations; and

wherein

said detectors using a greater number of feature characterizations are arranged at the bottom of the pyramid, and wherein

said detectors arranged to detect finer ranges of face pose are arranged at the bottom of the pyramid;

inputting a portion of an input image into the plurality of detectors arranged in a pyramid architecture, wherein inputting a portion of an input image into the plurality of detectors arranged in a pyramid architecture further comprises arbitrating between two or more detectors that detect a face in the same detector layer to determine if the detections represent two different faces or two detections of one face, wherein arbitrating between two or more detectors further comprises

determining if the detections by each of the two or more detectors overlap;

specifying that if the detections by each of the two or more detectors do not overlap then arbitration is not necessary and each face detection is determined to be a separate face;

combining the output of some of the detector view ranges into one class by creating new classes of view ranges from the various pose range detectors at the detector pyramid'"'"'s outputs;

arbitrating between the new classes of view ranges to categorize each overlapping detection into one of the new classes of view ranges; and

interpreting the output of the plurality of detectors to determine whether the portion of the input image contains a face and if so to identify the pose associated with each detected face.

12. The process of claim 11 wherein the aribitrating between the new classes of view ranges comprises using Rowley'"'"'s heuristic method.

13. The process of claim 12 wherein arbitrating between the new classes of view ranges comprises

determining whether a face detection at any detector is identified as a frontal face;

if said face detection is determined to be a frontal face then all other face locations detected by profile or half profile detectors that are overlapping in the input image are determined to be errors and are eliminated, and the face detection that is determined to be a frontal face is determined to be a single frontal face;

if the face detection is not identified as a frontal face, determining whether the given location is identified as a half frontal face;

if the location is identified as a half profile face then all other locations detected by profile face detectors are eliminated and the particular location is determined to be a half profile face; and

if the location is not a non-face, nor a frontal face, nor a half profile face, then the location is determined to be a profile face.