Method for lighting- and view -angle-invariant face description with first- and second-order eigenfeatures

• US 6,996,257 B2
• Filed: 12/18/2001
• Issued: 02/07/2006
• Est. Priority Date: 12/19/2000
• Status: Expired due to Term

• Current Assignee: Panasonic Intellectual Property Corporation of America (Panasonic Holdings Corporation)
• Original Assignee: Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Inventors: Wang, Lei
• Primary Examiner(s): Mehta, Bhavesh M.
• Assistant Examiner(s): Strege, John B.
• Application Number:
  US10/020,293
• Publication Number:
  US 20020114464A1
• Time in Patent Office: 1,512 Days
• Field of Search: 382/118, 382/190, 382/117, 382/115, 340/5.53, 340/5.83
• US Class Current: 382/118
• CPC Class Codes: G06F 18/213 Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation

Abstract

This invention provides a method to interpret and recognize huma faces. The 2^nd-order eigenfeatures are used for lighting-invariant face description and both the 1^st-order and 2^nd-order eigenfeatures are used for view-angle-invariant face description. After normalization and quantization, these features can describe face effectively and efficiently. In order to further reduce the size of face descriptor, variable length code can be used to encode the quantized eigenfeatures. This invention can be used in internet multimedia database retrieval, digital library, video editing, surveillance and tracking, and other applications using face recognition and verification.

Claims

1. A method of extracting features for a lighting-invariant face description, comprising:

getting adjusted second-order eigenfeatures of a face image;

quantizing the adjusted second-order eigenfeatures; and

selecting features to construct a face descriptor to describe faces from the quantized second-order eigenfeatures.

5. The method of claim 1, wherein getting adjusted second-order eigenfeatures of a face image comprising:

getting a dot product of the face image and an adjusted second-order eigenface matrix.

8. The method of claim 5, further comprising computing the adjusted second-order eigenface matrix by:

calculating a second-order eigenface matrix; and

adjusting the second-order eigenface matrix.

10. The method of claim 8, wherein calculating second-order eigenfaces comprises:

calculating a first-order eigenface matrix;

getting first-order eigenfeatures from training face images;

calculating a pseudo-inverse of the first-order eigenface matrix;

calculating first-order reconstructed face images by multiplying the first-order eigenfeatures to the pseudo-inverse of the first-order eigenface matrix;

getting second-order residue images by subtracting the first-order reconstructed face images from original images; and

getting the second-order eigenfaces by calculating eigenvectors of the second-order residue images.

11. The method of claim 8, wherein adjusting second-order eigenfaces comprises:

getting second-order eigenfeatures of training images by computing dot products of face images and second-order eigenfaces;

re-shaping the second-order eigenfaces to an original image shape and getting left-right mirrored eigenface images;

weighting the mirrored eigenface images and adding them to corresponding original image shaped second-order eigenface images;

re-arranging original image shaped second-order eigenfaces to obtain one-dimensional adjusted second-order eigenfaces;

normalizing adjusted second-order eigenfaces;

getting weights of second-order eigenfeatures for a distance computation;

multiplying the weights of second-order eigenfeatures for the distance computation to adjusted second-order eigenface matrix; and

quantizing the second-order eigenface matrix.

13. The method of claim 11, wherein quantizing the second-order eigenface matrix comprises:

getting a maximum and a minimum of said adjusted second-order eigenface matrix;

getting a quantization by dividing an interval between the maximum and the minimum into a plurality of quantization levels;

dividing the adjusted second-order eigenface matrix with the quantization; and

rounding divided values to nearest integers.

14. The method of claim 1, wherein getting adjusted second-order eigenfeatures comprises:

getting a recovered adjusted second-order eigenface by multiplying the quantized second-order eigenface matrix with a quantization; and

getting second-order eigenfeatures by multiplying each column-wise recovered adjusted second-order eigenface with a row-wise image.

16. The method of claim 1, wherein quantizing adjusted second-order eigenfeatures comprises:

getting a maximum and a minimum of the adjusted second-order eigenfeatures of training images;

getting the quantization by dividing an interval between the maximum and the minimum into a plurality of quantization levels;

dividing adjusted second-order eigenfeatures with the quantization; and

rounding divided values to nearest integers.

17. The method of claim 1, wherein quantizing adjusted second-order eigenfeatures comprises:

getting a maximum and a minimum of adjusted second-order eigenfeatures of training images;

getting the quantization by dividing an interval between the maximum and the minimum into a plurality of quantization levels;

dividing adjusted second-order eigenfeatures with the quantization;

rounding divided values to nearest integers;

allocating different number of bits to different eigenfeatures by computing a rounded logarithm of a standard deviation of corresponding eigenfeatures in a training set divided by a minimum of the standard deviations; and

quantizing the eigenfeatures according to the corresponding bit allocation.

21. The method of claim 11, wherein getting weights of adjusted second-order eigenfeatures for a distance computation comprises:

getting variances of adjusted second-order eigenfeatures of training face images; and

getting weights of adjusted second-order eigenfeatures by extracting square roots of said variances.

22. A method of measuring lighting-invariant similarity between faces, comprising:

extracting eigenfeatures of faces for a lighting-invariant face description with the method of claim 1;

getting Euclidean distances of eigenfeatures of the faces; and

choosing a smallest Euclidean distance to indicate a best matching pair of faces.

24. A method of getting code tables of variable length coding (VLC) for a light-invariant face descriptor, comprising:

getting quantized eigenfeatures of a training set with the method of claim 1;

classifying eigenfeatures into groups depending on bit allocations; and

constructing a code table for each group of eigenfeatures with the same bit allocation, using an entropy coding method.

26. The method of claim 24, wherein the entropy coding method comprises a Huffman coding method based on a probability of a quantization level.

27. The method of claim 24, wherein the entropy coding method comprises an Arithmetic coding method based on a probability of a quantization level.

28. The method of claim 1, further comprising coding a lighting-invariant face descriptor by:

looking up a code table generated for each quantized eigenfeature and using a corresponding code word to represent the quantized eigenfeature.

2. A method of extracting features for a lighting-invariant face description, comprising:

getting adjusted second-order eigenfeatures of a face image;

quantizing the adjusted second-order eigenfeatures;

selecting features to construct a face descriptor to describe faces from the quantized second-order eigenfeatures; and

coding the selected eigenfeatures in a lighting-invariant face descriptor.

3. A method of extracting features for a view-angle-invariant face description, comprising:

getting adjusted first-order eigenfeatures of a face image;

getting adjusted second-order eigenfeatures of a face image;

quantizing the adjusted first-order eigenfeatures;

quantizing the adjusted second-order eigenfeatures; and

selecting features to construct a face descriptor to describe faces from the quantized first-order and second-order eigenfeatures.

6. The method of claim 3, wherein getting adjusted first-order eigenfeatures of a face image comprises:

getting a dot product of the face image and an adjusted first-order eigenface matrix.

7. The method of claim 6, further comprising computing an adjusted first-order eigenface matrix by:

calculating a first-order eigenface matrix; and

adjusting the first-order eigenface matrix.

9. The method of claim 7, wherein adjusting the first-order eigenfaces comprises:

getting first-order eigenfeatures of training face images;

arranging the first-order eigenface as a two dimensional array of original images;

getting a mirrored eigenface of the two dimensional array;

weighting the mirrored eigenface image;

adding the weighted eigenface image to the first-order eigenface;

re-arranging the first-order eigenface to obtain a one-dimensional adjusted first-order eigenface;

normalizing adjusted first-order eigenface;

getting weights of adjusted first-order eigenfeatures for a distance computation;

multiplying the weights of the first-order eigenfeatures for distance computation to adjusted first-order eigenface matrix; and

quantizing the first-order eigenface matrix.

12. The method of claim 9, wherein quantizing the first-order eigenface matrix comprises:

getting a maximum and a minimum of adjusted first-order eigenface matrix;

getting a quantization by dividing an interval between the maximum and the minimum into a plurality of quantization levels;

dividing the adjusted first-order eigenface matrix with the quantization; and

rounding divided values to nearest integers.

15. The method of claim 3, wherein getting adjusted first-order eigenfeatures comprises:

getting a recovered adjusted first-order eigenface by multiplying the quantized first-order eigenface matrix with a quantization; and

getting first-order eigenfeatures by multiplying each column-wise recovered adjusted first-order eigenface with a row-wise image.

18. The method of claim 3, wherein quantizing adjusted first-order eigenfeatures comprises:

getting a maximum and a minimum of adjusted first-order eigenfeatures of training images;

getting the quantization by dividing an interval between the maximum and the minimum into a plurality of quantization levels;

dividing the adjusted first-order eigenfeatures with the quantization; and

rounding divided values to nearest integers.

19. The method of claim 3, wherein quantizing adjusted first-order eigenfeatures comprises:

getting a maximum and a minimum of adjusted first-order eigenfeatures of training images;

getting the quantization by dividing an interval between the maximum and the minimum into a plurality of quantization levels;

dividing the adjusted first-order eigenfeatures with quantization;

rounding divided values to nearest integers;

allocating different number of bits to different eigenfeatures by computing a rounded logarithm of a standard deviation of corresponding eigenfeatures in a training set divided by a minimum of the standard deviations; and

quantizing eigenfeatures according to the corresponding bit allocation.

20. The method of claim 9, wherein getting weights of adjusted first-order eigenfeatures for a distance computation comprises:

getting standard deviations of adjusted first-order eigenfeatures of training face images; and

getting weights by extracting square roots of standard deviations.

23. A method of measuring a view-angle-invariant similarity between faces, comprising:

extracting eigenfeatures of faces for a view-angle-invariant face description with the method of claim 3;

getting Euclidean distances of eigenfeatures of the faces; and

choosing a smallest Euclidean distance to indicate a best matching pair of faces.

25. A method of getting code tables of variable length coding (VLC) for a view-angle-invariant face descriptor, comprising:

getting quantized eigenfeatures of a training set with the method of claim 3;

classifying eigenfeatures into groups depending on bit allocations; and

constructing a code table for each group of eigenfeatures with the same bit allocation, using an entropy coding method.

29. The method of claim 3, further comprising coding a viewing-angle-invariant face descriptor by:

looking up a code table generated for each quantized eigenfeature and using a corresponding code word to represent the quantized eigenfeature.

30. A method of extracting features for a general face description, comprising:

getting adjusted first-order eigenfeatures of a face image with the method in claim 7;

quantizing adjusted first-order eigenfeatures; and

selecting features to construct face descriptor to describe faces from the quantized first-order eigenfeatures.

31. A method of extracting features for general face description, comprising:

getting adjusted first-order eigenfeatures of a face image with the method in claim 7;

quantizing adjusted first-order eigenfeatures;

selecting features to construct face descriptor to describe faces from the quantized first-order eigenfeatures; and

coding selected eigenfeatures in the face descriptor.

32. The method of claim 30, wherein selecting features to construct a face descriptor to describe faces from the quantized first-order eigenfeatures comprises:

selecting eigenfeatures corresponding to a top N largest eigenvalues for a view-angle invariant face description.

33. The method of claim 30, wherein selecting features to construct a face descriptor to describe faces from the quantized first-order eigenfeatures comprises:

selecting eigenfeatures corresponding to a k-th to N-th largest eigenvalues (0<

k<

N) for a lighting-invariant face description.

4. A method of extracting features for a view-angle-invariant face description, comprising:

getting adjusted first-order eigenfeatures;

getting adjusted second-order eigenfeatures;

quantizing the adjusted first-order eigenfeatures;

quantizing the adjusted second-order eigenfeatures;

selecting features to construct a face descriptor to describe faces from the quantized first-order and second-order eigenfeatures; and

coding selected eigenfeatures in a view-angle-invariant face descriptor.

34. A method of lighting-invariant a view-angle-invariant face description, comprising:

getting first-order residual image data Γ

⁽¹⁾, by taking face image data Φ

as a vector and calculating its difference from a mean face image data Ψ

;

multiplying the first-order residual image data Γ

⁽¹⁾ and a combination of a first-order eigenmatrix obtained by decomposing the first-order residual image data Γ

⁽¹⁾, and a second-order eigenmatrix obtained by decomposing second-order residual image data Γ

⁽²⁾ obtained by subtracting first-order reconstructed image data from original face image data Φ

, where the first-order reconstructed image data is obtained by adding mean face image data Ψ and

a substantial low-frequency component extracted from the first-order residual image data Γ

⁽¹⁾,

taking a result of the multiplication as an eigenfeature of the face image data Φ

;

quantizing the eigenfeature;

encoding the quantized eigenfeature into variable length codes; and

taking a result of encoding as the face descriptor.