System and method for face recognition robust to multiple degradations

• US 10,002,286 B1
• Filed: 04/28/2016
• Issued: 06/19/2018
• Est. Priority Date: 04/28/2015
• Status: Active Grant

• Current Assignee: Carnegie Mellon University
• Original Assignee: Carnegie Mellon University
• Inventors: Prabhu, Utsav, Savvides, Marios
• Primary Examiner(s): Werner, Brian
• Application Number:
  US15/140,857
• Time in Patent Office: 782 Days
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06V 10/70 using pattern recognition or machine learning optical pattern recognition or electronic computations therefor G06V10/88
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

A novel, unified technique for analyzing, synthesizing, and recognizing faces captured by various modalities, and under a variety of often seen real-world circumstances, using thin-plate splines for densification of points of a face from an initial set of fiducial points.

Claims

1. A system for correcting for degradations in a 2D image of a face captured with an image sensor, comprising:

a computing device executing software for performing the functions of;

(a) placing a plurality of fiducial points at landmarks on said face, each of said fiducial points corresponding to a fiducial point for a mean face calculated over a database set of faces;

(b) calculating a function which captures the deviation of each fiducial point from said mean fiducial point;

(c) using a thin-plate spline function to create a test face by minimizing a function which provides a measure of the smoothness of the function calculated in step (b);

(d) selecting a set of uniformly-distributed points on said mean face; and

(e) finding a set of points on said test face corresponding to said uniformly-distributed set of points on said mean face.

2. The system of claim 1 wherein said degradation is a 3D pose variation, said software further performing the functions of:

adjusting said 2D image to correct for pitch, roll and yaw;

forming triangles from said set of points on said test face;

computing the normal direction n to all of said triangles;

computing the vertex normals as a weighted average of the normal directions of the triangles that it belongs to, where the weights are in direct proportion to the area of each triangle; and

using said vertex normals as an indicator of the visibility of the corresponding vertex.

3. The system of claim 1 wherein said degradation is an occlusion, said software further performing the functions of:

accepting an occlusion mask defining areas of occlusion; and

approximating the locations of fiducial points in said areas of occlusion.

4. The system of claim 3 wherein said approximated locations of fiducial points in said areas of occlusion are accepted manually.

5. The system of claim 4 wherein textural and structural information in the occluded area of said test face is ignored.

6. The system of claim 1 wherein said degradation is low resolution, said software further performing the function of:

computing said thin-plate spline-based representation of said face using a mean face from said database which has been lowered in resolution to match the resolution of said image.

7. The system of claim 1 wherein said software further performs the function of providing a textural value for each point in said set of points on said test face.

8. The system of claim 7 wherein said textural value is a grayscale value.

9. The system of claim 1 wherein said software further performs the function of calculating an observational confidence value for each point in said set of points on said test face.

10. The system of claim 9 wherein a point of said test face is assigned a placeholder value if the actual value of said point is not available due to degradation, and further wherein the corresponding observational confidence value is set to 0.

11. The system of claim 9 wherein said observational confidence values are calculated as a function of a camera projection model and a 3D rotation estimate.

12. The system of claim 11 wherein said 3D rotation estimate is calculated along the pitch and yaw axes.

13. The system of claim 2 wherein said visibility of each point in said test face is computed by:

forming triangles between all of said points on said test face;

computing the normal direction to all of said triangles; and

computing the normals of each of said points as the weighted average of all triangles to which said point belongs.

14. The system of claim 13 wherein said weights are proportional to the area of each of said triangles.

15. The system of claim 14 wherein said visibility of each point in said test face is a function of the angle between said normal of each point and the viewing direction.

16. The system of claim 1 wherein said landmarks on said face are at pre-defined locations and further wherein there are 79 fiducial points.

17. The system of claim 1 further comprising using said face corrected for degradations in a face-recognition application.