Methods and software for hallucinating facial features by prioritizing reconstruction errors

• US 10,115,004 B2
• Filed: 06/17/2015
• Issued: 10/30/2018
• Est. Priority Date: 06/17/2014
• Status: Active Grant

• Current Assignee: Carnegie Mellon University
• Original Assignee: Carnegie Mellon University
• Inventors: Juefei-Xu, Felix, Pal, Dipan K., Savvides, Marios
• Primary Examiner(s): Vu, Kim
• Assistant Examiner(s): BLOOM, NATHAN J
• Application Number:
  US15/307,099
• Publication Number:
  US 20170046562A1
• Time in Patent Office: 1,231 Days
• Field of Search: None
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06V 10/772 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

Identifying a masked suspect is one of the toughest challenges in biometrics that exist. This is an important problem faced in many law-enforcement applications on almost a daily basis. In such situations, investigators often only have access to the periocular region of a suspect'"'"'s face and, unfortunately, conventional commercial matchers are unable to process these images in such a way that the suspect can be identified. Herein, a practical method to hallucinate a full frontal face given only a periocular region of a face is presented. This approach reconstructs the entire frontal face based on an image of an individual'"'"'s periocular region. By using an approach based on a modified sparsifying dictionary learning algorithm, faces can be effectively reconstructed more accurately than with conventional methods. Further, various methods presented herein are open set, and thus can reconstruct faces even if the algorithms are not specifically trained using those faces.

Claims

1. A method of hallucinating facial features of a first face by prioritizing reconstruction errors, wherein the first face is present in an image in which a first portion of the first face is un-occluded and a second portion of the first face is occluded, the method comprising:

receiving the image of the first face, the first portion containing one or more first facial features;

training a machine-learning algorithm using a set of images each containing a region of a face of an individual corresponding to the first portion of the first face and a region of the face of the individual corresponding to the second portion of the first face so as to produce machine-learning data or receiving machine-learning data corresponding to a previous implementation of such training; and

hallucinating one or more second facial features within the second portion of the first face as a function of the machine-learning data by prioritizing reconstruction errors for hallucinating the one or more second facial features such that reconstruction error for the one or more first facial features is minimized with a higher priority than reconstruction error for hallucinating the one or more second facial features.

2. A method according to claim 1, wherein the first portion of the first face is a periocular region.

3. A method according to claim 1, wherein the machine-learning algorithm utilizes a K-SVD algorithm, a convolutional neural network algorithm, an autoencoder algorithm, a sparse filtering algorithm, a method of optimized directions algorithm, an online dictionary learning method, a recursive least squares dictionary learning algorithm, or a combination thereof.

4. A method according to claim 1, wherein the machine-learning algorithm utilizes a K-SVD algorithm and said training further comprises:

generating a first dictionary as a function of the regions of the faces of individuals corresponding to the first portion in the set of images; and

generating a second dictionary as a function of both the regions of the faces of individuals corresponding to the first portion and the regions of the faces of the second portion in the set of images.

5. A method according to claim 4, further comprising jointly and simultaneously minimizing reconstruction error for the one or more first facial features and reconstruction error for hallucinating the one or more second facial features for the same sparse coefficients.

6. A method according to claim 5, wherein jointly and simultaneously minimizing reconstruction errors includes forcing consistent sparse representations during said training.

7. A method according to claim 1, wherein the set of images consists of images of individuals of a particular gender or set of genders, ethnicity or set of ethnicities, age or range of ages, or a combination thereof, wherein the particular gender or set of genders, ethnicity or set of ethnicities, age or range of ages, or combination thereof is selected as a function of the first face.

8. A method according to claim 1, further comprising generating a composite image including the first and second features of the first face.

9. A method according to claim 8, further comprising comparing the composite image to one or more images of individuals, each image containing at least a region of a face of an individual corresponding to the first portion of the first face and a region of the face of the individual corresponding to the second portion.

10. A method according to claim 1, wherein prioritizing reconstruction errors for hallucinating the one or more second facial features is optimized by cross-validation.

11. A machine-readable storage medium containing machine-executable instructions for performing a method of hallucinating facial features of a first face by prioritizing reconstruction errors, wherein the first face is present in an image in which a first portion of the first face is un-occluded and a second portion of the first face is occluded, said machine-executable instructions comprising:

a first set of machine-executable instructions for receiving the image of the first face, the first portion containing one or more first facial features;

a second set of machine-executable instructions for training a machine-learning algorithm using a set of images each containing a region of a face of an individual corresponding to the first portion of the first face and a region of the face of the individual corresponding to the second portion of the first face so as to produce machine-learning data or receiving machine-learning data corresponding to a previous implementation of such training; and

a third set of machine-executable instructions for hallucinating one or more second facial features with the second portion of the first face as a function of the machine-learning data by prioritizing reconstruction errors for hallucinating the one or more second facial features such that reconstruction error for the one or more first facial features is minimized with a higher priority than reconstruction error for hallucinating the one or more second facial features.

12. A machine-readable storage medium according to claim 11, wherein the first portion of the first face is a periocular region.

13. A machine-readable storage medium according to claim 11, wherein the machine-learning algorithm utilizes a K-SVD algorithm, a convolutional neural network algorithm, an autoencoder algorithm, a sparse filtering algorithm, a method of optimized directions algorithm, an online dictionary learning method, a recursive least squares dictionary learning algorithm, or a combination thereof.

14. A machine-readable storage medium according to claim 11, wherein the machine-learning algorithm utilizes a K-SVD algorithm and said second set of machine-executable instructions for training further comprises:

a set of machine-executable instructions for generating a first dictionary as a function of the regions of the faces of individuals corresponding to the first portion in the set of images; and

a set of machine-executable instructions for generating a second dictionary as a function of both the regions of the faces of individuals corresponding to the first portion and the regions of the faces of the second portion in the set of images.

15. A machine-readable storage medium according to claim 14, further comprising a fourth set of machine-executable instructions for jointly and simultaneously minimizing reconstruction error for the one or more first facial features and reconstruction error for hallucinating the one or more second facial features for the same sparse coefficients.

16. A machine-readable storage medium according to claim 15, wherein jointly and simultaneously minimizing reconstruction errors includes forcing consistent sparse representations during said training.

17. A machine-readable storage medium according to claim 11, wherein the set of images consists of images of individuals of a particular gender or set of genders, ethnicity or set of ethnicities, age or range of ages, or a combination thereof, wherein the particular gender or set of genders, ethnicity or set of ethnicities, age or range of ages, or combination thereof is selected as a function of the first face.

18. A machine-readable storage medium according to claim 11, further comprising a fourth set of machine-executable instructions for generating a composite image including the first and second features of the first face.

19. A machine-readable storage medium according to claim 18, further comprising a fifth set of machine-executable instructions for comparing the composite image to one or more images of individuals, each image containing at least a region of a face of an individual corresponding to the first portion of the first face and a region of the face of the individual corresponding to the second portion.

20. A machine-readable storage medium according to claim 11, wherein prioritizing reconstruction errors for hallucinating the one or more second facial features is optimized by cross-validation.