Face hallucination using convolutional neural networks

• US 9,405,960 B2
• Filed: 06/17/2014
• Issued: 08/02/2016
• Est. Priority Date: 06/17/2014
• Status: Active Grant

• Current Assignee: Beijing Kuangshi Technology Co., Ltd.
• Original Assignee: Beijing Kuangshi Technology Co., Ltd.
• Inventors: Yin, Qi, Cao, Zhimin, Zhou, Erjin
• Primary Examiner(s): Chawan, Sheela C
• Application Number:
  US14/375,683
• Publication Number:
  US 20150363634A1
• Time in Patent Office: 777 Days
• Field of Search: 382/118, 382/156, 382/155, 382/168, 382/255, 382/263, 382/264, 382/274, 382/275
• US Class Current: 1/1
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/164 using holistic features
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

Face hallucination using a bi-channel deep convolutional neural network (BCNN), which can adaptively fuse two channels of information. In one example, the BCNN is implemented to extract high level features from an input image. The extracted high level features are combined with low level details in the input image to produce the higher resolution image. Preferably, a proper coefficient is obtained to adaptively combine the high level features and the low level details.

Claims

1. A system for generating higher resolution output face images from input face images, the system comprising:

a convolutional neural network (CNN) that generates a face representation of an input face image, the CNN including convolution, non-linearity and down-sampling;

a face hallucinator that generates a hallucinated face image from the face representation, the hallucinated face image having a higher resolution than the input face image;

a coefficient estimator that generates a coefficient from the face representation; and

a face combiner that combines the hallucinated face image with an up-sampled version of the input face image to produce an output face image, wherein the face combiner generates the output face image as a linear combination of the hallucinated face image and the up-sampled version of the input face image, and the coefficient determines the linear combination,

wherein the output face image has more texture than the hallucinated face image.

2. The system of claim 1 wherein the face representation is a representation of features in the input face image, and the CNN extracts the features from the input face image.

3. The system of claim 2 wherein the CNN comprises two or more CNN layers in a cascade progressing from lower level features to higher level features.

4. The system of claim 1 wherein the face hallucinator is a neural network that generates the hallucinated face image from the face representation.

5. The system of claim 4 wherein the face hallucinator is a fully-connected neural network.

6. The system of claim 1 wherein the coefficient estimator is a neural network that generates the coefficient from the face representation.

7. The system of claim 6 wherein the coefficient estimator is a fully-connected neural network.

8. The system of claim 1 wherein the up-sampling is based on bicubic interpolation.

9. The system of claim 1 wherein the output face image contains at least four times a number of pixels in the input face image.

10. The system of claim 1 wherein the output face image is at least 100×

100 pixels.

11. The system of claim 1 wherein the input face image is not more than 50×

50 pixels.

12. The system of claim 1 wherein the output face image compensates for Gaussian blur in the input face image.

13. The system of claim 1 wherein the output face image compensates for motion blur in the input face image.

14. The system of claim 1 further comprising:

a landmark detection module that detects face landmarks from the output face image.

15. The system of claim 1 wherein:

the coefficient estimator is a fully-connected neural network; and

the face hallucinator is a fully-connected neural network that generates the hallucinated face image from the face representation.

16. The system of claim 15 wherein the output face image contains at least four times a number of pixels in the input face image.

17. A method for generating higher resolution output face images from input face images, the method comprising:

generating a face representation of an input face image;

generating a hallucinated face image from the face representation, the hallucinated face image having a higher resolution than the input face image;

generating a coefficient from the face representation;

up-sampling the input face image; and

combining the hallucinated face image with the up-sampled input face image to produce an output face image, wherein the output face image is generated as a linear combination of the hallucinated face image and the up-sampled input face image, and the coefficient determines the linear combination,

wherein the output face image has more texture than the hallucinated face image, and

wherein the method is performed with one or more processors.

18. A non-transitory computer readable medium configured to store program code, the program code comprising instructions for generating higher resolution output face images from input face images, the instructions when executed by a processor cause the processor to:

generate a face representation of an input face image;

generate a hallucinated face image from the face representation, the hallucinated face image having a higher resolution than the input face image;

generate a coefficient from the face representation;

up-sample the input face image; and

combine the hallucinated face image with the up-sampled input face image to produce an output face image, wherein the output face image is generated as a linear combination of the hallucinated face image and the up-sampled input face image, and the coefficient determines the linear combination,

wherein the output face image has more texture than the hallucinated face image.