Method and system for face recognition using deep collaborative representation-based classification

• US 9,430,697 B1
• Filed: 07/03/2015
• Issued: 08/30/2016
• Est. Priority Date: 07/03/2015
• Status: Active Grant

• Current Assignee: TCL Research America, Inc. (TCL Technology Group Corp.)
• Original Assignee: TCL Research America, Inc. (TCL Technology Group Corp.)
• Inventors: Kappeler, Armin, Iliadis, Michael, Wang, Haohong
• Primary Examiner(s): Seth, Manav
• Application Number:
  US14/791,388
• Time in Patent Office: 424 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06V 10/772 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: H04N 2201/3274 Storage or retrieval of prestored additional information

Abstract

The present invention provides a face recognition method. The method includes obtaining a plurality of training face images which belongs to a plurality of face classes and obtaining a plurality of training dictionaries corresponding to the training face images. A face class includes one or more training face images. The training dictionaries include a plurality of deep feature matrices. The method further includes obtaining an input face image. The input face image is partitioned into a plurality of blocks, whose corresponding deep feature vectors are extracted using a deep learning network. A collaborative representation model is applied to represent the deep feature vectors with the training dictionaries and representation vectors. A summation of errors for all blocks corresponding to a face class is computed as a residual error for the face class. The input face image is classified by selecting the face class that yields a minimum residual error.

Claims

1. A face recognition method on a computing device, comprising:

obtaining a plurality of training face images which belongs to a plurality of face classes, wherein a face class includes one or more training face images and represents an identification of the one or more training face images;

obtaining a plurality of training dictionaries corresponding to the plurality of training face images, wherein the plurality of training dictionaries include a plurality of deep feature matrices;

obtaining an input face image;

partitioning the input face image into a plurality of blocks;

extracting corresponding deep feature vectors of the plurality of blocks of the input face image using a deep learning network;

applying a collaborative representation model to represent the deep feature vectors of the blocks of the input face image with the training dictionaries and representation vectors;

computing residual errors for the face classes, a residual error for a face class being a summation of errors for all blocks corresponding to the training face images in the face class, wherein an error for a block exists between a feature vector of the block in the input face image and the collaborative representation model of the block corresponding to the face class;

classifying the input face image by selecting a face class that yields a minimum residual error as a recognition face class; and

presenting the recognition face class of the input face image.

2. The face recognition method according to claim 1, wherein:

the training face images are partitioned into a plurality of non-overlapping blocks;

a deep feature matrix is extracted from blocks at a same location in the training face images using the deep learning network; and

the plurality of blocks of the input face image are non-overlapping.

3. The face recognition method according to claim 1, wherein:

the deep learning network is a convolutional neural network; and

the convolutional neural network includes at least five convolutional layers, max-pooling layers and two fully connected layers.

4. The face recognition face recognition method according to claim 1, wherein:

obtaining an input face image further includes;

dividing an input video into different sets of frames;

detecting faces of each frame in the input video;

generating face tracks for the whole video, a face track being a group of detected faces within a same camera take; and

obtaining detected faces and face tracks information, wherein the input face image is obtained from the detected faces; and

presenting the face class of the input face image further includes outputting a video by adding annotations about the face classes of the detected faces in the input video according to the face tracks.

5. The face recognition method according to claim 1, further comprising:

storing user preference information along with the training dictionaries corresponding to one or more users; and

after obtaining the face class of the input face image, recommending personalized contents to a user corresponding to the face class according to the user preference information.

6. The face recognition method according to claim 1, wherein:

provided that T denotes the training face images, y denotes the input face image, the training face images and the test face image are partitioned into p blocks, F(•

) denotes deep features extracted from blocks in training face images, F(T^r) denotes the deep feature matrix of a r^th block in the training face images, F(T_i^r) is a submatrix of F(T^r) and denotes the deep feature for the r^th block in a i^th face class, f(y^r) denotes the deep feature vector of the r^th block from the input face image, a*r denotes a resultant representation vector of the r^th block obtained from the collaborative representation model, w*^r denotes a weight corresponds to the resultant representation vector a*^r, a*_i^r denotes a segment of a resultant representation vector a*^r associated with class i, and e_i(y) denotes the residual error for the class i,

the residual for the face class i is computed as e_i(y)=Σ

_r=1^pw*^r|f(y^r)−

F(T_i^r)a*_i^r∥

₂²; and

the identity of y is given as, Identity(y)=argmin_i{e_i}.

7. The face recognition method according to claim 1, wherein

provided that T denotes the training face images, y denotes the input face image, the training face images and the test face image are partitioned into p blocks, F(®

) denotes deep features extracted from blocks in training face images, F(T^r) denotes the deep feature matrix of a r^th block in the training face images, f(y^r) denotes the deep feature vector of the r^th block from the input face image, a^r denotes a representation vector of the r^th block, and a*^r denotes a resultant representation vector of the r^th block,

the representation vector is obtained by solving a relaxed collaborative representation problem

8. The face recognition method according to claim 7, wherein the relaxed collaborative representation problem is solved in an alternating fashion between a^r and w^r, by solving a regularized least squares problem for the coefficients a^r having the weights w^r and a fixed, and updating w^r and ā

by fixing a^r.

9. The face recognition method according to claim 1, wherein

provided that T denotes the training face images, y denotes the input face image, the training face images and the test face image are partitioned into p blocks, F(•

) denotes deep features extracted from blocks in training face images, F(T^r) denotes the deep feature matrix of a r^th block in the training face images, f(y^r) denotes the deep feature vector of the r^th block from the input face image, a^r denotes a representation vector of the r^th block, and a*^r denotes a resultant representation vector of the r^th block,

the representation vector is obtained by solving a collaborative representation problem with regularized least square

10. A face recognition system based on a computing device, comprising:

a storage module configured to store a plurality of training face images which belongs to a plurality of face classes, wherein a face class includes one or more training face images and represents an identification of the one or more training face images, and to store a plurality of training dictionaries corresponding to the plurality of training face images, wherein the plurality of training dictionaries include a plurality of deep feature matrices;

an acquisition module configured to obtain an input face image;

a computation module configured to;

partition the input face image into a plurality of blocks;

extract corresponding deep feature vectors of the plurality of blocks of the input face image using a deep learning network;

apply a collaborative representation model to represent the deep feature vectors of the blocks of the input face image with the training dictionaries and representation vectors;

compute residual errors for the face classes, a residual error for a face class being a summation of errors for all blocks corresponding to the training face images in the face class, wherein an error for a block exists between a feature vector of the block in the input face image and the collaborative representation model of the block corresponding to the face class; and

classify the input face image by selecting a face class that yields a minimum residual error as a recognition face class; and

an output module configured to present the recognition face class of the input face image.

11. The face recognition system according to claim 10, wherein:

the training face images are partitioned into a plurality of non-overlapping blocks;

a deep feature matrix is extracted from blocks at a same location in the training face images using the deep learning network; and

the plurality of blocks of the input face image are non-overlapping.

12. The face recognition system according to claim 10, wherein:

the deep learning network is a convolutional neural network; and

the convolutional neural network includes at least five convolutional layers, max-pooling layers and two fully connected layers.

13. The face recognition system according to claim 10, wherein:

the acquisition module is further configured to;

divide an input video into different sets of frames;

detect faces of each frame in the input video;

generate face tracks for the whole video, a face track being a group of detected faces within a same camera take; and

obtain detected faces and face track information, wherein the input face image is obtained from the detected faces; and

the output module is further configured to output a video by adding annotations about the face class of the detected faces in the input video according to the face tracks.

14. The face recognition system according to claim 10, wherein:

the storage module is further configured to store user preference information along with the training dictionaries corresponding to one or more users; and

the output module is further configured to recommend personalized contents to a user corresponding to the face class according to the user preference information.

15. The face recognition system according to claim 10, wherein:

provided that T denotes the training face images, y denotes the input face image, the training face images and the test face image are partitioned into p blocks, F(•

) denotes deep features extracted from blocks in training face images, F(T^r) denotes the deep feature matrix of a r^th block in the training face images, F(T_i^r) is a submatrix of F(T^r) and denotes the deep feature for the r^th block in a i^th face class, f(y^r) denotes the deep feature vector of the r^th block from the input face image, a*^r denotes a resultant representation vector of the r^th block obtained from the collaborative representation model, w*^r denotes a weight corresponds to the resultant representation vector a*^r, a*_i^r denotes a segment of a resultant representation vector a*^r associated with class i, and e_i(y) denotes the residual error for the class i,

the residual for each class i is computed as e_i(y)Σ

_r=1^pw^r∥

f(y^r)−

F(T_i^r)a*_i^r∥

₂²;

and

the identity of y is given as, Identity(y)=argmin_i{e_i}.

16. The face recognition system according to claim 10, wherein:

provided that T denotes the training face images, y denotes the input face image, the training face images and the test face image are partitioned into p blocks, F(•

) denotes deep features extracted from blocks in training face images, F(T^r) denotes the deep feature matrix of a r^th block in the training face images, f(y^r) denotes the deep feature vector of the r^th block from the input face image, a^r denotes a representation vector of the r^th block, and a*^r denotes a resultant representation vector of the r^th block,

the representation vector is obtained by solving a relaxed collaborative representation problem

17. The face recognition system according to claim 16, wherein the relaxed collaborative representation problem is solved in an alternating fashion between a^r and w^r, by solving a regularized least squares problem for the coefficients a^r having the weights w^r and a fixed, and updating w^r and ā

by fixing a^r.

18. The face recognition system according to claim 10, wherein:

provided that T denotes the training face images, y denotes the input face image, the training face images and the test face image are partitioned into p blocks, F(•

) denotes deep features extracted from blocks in training face images, F(T^r) denotes the deep feature matrix of a r^th block in the training face images, f(y^r) denotes the deep feature vector of the r^th block from the input face image, a^r denotes a representation vector of the r^th block, and a*^r denotes a resultant representation vector of the r^th block,

the representation vector is obtained by solving a collaborative representation problem with regularized least square