Building systems for adaptive tracking of facial features across individuals and groups

• US 9,104,908 B1
• Filed: 09/06/2013
• Issued: 08/11/2015
• Est. Priority Date: 05/22/2012
• Status: Active Grant

• Current Assignee: Image Metrics Ltd. (Image Metrics, Inc.)
• Original Assignee: Image Metrics Ltd. (Image Metrics, Inc.)
• Inventors: Rogers, Michael, Walker, Kevin, Williams, Tomos G., Deena, Salil
• Primary Examiner(s): Liew, Alex
• Application Number:
  US14/019,748
• Time in Patent Office: 704 Days
• Field of Search: 382115-118, 382155-161
• US Class Current: 1/1
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06V 10/7557 based on appearance, e.g. active appearance models [AAM]
• CPC Class Codes: G06V 40/176 Dynamic expression

Abstract

Computer implemented methods for generating a non-transient record of feature locations and/or facial expression parameters characterizing a person'"'"'s face. A video sequence of a specified individual person is received and a feature locator update model is applied to the video sequence. The feature locator update model is derived by defining a set of training images, generating a set of facial feature displacements for each training image with associated image sample vectors, and training a regularized linear regression which maps from image sample vectors to displacement vectors, wherein the regularization includes a spatial smoothness term within the shape-free sample space. A feature location and/or a facial expression parameter is then extracted, based on the feature update model, characterizing the location, and/or the expression, of a selected set of features of the face of the specified individual person that correspond to an adaptive set of feature locations.

Claims

1. A computer implemented method for generating a non-transient record embodying measures of an adaptive set of feature locations characterizing a face of a person, the method comprising:

a. receiving a video sequence that constitutes a physical record of the face of a specified individual person;

b. applying a feature locator update model to the video sequence, the feature locator update model derived by steps of;

i. defining a set of training data consisting of training images with associated facial feature locations;

ii. generating a set of facial feature displacements for each training image with associated image sample vectors; and

iii. training a regularized linear regression which maps from image sample vectors to displacement vectors, wherein the regularized linear regression is characterized by regularization that includes a spatial smoothness term within the shape-free sample space;

c. extracting a plurality of locations corresponding to the adaptive set of feature locations;

d. based on the feature locator update model, characterizing the locations of features of the face of the specified individual person; and

e. storing the locations of features to the non-transient record.

2. A computer-implemented method in accordance with claim 1, further comprising selecting features included in the adaptive set of feature locations.

3. A computer-implemented method in accordance with claim 2, wherein the step of selecting features is performed with reference to a threshold.

4. A computer-implemented method in accordance with claim 1, further comprising:

a. extracting a facial expression parameter, based on the adaptive set of feature locations, characterizing the expression of the feature of the face of the specified individual person; and

b. storing the facial expression parameter to the non-transient record.

5. A computer-implemented method in accordance with claim 1, wherein the feature locator update model is trained on the basis of a plurality of human subjects.

6. A method in accordance with claim 5, wherein the feature locator update model is trained on the basis of calibration images of each of the plurality of human subjects.

7. A computer-implemented method in accordance with claim 5, further comprising:

a. training the feature locator update model from the plurality of subjects in such a manner as to incorporate information from calibration images of each of a plurality of subjects;

b. obtaining a calibration image of a specific subject;

c. tailoring the feature locator update model to the specific individual using the captured calibration image; and

d. applying the tailored feature locator to the video sequence of the specified individual person.

8. A computer-implemented method, in accordance with claim 1, wherein the feature locator update model is derived using a further step of computing matrices representing the contribution of optical flow to an update equation using a set of annotated training images during an offline training phase.

9. A computer-implemented method, in accordance with claim 5, wherein the feature locator update model is derived using a further step of combining a vector of calibration samples said matrices to “

bake”

the optical flow calculation into the feature locator update model in such a manner as to retain an original computational cost.

10. A computer implemented method in accordance with claim 1, wherein the adaptive set is determined by at least one of the following conditions:

subject head-pose, expression, occlusion and lighting conditions.

11. A computer implemented method in accordance with claim 1, wherein the feature locator update model is adapted to characterize solely such locations corresponding to the adaptive set of feature locations.

12. A computer-implemented method in accordance with claim 1, the method further comprising:

a. creating a plurality of expression extractor containing controls representing both identity and expression variation;

b. fitting, in a fitting process, 3D pose, identity and expression parameters to features from a specified individual person in a calibration pose;

c. fixing a set of identity controls to produce a tailored person-specific expression extractor; and

d. applying the above fitting process on a person-specific expression extract to the video sequence that constitutes a physical record of the face of a specified individual person to produce facial expression parameters,

wherein the facial expression parameters represent facial feature locations which are uncontaminated by identity variation between individuals.

13. A computer-implemented method in accordance with claim 1, wherein the computer-implemented method is performed on a mobile device.

14. A computer-implemented method in accordance with claim 1, wherein the computer-implemented method is performed on a smart TV.

15. A computer-implemented method in accordance with claim 7, further comprising measuring an expression of the person in response to pre-rendered video footage.

16. A computer-implemented method in accordance with claim 7, further comprising measuring an expression of the person in response to dynamically rendered display content.

17. A computer-implemented method in accordance with claim 16, wherein the dynamically rendered display content includes a software display.

18. A computer-implemented method in accordance with claim 7, further comprising producing real-time pre-visualisation of character animation during performance capture in media generation.

19. A computer-implemented method in accordance with claim 7, further comprising creating video content containing avatar animation.

20. A computer-implemented method in accordance with claim 7, further comprising creating character animation in real-time.

21. A non-transitory computer readable medium for use on a computer system for storing computer-readable program code, the computer readable program code comprising:

a. a computer code module for receiving a video sequence that constitutes a physical record of the face of a specified individual person;

b. a computer code module for applying a feature locator update model to the video sequence;

c. a computer code module for defining a set of training data consisting of training images with associated facial feature locations;

d. a computer code module for generating a set of facial feature displacements for each training image with associated image sample vectors;

e. a computer code module for training a regularized linear regression which maps from image sample vectors to displacement vectors, wherein the regularized linear regression is characterized by regularization that includes a spatial smoothness term within the shape-free sample space, thereby generating the feature locator update model;

f. a computer code module for extracting a plurality of feature locations corresponding to an adaptive set of feature locations;

g. a computer code module characterizing the plurality of feature locations of the face of the specified individual person based on the feature update model; and

h. a computer code module for storing the plurality of feature locations to a non-transient medium.

22. A non-transitory computer readable medium in accordance with claim 21, further comprising:

i. a computer code module for extracting a facial expression parameter, based on the feature update model, characterizing an expression of a feature of the face of the specified individual person; and

j. a computer code module for storing the facial expression parameter to the non-transient record.