Avatar image animation using translation vectors

• US 10,628,985 B2
• Filed: 11/30/2018
• Issued: 04/21/2020
• Est. Priority Date: 12/01/2017
• Status: Active Grant

• Current Assignee: Affectiva, Inc. (Smart Eye AB)
• Original Assignee: Affectiva, Inc. (Smart Eye AB)
• Inventors: Mishra, Taniya, Reichenbach, George Alexander, el Kaliouby, Rana
• Primary Examiner(s): Robinson, Terrell M
• Application Number:
  US16/206,051
• Publication Number:
  US 20190172243A1
• Time in Patent Office: 508 Days
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/24133 Distances to prototypes
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G10L 13/027 Concept to speech synthesisers; Generation of natural phrases from machine-based concepts generation of parameters for speech synthesis out of text G10L13/08
• CPC Class Codes: G10L 15/005 Language recognition
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/16 using artificial neural networks
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 25/63 for estimating an emotional state
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10

Abstract

Techniques are described for image generation for avatar image animation using translation vectors. An avatar image is obtained for representation on a first computing device. An autoencoder is trained, on a second computing device comprising an artificial neural network, to generate synthetic emotive faces. A plurality of translation vectors is identified corresponding to a plurality of emotion metrics, based on the training. A bottleneck layer within the autoencoder is used to identify the plurality of translation vectors. A subset of the plurality of translation vectors is applied to the avatar image, wherein the subset represents an emotion metric input. The emotion metric input is obtained from facial analysis of an individual. An animated avatar image is generated for the first computing device, based on the applying, wherein the animated avatar image is reflective of the emotion metric input and the avatar image includes vocalizations.

Claims

1. A computer-implemented method for image generation comprising:

obtaining an avatar image for representation on a first computing device;

training an autoencoder, on a second computing device comprising an artificial neural network, to generate synthetic emotive faces;

identifying, using a bottleneck layer within the autoencoder, a plurality of translation vectors corresponding to a plurality of emotion metrics, based on the training;

generating a first set of bottleneck layer parameters, from the bottleneck layer, learned for a neutral face;

applying a subset of the plurality of translation vectors to the avatar image, wherein the subset represents an emotion metric input; and

generating an animated avatar image for the first computing device, based on the applying, wherein the animated avatar image is reflective of the emotion metric input.

2. The method of claim 1 wherein the emotion metric input is obtained from facial analysis of an individual.

3. The method of claim 2 wherein the animated avatar image represents a facial expression of the individual.

4. The method of claim 3 wherein the facial expression is identified using a software development kit.

5. The method of claim 2 wherein the animated avatar image represents an empathetic mirroring of the individual.

6. The method of claim 5 wherein the empathetic mirroring causes the animated avatar image to have a similar expression to the individual.

7. The method of claim 5 wherein the empathetic mirroring causes the animated avatar image to have a complementary expression to the individual.

8. The method of claim 1 wherein the training is based on reinforced learning.

9. The method of claim 1 wherein the autoencoder is a variational autoencoder.

10. The method of claim 1 wherein the autoencoder is a generative autoencoder.

11. The method of claim 1 wherein the training is based on a plurality of facial videos of pre-catalogued facial emotion expressions.

12. The method of claim 1 further comprising generating a second set of bottleneck layer parameters for an emotional face.

13. The method of claim 12 further comprising subtracting the first set of bottleneck layer parameters from the second set of bottleneck layer parameters for use in the plurality of translation vectors.

14. The method of claim 1 wherein the avatar image includes vocalizations.

15. The method of claim 1 wherein the generating further comprises vocalizations based on the emotion metric input.

16. The method of claim 15 wherein the vocalizations include non-speech vocalizations.

17. The method of claim 15 wherein the vocalizations are based on preprocessing a voice used with the animated avatar image.

18. The method of claim 1 wherein the avatar image includes vocal mirroring.

19. The method of claim 1 wherein the avatar image includes complementary emotions.

20. The method of claim 1 wherein the avatar image includes empathetic mirroring.

21. The method of claim 1 further comprising altering a self-avatar image of a person based on facial analysis of images of the person obtained after the self-avatar image was generated.

22. A computer program product embodied in a non-transitory computer readable medium for image generation, the computer program product comprising code which causes one or more processors to perform operations of:

obtaining an avatar image for representation on a first computing device;

training an autoencoder, on a second computing device comprising an artificial neural network, to generate synthetic emotive faces;

identifying, using a bottleneck layer within the autoencoder, a plurality of translation vectors corresponding to a plurality of emotion metrics, based on the training;

generating a first set of bottleneck layer parameters, from the bottleneck layer, learned for a neutral face;

applying a subset of the plurality of translation vectors to the avatar image, wherein the subset represents an emotion metric input; and

generating an animated avatar image for the first computing device, based on the applying, wherein the animated avatar image is reflective of the emotion metric input.

23. A computer system for image generation comprising:

a memory which stores instructions;

one or more processors attached to the memory wherein the one or more processors, when executing the instructions which are stored, are configured to;

obtain an avatar image for representation on a first computing device;

train an autoencoder, on a second computing device comprising an artificial neural network, to generate synthetic emotive faces;

identify, using a bottleneck layer within the autoencoder, a plurality of translation vectors corresponding to a plurality of emotion metrics, based on the training;

generate a first set of bottleneck layer parameters, from the bottleneck layer, learned for a neutral face;

apply a subset of the plurality of translation vectors to the avatar image, wherein the subset represents an emotion metric input; and

generate an animated avatar image for the first computing device, based on the applying, wherein the animated avatar image is reflective of the emotion metric input.