Emotion augmented avatar animation

• US 10,176,619 B2
• Filed: 07/30/2015
• Issued: 01/08/2019
• Est. Priority Date: 07/30/2015
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Jiao, Shaohui, Shen, Xiaolu, Zhang, Lidan, Li, Qiang, Li, Wenlong
• Primary Examiner(s): Nguyen, Vu
• Application Number:
  US15/102,200
• Publication Number:
  US 20170206694A1
• Time in Patent Office: 1,258 Days
• Field of Search: None
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 2207/20164 Salient point detection; Corner detection
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2210/56 Particle system, point based geometry or rendering
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 40/174 Facial expression recognition

Abstract

Apparatuses, methods and storage medium associated with emotion augmented animation of avatars are disclosed herein. In embodiments, an apparatus may comprise an animation augmentation engine to receive facial data of a user, analyze the facial data to determine an emotion state of the user, and drive additional animation that supplements animation of the avatar based at least in part on a result of the determination of the emotion state of the user. Other embodiments may be described and/or claimed.

Claims

1. An apparatus for animating an avatar, comprising:

one or more processors; and

an animation augmentation engine to be operated by the processor to receive facial expression data of a user, analyze the facial expression data to determine an emotion state of the user, the emotion state being one of a plurality of emotion states including at least one of a disgusted state, a fear state, or a surprise state, and drive additional animation that supplements animation of the avatar based at least in part on a result of the determination of the emotion state of the user, the additional animation and the animation of the avatar being separate animations;

wherein the animation augmentation engine is to further determine an intensity of the determined emotion state, acquire a movement trajectory, and calculate velocity in the movement trajectory for a collection of particles to be emitted from an emitter based at least in part on the determined intensity of the determined emotion state.

2. The apparatus of claim 1, wherein to receive facial expression data, the animation augmentation engine is to receive facial landmark data of a facial mesh of the user.

3. The apparatus of claim 1, wherein to analyze the facial expression data, the animation augmentation engine comprises a plurality of emotion classifiers to respectively process the facial expression data, and generate a plurality of confidence values for corresponding emotion states of the emotion classifiers.

4. The apparatus of claim 3, wherein the plurality of emotion classifiers comprise at least one of a disgusted state classifier, a fear state classifier, or a surprise state classifier.

5. The apparatus of claim 1, wherein to drive additional animation, the animation augmentation engine is to create the emitter for the additional animation, and generate the collection of particles to be emitted from the emitter.

6. The apparatus of claim 1, further comprising a facial expression tracker to be operated by the one or more processors to receive one or more image frames of the user, analyze the one or more image frames to detect and track facial expressions of the user, and to provide the facial expression data to the animation augmentation engine based at least in part on a result of the analysis of the one or more image frames.

7. The apparatus of claim 6, further comprising an avatar animation engine to be operated by the one or more processors;

wherein the facial expression tracker is to drive the avatar animation engine to animate the avatar based on facial expressions of the user detected and tracked; and

wherein to drive additional animation, the animation augmentation engine is to drive the avatar animation engine to separately supplement the animation of the avatar based on facial expressions of the user detected and tracked, with the emotion state based additional animation.

8. The apparatus of claim 7, wherein the facial expression tracker comprises a primary animation message generator to generate primary animation messages based on facial expressions of the user detected and tracked, and send the primary animation messages to the avatar animation engine to drive the avatar animation engine to animate the avatar based on facial expressions of the user;

and the animation augmentation engine comprises a supplemental animation message generator to generate supplemental animation messages based on the emotion state of the user detected, and send the supplemental animation messages to the avatar animation engine to drive the avatar animation engine to separately supplement the animation of the avatar based on facial expressions of the user, with the emotion state based additional animation.

9. The apparatus of claim 6, wherein the apparatus is a selected one of a smartphone, a computing tablet, an ultrabook, an ebook, or a laptop computer.

10. A method for animating an avatar, comprising:

receiving, by a computing device, facial expression data of a user;

analyzing, by the computing device, the facial expression data to determine an emotion state of the user, the emotion state being one of a plurality of emotion states including at least one of a disgusted state, a fear state, or a surprise state;

driving, by the computing device, additional animation that supplements animation of the avatar based at least in part on a result of the determination of the emotion state of the user, the additional animation and the animation of the avatar being separate animations;

determining, by the computing device, an intensity of the determined emotion state;

acquiring, by the computing device, a movement trajectory; and

calculating, by the computing device, velocity in the movement trajectory for a collection of particles to be emitted from an emitter, based at least in part on the determined intensity of the determined emotion state.

11. The method of claim 10, wherein receiving facial expression data comprises receiving facial landmark data of a facial mesh of the user.

12. The method of claim 10, wherein analyzing the facial expression data to determine an emotion state of the user further comprises analyzing the facial expression data to determine the intensity of the determined emotion state;

wherein driving additional animation comprises creating an emitter for the additional animation, and generating a collection of particles to be emitted from the emitter, the collection of particles to be emitted being associated with the determined emotion state of the user.

13. The method of claim 10, further comprising receiving one or more image frames of the user, analyzing the one or more image frames to detect and track facial expressions of the user, and providing the facial expression data based at least in part on a result of the analysis of the one or more image frames.

14. The method of claim 13, further comprising driving animation of the avatar based on facial expressions of the user detected and tracked;

and wherein driving additional animation comprises separately supplementing the animation of the avatar based on facial expressions of the user detected and tracked, with the emotion state based additional animation.

15. One or more non-transitory computer-readable media comprising instructions that cause a computing device, in response to execution of the instructions by the computing device, to operate an animation augmentation engine to:

receive facial expression data of a user,

analyze the facial expression data to determine an emotion state of the user, the emotion state being one of a plurality of emotion states including at least one of a disgusted state, a fear state, or a surprise state, and

drive additional animation that supplements animation of an avatar based at least in part on a result of the determination of the emotion state of the user, the additional animation and the animation of the avatar being separate animations;

wherein the computing device is to further determine an intensity of the determined emotion state, acquire a movement trajectory, and calculate velocity in the movement trajectory for a collection of particles to be emitted from an emitter based at least in part on the determined intensity of the determined emotion state.

16. The non-transitory computer-readable media of claim 15, wherein to receive facial expression data comprises to receive facial landmark data of a facial mesh of the user.

17. The non-transitory computer-readable media of claim 15, wherein to analyze the facial expression data comprises to respectively process the facial expression data with a plurality of emotion classifiers to generate a plurality of confidence values for corresponding emotion states of the emotion classifiers.

18. The non-transitory computer-readable media of claim 17, wherein the plurality of emotion classifiers comprise at least one of a disgusted state classifier, a fear state classifier, or a surprise state classifier.

19. The non-transitory computer-readable media of claim 17, wherein to drive additional animation comprises to create the emitter for the additional animation, and generate a collection of particles to be emitted from the emitter, the collection of particles to be emitted being associated with the determined emotion state of the user.

20. The non-transitory computer-readable media of claim 15, wherein the computing device is further caused to receive one or more image frames of the user, analyze the one or more image frames to detect and track facial expressions of the user, and to provide the facial expression data based at least in part on a result of the analysis of the one or more image frames.

21. The non-transitory computer-readable media of claim 20, wherein the computing device is further caused to drive an avatar animation engine to animate the avatar based on facial expressions of the user detected and tracked;

and wherein to drive additional animation comprises to separately supplement the animation of the avatar based on facial expressions of the user detected and tracked, with the emotion state based additional animation.

22. The non-transitory computer-readable media of claim 21, wherein to drive animation comprises to generate primary animation messages based on facial expressions of the user detected and tracked, and send the primary animation messages to the avatar animation engine to drive the avatar animation engine to animate the avatar based on facial expressions of the user;

and to drive additional animation comprises to generate supplemental animation messages based on the emotion state of the user detected, and send the supplemental animation messages to the avatar animation engine to drive the avatar animation engine to separately supplement the animation of the avatar based on facial expressions of the user, with the emotion state based additional animation.

23. The non-transitory computer-readable media of claim 15, wherein to analyze the facial expression data to determine an emotion state of the user further comprises to analyze the facial expression data to determine a sub-state of the determined emotion state of the user;

wherein to drive additional animation that separately supplements animation of the avatar comprises to drive additional animation that separately supplements animation of the avatar, based at least in part on a result of the determination of the emotion state of the user and a result of the determination of the sub-state of the determined emotion state of the user.

24. The non-transitory computer-readable media of claim 15, wherein the plurality of emotion states includes a happy state, and the happy state includes a romantic sub-state;

and wherein on determination of the user being in the happy-romantic state, wherein to drive additional animation that separately supplements animation of the avatar comprises to drive additional animation of a shower of hearts to separately supplement animation of the avatar, based at least in part on the determined happy-romantic state.