Augmented facial animation

• US 9,830,728 B2
• Filed: 12/23/2014
• Issued: 11/28/2017
• Est. Priority Date: 12/23/2014
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Fang, Yikai, Du, Yangzhou, Li, Qiang Eric, Tong, Xiaofeng, Li, Wenlong, Park, Minje, Ju, Myung-Ho, Yi, Jihyeon Kate, Kim, Tae-Hoon Pete
• Primary Examiner(s): Du, Haixia
• Application Number:
  US14/779,501
• Publication Number:
  US 20160328875A1
• Time in Patent Office: 1,071 Days
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20092 Interactive image processing based on input by user
• CPC Class Codes: G06T 2207/20182 Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Examples of systems and methods for augmented facial animation are generally described herein. A method for mapping facial expressions to an alternative avatar expression may include capturing a series of images of a face, and detecting a sequence of facial expressions of the face from the series of images. The method may include determining an alternative avatar expression mapped to the sequence of facial expressions, and animating an avatar using the alternative avatar expression.

Claims

1. A system for mapping facial expressions to an alternative avatar expression, the system comprising:

an image capture device to capture a series of images of a face; and

a processor to;

detect a sequence of facial expressions of the face from the series of images;

determine the alternative avatar expression is mapped to a subset of facial expressions of the sequence of facial expressions, wherein the alternative avatar expression is an exaggerated expression beyond capabilities of a human to express on a face; and

animate an avatar using the alternative avatar expression and a sequence of avatar expressions based on the sequence of facial expressions, wherein the alternative avatar expression and the sequence of avatar expressions are used concurrently to animate the avatar.

2. The system of claim 1, wherein to detect the sequence of facial expressions, the processor is further to detect the sequence of facial expressions during a specific time frame.

3. The system of claim 1, wherein to determine the alternative avatar expression, the processor is further to use hand gestures from the series of images.

4. The system of claim 1, wherein to determine the alternative avatar expression, the processor is further to use shoulder motions from the series of images.

5. The system of claim 1, wherein to detect the sequence of facial expressions, the processor is further to detect movement of a specified landmark point on the face.

6. The system of claim 5, wherein to detect movement of the specified landmark point on the face, the processor is further to detect repeated movement.

7. The system of claim 5, wherein to detect movement of the specified landmark point on the face, the processor is further to use dynamic time warping.

8. The system of claim 1, wherein the alternative avatar expression is a user defined alternative avatar expression.

9. The system of claim 1, wherein the processor is further to animate the avatar by mimicking the sequence of facial expressions.

10. The system of claim 9, wherein the processor is further to animate the avatar in a blended animation when mimicking the sequence of facial expressions and using the alternative avatar expression.

11. The system of claim 10, wherein the processor is further to smooth out interference in the blended animation by removing artifacts.

12. The system of claim 1, wherein the alternative avatar expression includes a Graphics Interchange Format (GIF) image.

13. The system of claim 1, wherein the processor is further to detect the sequence of facial expressions of the face after receiving a user indication to enter an alternative avatar expression mode.

14. The system of claim 1, wherein the processor is further to determine an emotion classification expressed in a facial expression from the sequence of facial expressions.

15. The system of claim 14, wherein to determine the alternative avatar expression, the processor is further to determine the alternative avatar expression using the emotion classification.

16. At least one non-transitory machine-readable medium including instructions for mapping a facial expression to an alternative avatar expression, which when executed by a machine, cause the machine to:

capture a series of images of a face;

detect a sequence of facial expressions of the face from the series of images;

determine the alternative avatar expression is mapped to a subset of facial expressions of the sequence of facial expressions, wherein the alternative avatar expression is an exaggerated expression beyond capabilities of a human to express on a face; and

animate an avatar using the alternative avatar expression and a sequence of avatar expressions based on the sequence of facial expressions, wherein the alternative avatar expression and the sequence of avatar expressions are used substantially concurrently to animate the avatar.

17. The at least one non-transitory machine readable-medium of claim 16, further comprising operations to animate the avatar by mimicking the sequence of facial expressions.

18. The at least one non-transitory machine-readable medium of claim 17, further comprising operations to animate the avatar in a blended animation when mimicking the sequence of facial expressions and using the alternative avatar expression.

19. The at least one non-transitory machine-readable medium of claim 16, wherein operations to detect the sequence of facial expressions include operations to detect movement of a specified landmark point on the face.

20. The at least one non-transitory machine-readable medium of claim 19, wherein operations to detect movement of the specified landmark point on the face include operations to detecting repeated movement.

21. The at least one non-transitory machine-readable medium of claim 19, wherein operations to detect movement of the specified landmark point on the face include using dynamic time warping.

22. The at least one non-transitory machine-readable medium of claim 16, further comprising operations to receive a user indication selecting the alternative avatar expression.

23. A method for mapping facial expressions to an alternative avatar expression, the method comprising:

capturing a series of images of a face;

detecting a sequence of facial expressions of the face from the series of images;

determining the alternative avatar expression is mapped to a subset of facial expressions of the sequence of facial expressions, wherein the alternative avatar expression is an exaggerated expression beyond capabilities of a human to express on a face; and

animating an avatar using the alternative avatar expression and a sequence of avatar expressions based on the sequence of facial expressions, wherein the alternative avatar expression and the sequence of avatar expressions are used concurrently to animate the avatar.

24. The method of claim 23, wherein detecting the sequence of facial expressions includes detecting the sequence of facial expressions during a specific time frame.