AVATAR-BASED VIDEO ENCODING

• US 20150213604A1
• Filed: 06/04/2013
• Published: 07/30/2015
• Est. Priority Date: 06/04/2013
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Li, Wenlong, Du, Yangzhou, Tong, Xiaofeng
• Granted Patent: US 9,589,357 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

Techniques are disclosed for performing avatar-based video encoding. In some embodiments, a video recording of an individual may be encoded utilizing an avatar that is driven by the facial expression(s) of the individual. In some such cases, the resultant avatar animation may accurately mimic facial expression(s) of the recorded individual. Some embodiments can be used, for example, in video sharing via social media and networking websites. Some embodiments can be used, for example, in video-based communications (e.g., peer-to-peer video calls; videoconferencing). In some instances, use to the disclosed techniques may help to reduce communications bandwidth use, preserve the individual'"'"'s anonymity, and/or provide enhanced entertainment value (e.g., amusement) for the individual, for example.

Claims

1. A computer-readable medium encoded with instructions that, when executed by one or more processors, causes a process for avatar-based video encoding to be carried out, the process comprising:

recording a video of a user; and

reconstructing the recorded video as an avatar-based video using an avatar that is based on one or more facial expressions of the user detected during video recording.

2. The computer-readable medium of claim 1, wherein reconstructing the recorded video as the avatar-based video comprises:

decomposing the recorded video into a video frame stream comprising a plurality of video frames;

for each video frame of the video frame stream, extracting detected facial expression(s) of the user and mapping them to the avatar to animate the avatar;

for each video frame of the video frame stream, presenting the resultant animated avatar in a non-screen frame buffer and extracting a screenshot image thereof; and

encoding the extracted avatar screenshot image(s) to provide the avatar-based video.

3. The computer-readable medium of claim 2, wherein extracting detected facial expression(s) of the user and mapping them to the avatar to animate the avatar comprises:

identifying facial feature points of the user and tracking movements thereof;

determining facial expression(s) from movement of the facial feature points;

retrieving expression sequence(s) similar to the facial expression(s); and

animating the avatar by mapping the retrieved expression sequence(s) thereto.

4. The computer-readable medium of claim 2, wherein presenting the resultant animated avatar in a non-screen frame buffer is performed using an OpenGL application programming interface or an OpenGL for Embedded Systems (OpenGL ES) application programming interface.

5. The computer-readable medium of claim 2, wherein extracting a screenshot image is performed using glReadPixels.

6. The computer-readable medium of claim 2, wherein encoding the extracted avatar screenshot image(s) is performed using a video encoder.

7. The computer-readable medium of claim 2, wherein the decomposing further comprises decomposing the recorded video into an audio stream, and the encoding further comprises combining the extracted avatar screenshot image(s) with the audio stream to provide the avatar-based video.

8. The computer-readable medium of claim 1, wherein prior to reconstructing the recorded video as the avatar-based video, the process further comprises:

receiving a selection of the avatar to represent the user.

9. The computer-readable medium of claim 1, wherein during recording the video of the user, the process further comprises:

extracting from the video a video frame which captures a front-face image of the user; and

generating the avatar from the front-face image of the user.

10. The computer-readable medium of claim 9, wherein after generating the avatar from the front-face image of the user but prior to reconstructing the recorded video as the avatar-based video, the process further comprises:

receiving a selection of the avatar generated from the extracted video frame to represent the user.

11. The computer-readable medium of claim 1, wherein recording the video of the user includes activating a video recording application based on user input or an always-on video sensor that triggers recording.

12. The computer-readable medium of claim 1, wherein the avatar-based video has a file size that is about 20-70% less than the recorded video.

13. The computer-readable medium of claim 1, wherein the avatar-based video is in MP4 file format.

14. The computer-readable medium of claim 1, wherein the avatar-based video is provided with a static image background.

15. A system comprising:

a video device configured to record video;

an imaging and audio framework module configured to receive and interpret video input from the video device;

a face detection module operatively coupled with the imaging and audio framework module and configured to detect a face within the video input and to track changes in facial characteristics of the detected face;

an avatar database having one or more avatars stored therein; and

an avatar control module configured to animate a selected avatar using changes in facial characteristics of the detected face as tracked by the face detection module.

16. The system of claim 15, wherein at least one avatar stored in the avatar database is generated by a remote content provider.

17. The system of claim 15, wherein at least one avatar stored in the avatar database is generated from a video frame of the video input recorded by the video device.

18. The system of claim 15, wherein the video device comprises at least one of a camera, a microphone, and/or a speaker.

19. A computing device comprising the system of claim 15.

20. The computing device of claim 19, wherein the computing device comprises at least one of a laptop/notebook computer, a tablet computer, a mobile phone, a smartphone, a personal digital assistant (PDA), a cellular handset, a handheld gaming device, a gaming platform, a desktop computer, and/or a videoconferencing system.

21. The system of claim 19, wherein the computing device is configured for at least one of wired and/or wireless communication with a network.

22. The system of claim 19, wherein the computing device is configured for at least one of wired and/or wireless communication with a remote computing device.

23. A system comprising:

an avatar database having one or more avatars stored therein; and

a mobile computing device configured to;

record a video;

detect a face within the video;

track changes in facial characteristics of the detected face; and

reconstruct the recorded video as an avatar-based video using an avatar that is animated using the tracked changes in facial characteristics of the detected face.

24. The system of claim 23, wherein the mobile computing device is further configured to generate an avatar derived from the detected face, and wherein the avatar used in reconstructing the recorded video comprises the avatar generated from the detected face.

25. The system of claim 23, wherein the mobile computing device is further configured to communicate with a network comprising at least one of a second-generation (2G) mobile communication network, a third-generation (3G) mobile communication network, a fourth-generation (4G) mobile communication network, and/or a wireless local area network (WLAN).