Dynamic facial feature substitution for video conferencing

• US 9,332,227 B2
• Filed: 02/16/2015
• Issued: 05/03/2016
• Est. Priority Date: 06/30/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Cunico, Hernan A., Silva, Asima
• Primary Examiner(s): SMITH, CREIGHTON H
• Application Number:
  US14/623,079
• Publication Number:
  US 20150381939A1
• Time in Patent Office: 442 Days
• Field of Search: 348/14.07
• US Class Current: 1/1
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06Q 99/00 Subject matter not provided for in other groups of this subclass
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 10/75 Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: G10L 17/06 Decision making techniques; Pattern matching strategies
• CPC Class Codes: H04L 12/18 for broadcast or conference , e.g. multicast
• CPC Class Codes: H04L 12/1827 Network arrangements for conference optimisation or adaptation
• CPC Class Codes: H04L 12/1831 Tracking arrangements for later retrieval, e.g. recording contents, participants activities or behavior, network status
• CPC Class Codes: H04L 65/4038 with floor control
• CPC Class Codes: H04L 65/75 Media network packet handling
• CPC Class Codes: H04N 7/152 Multipoint control units therefor
• CPC Class Codes: H04N 7/155 involving storage of or access to video conference sessions tracking arrangements for later retrieval of a computer conference content or participants activities H04L12/1831
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

In an approach to determine facial feature substitution in a video conference, a computer receives one or more pre-recorded videos of an attendee of a video conference. The one or more pre-recorded videos include at least one pre-recorded video of an attendee speaking. The computer determines an avatar for use in place of a video of the attendee in the video conference. Furthermore, the computer determines one or more portions of the one or more pre-recorded videos of the attendee corresponding to at least one targeted facial feature of the attendee in the video of the attendee in the video conference. The computer substitutes the one or more portions of the one or more pre-recorded videos into the avatar, the substitution corresponding to the at least one targeted facial feature of the attendee in the video conference.

Claims

1. A method for facial feature substitution in a video conference, the method comprising:

receiving, by one or more computing devices, one or more pre-recorded videos of an attendee of a video conference, the one or more pre-recorded videos including at least one pre-recorded video of an attendee speaking;

determining, by one or more computing devices, an avatar for use in place of a video of the attendee in the video conference;

determining, by one or more computing devices, one or more portions of the one or more pre-recorded videos of the attendee corresponding to at least one targeted facial feature of the attendee in the video of the attendee in the video conference; and

substituting, by one or more computing devices, the one or more portions of the one or more pre-recorded videos into the avatar, the substitution corresponding to the at least one targeted facial feature of the attendee.

2. The method of claim 1, wherein substituting, by one or more computing devices, the one or more portions of the one or more pre-recorded videos into the avatar further comprise:

creating, by one or more computing devices, a co-ordinate map of one or more key facial elements of the attendee in the one or more pre-recorded videos;

creating, by one or more computing devices, a co-ordinate map of one or more key facial elements of the attendee in the video of the attendee in the video conference;

matching, by one or more computing devices, the co-ordinate map of the one or more pre-recorded videos to the co-ordinate map of the video of the attendee in the video conference; and

substituting, by one or more computing devices, based, at least in part, on the matched co-ordinate maps, the one or more portions of the one or more pre-recorded videos into the avatar.

3. The method of claim 1, wherein determining, by one or more computing devices, the one or more portions of the one or more pre-recorded videos corresponding to the at least one targeted facial feature of the attendee in the video of the attendee in the video conference further comprises using one or more facial recognition algorithms to correlate facial expressions and facial movements in the video to the one or more pre-recorded videos.

4. The method of claim 1, wherein determining, by one or more computing devices, the one or more portions of the one or more pre-recorded videos corresponding to the at least one targeted facial feature of the attendee in the video of the attendee in the video conference further comprises using at least one of natural language processing and speech recognition to match corresponding one or more words in the video of the attendee to one or more words in the one or more pre-recorded videos.

5. The method of claim 1, wherein determining, by one or more computing devices, the one or more portions of the one or more pre-recorded videos corresponding to the at least one targeted facial feature of the attendee in the video of the attendee in the video conference further comprises using sentiment analysis to correlate the sentiment of the video of the attendee to sentiment of the one or more portions of the one or more pre-recorded videos.

6. The method of claim 1, wherein determining, by one or more computing devices, the one or more portions of the one or more pre-recorded videos of the attendee corresponding to at least one targeted facial feature of the attendee in the video of the attendee in the video conference further comprises using three dimensional facial recognition algorithms to correlate the video of the attendee with one or more portions of the one or more pre-recorded videos.

7. The method of claim 1, wherein the one or more pre-recorded videos of the attendee further include the attendee exhibiting one or more facial expressions.

8. The method of claim 1, further comprising:

monitoring, by one or more computing devices, the video of the attendee in a video conference;

determining, by one or more computing devices, whether the at least one targeted facial feature of the attendee changed;

responsive to determining, by one or more computing devices, the at least one targeted facial feature of the attendee changed, determining at least one additional portion of the one or more pre-recorded videos of the attendee corresponding to the changed at least one targeted facial feature; and

substituting, by one or more computing devices, the at least one additional portion of the one or more pre-recorded videos into the avatar.