Dynamic facial feature substitution for video conferencing

• US 9,277,180 B2
• Filed: 06/30/2014
• Issued: 03/01/2016
• Est. Priority Date: 06/30/2014
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Cunico, Hernan A., Silva, Asima
• Primary Examiner(s): SMITH, CREIGHTON H
• Application Number:
  US14/320,233
• Publication Number:
  US 20150381938A1
• Time in Patent Office: 610 Days
• Field of Search: 348/14.1, 704/9, 704/246
• US Class Current: 1/1
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06Q 99/00 Subject matter not provided for in other groups of this subclass
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 10/75 Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: G10L 17/06 Decision making techniques; Pattern matching strategies
• CPC Class Codes: H04L 12/18 for broadcast or conference , e.g. multicast
• CPC Class Codes: H04L 12/1827 Network arrangements for conference optimisation or adaptation
• CPC Class Codes: H04L 12/1831 Tracking arrangements for later retrieval, e.g. recording contents, participants activities or behavior, network status
• CPC Class Codes: H04L 65/4038 with floor control
• CPC Class Codes: H04L 65/75 Media network packet handling
• CPC Class Codes: H04N 7/152 Multipoint control units therefor
• CPC Class Codes: H04N 7/155 involving storage of or access to video conference sessions tracking arrangements for later retrieval of a computer conference content or participants activities H04L12/1831
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

In an approach to determine facial feature substitution in a video conference, a computer receives one or more pre-recorded videos of an attendee of a video conference. The one or more pre-recorded videos include at least one pre-recorded video of an attendee speaking. The computer determines an avatar for use in place of a video of the attendee in the video conference. Furthermore, the computer determines one or more portions of the one or more pre-recorded videos of the attendee corresponding to at least one targeted facial feature of the attendee in the video of the attendee in the video conference. The computer substitutes the one or more portions of the one or more pre-recorded videos into the avatar, the substitution corresponding to the at least one targeted facial feature of the attendee in the video conference.

Claims

1. A computer program product for facial feature substitution in a video conference, the computer program product comprising:

one or more computer readable storage devices and program instructions stored on the one or more computer readable storage devices, the program instructions executable by a processor, the program instructions comprising;

program instructions to receive one or more pre-recorded videos of an attendee participating in a video conference, the one or more pre-recorded videos including at least one pre-recorded video of an attendee speaking;

program instructions to determine an avatar for use in place of a live video feed of the attendee in the video conference;

program instructions to determine one or more portions of the one or more pre-recorded videos of the attendee corresponding to at least one targeted facial feature of the attendee in the live video feed of the attendee participating in the video conference; and

program instructions to substitute the one or more portions of the one or more pre-recorded videos into the avatar, the substitution corresponding to the at least one targeted facial feature of the attendee such that the avatar mimics the at least one targeted facial feature of the attendee in the video conference.

2. The computer program product of claim 1, wherein the program instructions to substitute the one or more portions of the one or more pre-recorded videos into the avatar further comprise:

program instructions to create a co-ordinate map of one or more key facial elements of the attendee in the one or more pre-recorded videos;

program instructions to create a co-ordinate map of one or more key facial elements of the attendee in the video of the attendee in the video conference;

program instructions to match the co-ordinate map of the one or more pre-recorded videos to the co-ordinate map of the video of the attendee in the video conference; and

program instructions to substitute, based, at least in part, on the matched co-ordinate maps, the one or more portions of the one or more pre-recorded videos into the avatar.

3. The computer program product of claim 1, wherein the program instructions to determine one or more portions of the one or more pre-recorded videos of the attendee corresponding to at least one targeted facial feature of the attendee in the live video feed of the attendee participating in the video conference further comprise program instructions to use facial recognition algorithms to correlate facial expressions and facial movements in the live video feed to the one or more pre-recorded videos.

4. The computer program product of claim 1, wherein the program instructions to determine one or more portions of the one or more pre-recorded videos of the attendee corresponding to at least one targeted facial feature of the attendee in the live video feed of the attendee participating in the video conference further comprise program instructions to use at least one of natural language processing and speech recognition to match corresponding one or more words in the live video feed of the attendee participating in the video conference to one or more words in the one or more pre-recorded videos.

5. The computer program product of claim 1, wherein the program instructions to determine one or more portions of the one or more pre-recorded videos of the attendee for at least one targeted facial feature of the attendee in the live video feed of the attendee participating in the video conference further comprise program instructions to use sentiment analysis to correlate a sentiment of the live video feed of the attendee participating in the video conference to a sentiment the one or more portions of the one or more pre-recorded videos.

6. The computer program product of claim 1, further comprises:

program instructions to monitor the live video feed of the attendee participating in the video conference;

program instructions to determine whether the at least one targeted facial feature of the attendee changed;

responsive to determining the at least one targeted facial feature of the attendee changed, program instructions to determine at least one additional portion of the one or more pre-recorded videos corresponding to the changed at least one targeted facial feature; and

program instructions to substitute the at least one additional portion of the one or more pre-recorded videos into the avatar.

7. A computer system for facial feature substitution in a video conference, the computer system comprising:

one or more computer processors;

one or more computer readable storage devices;

program instructions stored on the computer readable storage devices for execution by at least one of the one or more processors, the program instructions comprising;

program instructions to receive one or more pre-recorded videos of an attendee participating in a video conference, the one or more pre-recorded videos including at least one pre-recorded video of an attendee speaking;

program instructions to determine an avatar for use in place of a live video feed of the attendee in the video conference;

program instructions to determine one or more portions of the one or more pre-recorded videos of the attendee corresponding to at least one targeted facial feature of the attendee in the live video feed of the attendee participating in the video conference; and

program instructions to substitute the one or more portions of the one or more pre-recorded videos into the avatar, the substitution corresponding to the at least one targeted facial feature of the attendee such that the avatar mimics the at least one targeted facial feature of the attendee in the video conference.

8. The computer system of claim 7, wherein the program instructions to substitute the one or more portions of the one or more pre-recorded videos into the avatar further comprise:

program instructions to create a co-ordinate map of one or more key facial elements of the attendee in the one or more pre-recorded videos;

program instructions to create a co-ordinate map of one or more key facial elements of the attendee in the video of the attendee in the video conference;

program instructions to match the co-ordinate map of the one or more pre-recorded videos to the co-ordinate map of the video of the attendee in the video conference; and

program instructions to substitute, based, at least in part, on the matched co-ordinate maps, the one or more portions of the one or more pre-recorded videos into the avatar.

9. The computer system of claim 7, wherein the program instructions to determine one or more portions of the one or more pre-recorded videos of the attendee corresponding to at least one targeted facial feature of the attendee in the live video feed of the attendee participating in the video conference further comprise program instructions to use facial recognition algorithms to correlate facial expressions and facial movements in the live video feed to the one or more pre-recorded videos.

10. The computer system of claim 7, wherein the program instructions to determine one or more portions of the one or more pre-recorded videos of the attendee corresponding to at least one targeted facial feature of the attendee in the live video feed of the attendee participating in the video conference further comprise program instructions to use at least one of natural language processing and speech recognition to match corresponding one or more words in the live video feed of the attendee participating in the video conference to one or more words in the one or more pre-recorded videos.

11. The computer system of claim 7, wherein the program instructions to determine one or more portions of the one or more pre-recorded videos of the attendee for at least one targeted facial feature of the attendee in the live video feed of the attendee participating in the video conference further comprise program instructions to use sentiment analysis to correlate a sentiment of the live video feed of the attendee participating in the video conference to a sentiment the one or more portions of the one or more pre-recorded videos.

12. The computer system of claim 7, further comprises:

program instructions to monitor the live video feed of the attendee participating in the video conference;

program instructions to determine whether the at least one targeted facial feature of the attendee changed;

responsive to determining the at least one targeted facial feature of the attendee changed, program instructions to determine at least one additional portion of the one or more pre-recorded videos corresponding to the changed at least one targeted facial feature of the attendee; and

program instructions to substitute the at least one additional portion of the one or more pre-recorded videos into the avatar.