Modifying Video Call Data

• US 20160127681A1
• Filed: 01/13/2015
• Published: 05/05/2016
• Est. Priority Date: 10/31/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Turbell, Henrik Valdemar
• Granted Patent: US 9,531,994 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 17/205 Re-meshing
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: H04N 1/0035 User-machine interface; Control console input or output arrangements for computers G06F3/00
• CPC Class Codes: H04N 1/00352 Input means H04N1/00411 takes precedence; input arrangements for computers G06F3/00
• CPC Class Codes: H04N 21/47205 for manipulating displayed content, e.g. interacting with MPEG-4 objects, editing locally
• CPC Class Codes: H04N 21/4788 communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 7/141 between two video terminals, e.g. videophone telephonic communication systems combined with television receiver for reception of entertainment or information matter H04M11/085
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

A method comprising: displaying a UI for display of received video; detecting selection of a UI displayed button whilst a received video frame is displayed; in response, disabling the display of video frames received after the received video frame; determining a position of a face of a user in the received frame; receiving a plurality of drawing inputs whilst the button is selected, each drawing input defining image data to be applied at a position on said face; modifying the video frame in accordance with the drawing inputs by applying the image data to each of the positions; detecting a condition and in response, for each video frame received after the detection, determining a position of the face in the frame to determine the location of the positions in the frame, applying the image data to each of the positions, and displaying the modified video frame in the UI.

Claims

1. A method implemented at a user terminal during a video call conducted with at least one further user terminal over a communications network, the method comprising:

displaying a user interface on a display of the user terminal for display of received video frames;

detecting selection of a selectable button displayed in said user interface by a user using an input device of said user terminal whilst a received video frame is displayed in the user interface;

in response to said detection, disabling the display of video frames received after said received video frame;

determining a position of a face of a user in the received video frame by executing a face tracker algorithm on a processor of said user terminal;

receiving a plurality of drawing inputs whilst the selectable button is in a selected state, each drawing input defining image data to be applied at a facial position on said face;

modifying the displayed video frame in accordance with the plurality of received drawing inputs by applying the image data to each of said facial positions on the face; and

detecting a condition and in response to detecting said condition, for each video frame received after detecting said condition, the method further comprises;

determining a position of the face in the video frame by executing the face tracker algorithm to determine the location of the facial positions in the video frame, modifying the video frame by applying the image data to each of said positions on the face, and displaying the modified video frame in the user interface.

2. The method of claim 1, wherein detecting the condition comprises detecting a further selection of the selectable button displayed in said user interface.

3. The method of claim 1, wherein detecting the condition comprises detecting that a predetermined period of time has elapsed since selection of the selectable button displayed in said user interface.

4. The method of claim 1, wherein determining a position of a face of a user in the received video frame comprises identifying facial feature points of said face in two-dimensions and fitting a three-dimensional triangulated mesh to the tracked facial feature points, wherein each drawing input defines image data to be applied at a position on said three-dimensional triangulated mesh.

5. The method of claim 4, wherein the modifying the displayed video frame comprises mapping textures of each triangle of the three-dimensional triangulated mesh onto a two-dimensional representation of the three-dimensional triangulated mesh;

and for each received drawing input;

determining a position in the two-dimensional representation of the three-dimensional triangulated mesh that corresponds to the position on said three-dimensional triangulated mesh;

applying the image data to the determined position in the two-dimensional representation of the three-dimensional triangulated mesh; and

mapping textures of each triangle of the two dimensional representation of the three-dimensional triangulated mesh onto the three-dimensional triangulated mesh.

6. The method of claim 4, wherein the modifying the displayed video frame comprises mapping textures of each triangle of the three-dimensional triangulated mesh onto a first two-dimensional representation of the three-dimensional triangulated mesh;

and for each received drawing input;

determining a position in a second two-dimensional representation of the three-dimensional triangulated mesh that corresponds to the position on said three-dimensional triangulated mesh;

applying the image data to the determined position in the second two-dimensional representation of the three-dimensional triangulated mesh;

blending the second two-dimensional representation of the three-dimensional triangulated mesh onto the first two-dimensional representation of the three-dimensional triangulated mesh to create a blended two-dimensional representation of the three-dimensional triangulated mesh; and

mapping textures of each triangle of the blended two-dimensional representation of the three-dimensional triangulated mesh onto the three-dimensional triangulated mesh.

7. The method of claim 1, wherein determining a position of a face of a user in each video frame received after detecting said condition comprises identifying facial feature points of said face in two-dimensions and fitting a three-dimensional triangulated mesh to the tracked facial feature points, wherein each drawing input defines image data to be applied at a position on said three-dimensional triangulated mesh.

8. The method of claim 7, wherein the modifying each video frame received after detecting said condition comprises mapping textures of each triangle of the three-dimensional triangulated mesh onto a two-dimensional representation of the three-dimensional triangulated mesh;

and for each received drawing input;

determining a position in the two-dimensional representation of the three-dimensional triangulated mesh that corresponds to the position on said three-dimensional triangulated mesh;

applying the image data to the determined position in the two-dimensional representation of the three-dimensional triangulated mesh; and

mapping textures of each triangle of the two dimensional representation of the three-dimensional triangulated mesh onto the three-dimensional triangulated mesh.

9. The method of claim 7, wherein the modifying each video frame received after detecting said condition comprises mapping textures of each triangle of the three-dimensional triangulated mesh onto a first two-dimensional representation of the three-dimensional triangulated mesh;

and for each received drawing input;

determining a position in a second two-dimensional representation of the three-dimensional triangulated mesh that corresponds to the position on said three-dimensional triangulated mesh;

applying the image data to the determined position in the second two-dimensional representation of the three-dimensional triangulated mesh;

blending the second two-dimensional representation of the three-dimensional triangulated mesh onto the first two-dimensional representation of the three-dimensional triangulated mesh to create a blended two-dimensional representation of the three-dimensional triangulated mesh; and

mapping textures of each triangle of the blended two-dimensional representation of the three-dimensional triangulated mesh onto the three-dimensional triangulated mesh.

10. The method of claim 1, wherein the method further comprises displaying at least one selectable drawing tool button in the user interface.

11. The method of claim 10, wherein the at least one selectable drawing tool button comprises one or any combination of:

a pen drawing tool button, a lipstick drawing tool button, and a brush drawing tool button.

12. The method of claim 10, wherein the method further comprises in response to detecting selection of one of the at least one selectable drawing tool buttons by a user using an input device of said user terminal, displaying a plurality of selectable colours associated with the selected drawing tool button

13. The method of claim 10, wherein the image data is based on the selected drawing tool button and a selected colour of the plurality of selectable colours associated with the selected drawing tool button.

14. The method of claim 1, wherein video frames are received from a camera of said user terminal.

15. The method of claim 14, the method further comprising transmitting each modified video frame over the communications network to the at least one further user terminal.

16. The method of claim 14, further comprising preventing video frames received whilst display of video frames is disabled from being transmitted over the communications network to the at least one further user terminal.

17. The method of claim 14, further comprising transmitting video frames that are received whilst display of video frames is disabled over the communications network to the at least one further user terminal.

18. The method of claim 14, further comprising:

as each drawing input is received, transmitting an indication of the drawing input to the at least one further user terminal;

or

transmitting an indication of the plurality of drawing inputs in response to detecting said condition.

19. A user terminal comprising:

a display;

an input device;

a network interface configured to transmit and receive video data between the user terminal and a communication network during a video call between the user terminal and at least one further user terminal;

a processor configured to run an application operable during said video call to;

display a user interface on the display for display of received video frames;

detect selection of a selectable button displayed in said user interface by a user using said input device whilst a received video frame is displayed in the user interface;

in response to said detection, disable the display of video frames received after said received video frame;

determine a position of a face of a user in the received video frame by executing a face tracker algorithm;

receive a plurality of drawing inputs whilst the selectable button is in a selected state, each drawing input defining image data to be applied at a facial position on said face;

modify the displayed video frame in accordance with the plurality of received drawing inputs by applying the image data to each of said facial positions on the face; and

detect a condition and in response to said condition detection, for each video frame received after said condition detection, determine a position of the face in the video frame by executing the face tracker algorithm to determine the location of the facial positions in the video frame, modify the video frame by applying the image data to each of said positions on the face, and display the modified video frame in the user interface.

20. A computer program product, the computer program product being embodied on a computer-readable medium and configured so as when executed on a processor of a user terminal during a video call between the user terminal and at least one further user terminal, to:

display a user interface on a display of the user terminal for display of received video frames;

detect selection of a selectable button displayed in said user interface by a user using an input device of said user terminal whilst a received video frame is displayed in the user interface;

in response to said detection, disable the display of video frames received after said received video frame;

determine a position of a face of a user in the received video frame by executing a face tracker algorithm on a processor of said user terminal;

receive a plurality of drawing inputs whilst the selectable button is in a selected state, each drawing input defining image data to be applied at a facial position on said face;

modify the displayed video frame in accordance with the plurality of received drawing inputs by applying the image data to each of said facial positions on the face; and

detect a condition and in response to said condition detection, for each video frame received after said condition detection, determine a position of the face in the video frame by executing the face tracker algorithm to determine the location of the facial positions in the video frame, modify the video frame by applying the image data to each of said positions on the face, and display the modified video frame in the user interface.