APPLYING FACIAL MASKS TO FACES IN LIVE VIDEO

• US 20180075665A1
• Filed: 09/13/2016
• Published: 03/15/2018
• Est. Priority Date: 09/13/2016
• Status: Active Grant

• Current Assignee: Aleksey Konoplev
• Original Assignee: Aleksey Konoplev
• Inventors: Konoplev, Aleksey
• Granted Patent: US 10,062,216 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20028 Bilateral filtering
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 3/606 by memory addressing or mapping
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/169 Holistic features and representations, i.e. based on the facial image taken as a whole
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

A method for applying facial masks to faces in live video. The method includes receiving an image containing a face from a user, wherein the image is a frame of a video and identifying the coordinates of a face in the image. The method also includes identifying the coordinates of facial elements within the face previously identified and synchronizing a bitmap add-on, wherein synchronizing the bitmap add-on includes aligning the bitmap add-on with the identified facial elements. The method further includes applying the bitmap add-ons over the frame of the identified face.

Claims

1. A computer system, the computer system comprising:

one or more hardware processors;

system memory coupled to the one or more hardware processors, the system memory storing instructions that are executable by the one or more hardware processors;

the one or more hardware processors executing the instructions stored in the system memory to apply facial masks to faces in live video, including the following;

receive an image containing a face from a user, wherein the image is a frame of a video;

identify the coordinates of a face in the image;

identify the coordinates of facial elements within the face previously identified using a successive steps method, wherein the successive steps method includes;

placing an approximate grid into the region containing the face;

running an integer chooser at each coordinate and concatenating the outputs;

constructing a vector E;

calculating the (d_x,d_y) shifts for the k-th facial element; and

shifting the X and Y coordinates of f_k by the calculated (d_x,d_y) shifts;

synchronize a bitmap add-on, wherein synchronizing the bitmap add-on includes aligning the bitmap add-on with the identified facial elements; and

applying apply the bitmap add-ons over the frame of the identified face.

2. The system of claim 1, the instructions stored in the system memory further comprising:

applying 3D effects.

3. The system of claim 2, wherein applying 3D effects includes:

rotating a 3D model to match the rotation of the face.

4. The system of claim 2, wherein applying 3D effects includes:

animating a 3D model.

5. The system of claim 1, wherein the bitmap add-on includes a character'"'"'s face.

8. The system of claim 1, the instructions stored in the system memory further comprising:

repeating the successive steps methods.

9. The system of claim 8, wherein repeating the successive steps method includes:

repeating until the value of the (d_x,d_y) shifts falls below a predetermined value.

10. The system of claim 8, wherein repeating the successive steps method includes

creating a succession number and assigning the initial value to zero;

assigning a total succession number;

incrementing the succession number each time the successive steps method is run; and

repeating the successive steps method until the succession number is equal to or greater than the total succession number.

11. The system of claim 1, wherein the approximate grid is a facial grid containing the average coordinates of facial elements as detected in a predetermined number of faces.

12. The system of claim 1, wherein the approximate grid takes into account the position, size and the approximate rotation angle of the predetermined number of faces.

13. The system of claim 1, wherein calculating the (d_x,d_y) shifts for the k-th facial element includes:

taking a dot product of E with D_S,k=(D_S,k^x,D_S,k^y), where D_S,k=(D_S,k^x,D_S,k^y) is the trained data for the k-th facial element.

6. (canceled)

7. (canceled)

14. A computer system, the computer system comprising:

one or more hardware processors;

system memory coupled to the one or more hardware processors, the system memory storing instructions that are executable by the one or more hardware processors;

the one or more hardware processors executing the instructions stored in the system memory to apply facial masks to faces in live video, including the following;

receive an image containing a face from a user, wherein the image is a frame of a video;

identify the coordinates of a face in the image;

identify the coordinates of facial elements within the face previously identified using a successive steps method, wherein the successive steps method includes;

placing an approximate grid into the region containing the face;

running an integer chooser at each coordinate and concatenating the outputs;

constructing a vector E;

calculating the (d_x,d_y) shifts for the k-th facial element; and

shifting the X and Y coordinates of f_k by the calculated (d_x,d_y) shifts;

synchronize a bitmap add-on, wherein synchronizing the bitmap add-on includes aligning the bitmap add-on with the identified facial elements, the method for synchronizing the bitmap add-on including;

smoothing facial element coordinates in the current frame based on previous frames;

warping the face in the image; and

warping the bitmap add-on; and

applying apply the bitmap add-ons over the frame of the identified face.

15. The system of claim 14, wherein smoothing facial element coordinates in the current frame based on previous frames includes applying a smoothing filter.

16. The system of claim 15, wherein the smoothing filter includes:

a temporal Gaussian filter.

17. The system of claim 15, wherein the smoothing filter includes:

a bilateral Gaussian filter.

18. A computer system, the computer system comprising:

one or more hardware processors;

system memory coupled to the one or more hardware processors, the system memory storing instructions that are executable by the one or more hardware processors;

the one or more hardware processors executing the instructions stored in the system memory to apply facial masks to faces in live video, including the following;

receive an image containing a face from a user, wherein the image is a frame of a video;

identify the coordinates of a face in the image;

identify the coordinates of facial elements within the face previously identified using a successive steps method, wherein the successive steps method includes;

placing an approximate grid into the region containing the face;

running an integer chooser at each coordinate and concatenating the outputs;

constructing a vector E;

calculating the (d_x,d_y) shifts for the k-th facial element; and

shifting the X and Y coordinates of f_k by the calculated (d_x,d_y) shifts;

train a detector, wherein training a detector allows for synchronization of a bitmap add-on;

synchronize the bitmap add-on, wherein synchronizing the bitmap add-on includes aligning the bitmap add-on with the identified facial elements, the method for synchronizing the bitmap add-on including;

smoothing facial element coordinates in the current frame based on previous frames;

warping the face in the image; and

warping the bitmap add-on; and

apply the bitmap add-ons over the frame of the identified face.

19. The system of claim 18, wherein training the detector includes:

obtaining a database of photos and their corresponding grids with facial feature markup;

calculating the mean grid by averaging all obtained grids;

constructing a set that contains starting grids for each photo; and

repeating the steps;

calculating optimal displacements;

calculating the choose junctions;

calculating E_jq for each (I_j,G_jq);

finding the solution of the minimization problem that allows to displace k-th facial element of G_jq to make it closer to M_jq; and

shifting each (k-th) facial element of G_jq by (d_x,d_y).

20. The system of claim 19, wherein repeating the steps includes:

performing the steps a predetermined number of times.