REAL TIME PAINTING OF A VIDEO STREAM

• US 20180018079A1
• Filed: 07/18/2016
• Published: 01/18/2018
• Est. Priority Date: 07/18/2016
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Monastyrshyn, Yurii, Tulupov, Illia
• Granted Patent: US 10,609,324 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera

Abstract

Systems, devices, media, and methods are presented for generating graphical representations within frames of a video stream in real time. The systems and methods receive a frames depicting a portion of a face, identify user input, identify positions on the portion of the face corresponding to the user input. The systems and methods generate a graphical representation of the user input linked to positions on the portion of the face and render the graphical representation within frames of the video stream in real time.

Claims

1. A method, comprising:

receiving, by one or more processors, one or more frames of a video stream, at least a portion of the one or more frames depicting at least a portion of a face;

identifying a user input on an input device of a computing device;

identifying one or more positions on the portion of the face corresponding to the user input;

linking the user input to the one or more positions on the portion of the face;

generating a graphical representation of the user input, the graphical representation of the user input linked to the one or more positions on the portion of the face; and

rendering the graphical representation on the portion of the face within one or more subsequent frames of the video stream, the graphical representation presented on the portion of the face at the one or more positions.

2. The method of claim 1, wherein identifying the user input further comprises:

determining a first relative distance of the portion of the face to an image capture device of the computing device in a frame of the video stream;

determining a size characteristic of the graphical representation based on the first relative distance of the portion of the face; and

scaling the size characteristic of the graphical representation based on movement of the portion of the face from the first relative distance to a second relative distance in a subsequent frame of the video stream.

3. The method of claim 2, wherein scaling the size characteristic of the graphical representation further comprises:

identifying an first line weight for the graphical representation at the first relative distance;

determining a first relative position of at least one point on the graphical representation and two or more facial landmarks depicted on the portion of the face;

determining a change in distance between the two or more facial landmarks at the second relative distance; and

modifying the first line weight for the graphical representation based on the change in distance between the two or more facial landmarks to generate a second line weight for the graphical representation.

4. The method of claim 1, wherein identifying the one or more positions on the portion of the face corresponding to the user input further comprises:

determining a facial mesh on the portion of the face, the facial mesh identifying one or more facial landmarks depicted within the portion of the face;

identify one or more coordinates of the user input; and

mapping the one or more coordinates to at least a portion of the one or more facial landmarks.

5. The method of claim 1 further comprising:

generating a graphics plane positioned in front of the portion of the face;

linking one or more points on the graphics plane to one or more facial landmarks depicted on the portion of the face; and

tracking movement of the one or more facial landmarks and matching the movement of the one or more facial landmarks with movement of the graphics plane.

6. The method of claim 5, wherein the graphical plane is contoured such that a graphical representation linked to coordinates on the graphical plane includes two or more three dimensional positions with respect to the portion of the face.

7. The method of claim 5, wherein the graphics plane is transparent, extends outwardly from the face, and has a set of coordinates distributed about the graphics plane and further comprising:

determining the user input extends outwardly from the portion of the face;

determining one or more coordinates of the set of coordinates corresponding to a position of the user input;

linking the user input to the one or more coordinates on the graphical plane; and

rendering the graphical representation extending outwardly from the face on the graphical plane within one or more subsequent frames of the video stream.

8. The method of claim 7, wherein the graphical plane is a first graphical plane positioned at a first three dimensional position with respect to the portion of the face and the graphical representation is a first graphical representation, and further comprising:

generating a second graphics plane at a second three dimensional position with respect to the portion of the face;

determining a second user input positioned within a frame of the video stream;

linking the second user input to one or more coordinates on the second graphical plane;

generating a second graphical representation of the second user input at the one or more coordinates on the second graphical plane; and

rendering the first graphical representation on the first graphical plane and the second graphical representation on the second graphical plane.

9. The method of claim 1 further comprises tracking the one or more positions on the portion of the face to render the graphical representation as a persistent graphical representation by:

identifying the one or more positions on the portion of the face in a first subsequent frame at a first location within the first subsequent frame;

rendering the graphical representation at the one or more positions on the portion of the face at the first location;

identifying the one or more positions on the portion of the face in a second subsequent frame at a second location within the second subsequent frame; and

rendering the graphical representation at the one or more positions on the portion of the face at the second location.

10. The method of claim 1 further comprising:

storing the graphical representation within a processor-readable storage device;

in response to initiating an application, detecting the portion of the face within a new video stream; and

in response to detecting the portion of the face within the new video stream, rendering the graphical representation within one or more frames of the new video stream.

11. The method of claim 1, wherein the input device is a touch screen device configured to be sensitive to a pressure of a touch providing the user input and wherein identifying the user input further comprises:

determining a touch pressure of the user input;

identifying an edge characteristic associated with the touch pressure; and

assigning the edge characteristic to the graphical representation generated from the user input.

12. A system, comprising:

one or more processors; and

a non-transitory processor-readable storage medium coupled to the one or more processors and storing processor executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

receiving, by one or more processors, one or more frames of a video stream, at least a portion of the one or more frames depicting at least a portion of a face;

identifying a user input on an input device of a computing device;

identifying one or more positions on the portion of the face corresponding to the user input;

linking the user input to the one or more positions on the portion of the face;

generating a graphical representation of the user input, the graphical representation of the user input linked to the one or more positions on the portion of the face; and

rendering the graphical representation on the portion of the face within one or more subsequent frames of the video stream, the graphical representation presented on the portion of the face at the one or more positions.

13. The system of claim 12, wherein identifying the user input further comprises:

determining a first relative distance of the portion of the face to an image capture device of the computing device in a frame of the video stream;

determining a size characteristic of the graphical representation based on the first relative distance of the portion of the face; and

scaling the size characteristic of the graphical representation based on movement of the portion of the face from the first relative distance to a second relative distance in a subsequent frame of the video stream.

14. The system of claim 13, wherein scaling the size characteristic of the graphical representation further comprises:

identifying an first line weight for the graphical representation at the first relative distance;

determining a first relative position of at least one point on the graphical representation and two or more facial landmarks depicted on the portion of the face;

determining a change in distance between the two or more facial landmarks at the second relative distance; and

modifying the first line weight for the graphical representation based on the change in distance between the two or more facial landmarks to generate a second line weight for the graphical representation.

15. The system of claim 12, wherein identifying the one or more positions on the portion of the face corresponding to the user input further comprises:

determining a facial mesh on the portion of the face, the facial mesh identifying one or more facial landmarks depicted within the portion of the face;

identify one or more coordinates of the user input; and

mapping the one or more coordinates to at least a portion of the one or more facial landmarks.

16. The system of claim 12, wherein the operations further comprise:

generating a graphics plane positioned in front of the portion of the face;

linking one or more points on the graphics plane to one or more facial landmarks depicted on the portion of the face; and

tracking movement of the one or more facial landmarks and matching the movement of the one or more facial landmarks with movement of the graphics plane.

17. A non-transitory processor-readable storage medium storing processor executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:

receiving, by one or more processors, one or more frames of a video stream, at least a portion of the one or more frames depicting at least a portion of a face;

identifying a user input on an input device of a computing device;

identifying one or more positions on the portion of the face corresponding to the user input;

linking the user input to the one or more positions on the portion of the face;

generating a graphical representation of the user input, the graphical representation of the user input linked to the one or more positions on the portion of the face; and

rendering the graphical representation on the portion of the face within one or more subsequent frames of the video stream, the graphical representation presented on the portion of the face at the one or more positions.

18. The non-transitory processor-readable storage medium of claim 17, wherein identifying the user input further comprises:

determining a first relative distance of the portion of the face to an image capture device of the computing device in a frame of the video stream;

determining a size characteristic of the graphical representation based on the first relative distance of the portion of the face; and

scaling the size characteristic of the graphical representation based on movement of the portion of the face from the first relative distance to a second relative distance in a subsequent frame of the video stream.

19. The non-transitory processor-readable storage medium of claim 18, wherein scaling the size characteristic of the graphical representation further comprises:

identifying an first line weight for the graphical representation at the first relative distance;

determining a first relative position of at least one point on the graphical representation and two or more facial landmarks depicted on the portion of the face;

determining a change in distance between the two or more facial landmarks at the second relative distance; and

modifying the first line weight for the graphical representation based on the change in distance between the two or more facial landmarks to generate a second line weight for the graphical representation.

20. The non-transitory processor-readable storage medium of claim 17, wherein identifying the one or more positions on the portion of the face corresponding to the user input further comprises:

determining a facial mesh on the portion of the face, the facial mesh identifying one or more facial landmarks depicted within the portion of the face;

identify one or more coordinates of the user input; and

mapping the one or more coordinates to at least a portion of the one or more facial landmarks.