Real time painting of a video stream

• US 10,609,324 B2
• Filed: 07/18/2016
• Issued: 03/31/2020
• Est. Priority Date: 07/18/2016
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Monastyrshyn, Yurii, Tulupov, Illia
• Primary Examiner(s): Bloomquist, Keith D
• Application Number:
  US15/213,186
• Publication Number:
  US 20180018079A1
• Time in Patent Office: 1,352 Days
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera

Abstract

Systems, devices, media, and methods are presented for generating graphical representations within frames of a video stream in real time. The systems and methods receive a frames depicting a portion of a face, identify user input, identify positions on the portion of the face corresponding to the user input. The systems and methods generate a graphical representation of the user input linked to positions on the portion of the face and render the graphical representation within frames of the video stream in real time.

Claims

1. A method, comprising:

receiving, by one or more processors, a set of frames of a video stream, at least a portion of the set of frames depicting at least a portion of a face;

receiving input, during sequential presentation of a plurality of the set of frames of the video stream, depicting at least the portion of the face, the input representing a user selection of a timing scheme comprising a set period of time for presentation of a graphical representation of the input;

identifying one or more positions on the portion of the face corresponding to the input;

linking the input to the one or more positions on the portion of the face;

generating the graphical representation of the input, the graphical representation of the input linked to the one or more positions on the portion of the face; and

in response to detecting an event and based on the timing scheme, rendering the graphical representation on the portion of the face within one or more subsequent frames of the video stream, the graphical representation presented on the portion of the face at the one or more positions.

2. The method of claim 1, wherein receiving the input further comprises:

determining a first relative distance of the portion of the face to an image capture device in a frame of the video stream;

determining a size characteristic of the graphical representation based on the first relative distance of the portion of the face; and

scaling the size characteristic of the graphical representation based on movement of the portion of the face from the first relative distance to a second relative distance in a subsequent frame of the video stream.

3. The method of claim 2, wherein scaling the size characteristic of the graphical representation further comprises:

identifying an first line weight for the graphical representation at the first relative distance;

determining a first relative position of at least one point on the graphical representation and two or more facial landmarks depicted on the portion of the face;

determining a change in distance between the two or more facial landmarks at the second relative distance; and

modifying the first line weight for the graphical representation based on the change in distance between the two or more facial landmarks to generate a second line weight for the graphical representation.

4. The method of claim 1, wherein identifying the one or more positions on the portion of the face corresponding to the user input further comprises:

determining a facial mesh on the portion of the face, the facial mesh identifying one or more facial landmarks depicted within the portion of the face;

identify one or more coordinates of the input; and

mapping the one or more coordinates to at least a portion of the one or more facial landmarks.

5. The method of claim 1 further comprising:

generating a graphics plane positioned in front of the portion of the face;

linking one or more points on the graphics plane to one or more facial landmarks depicted on the portion of the face; and

tracking movement of the one or more facial landmarks and matching the movement of the one or more facial landmarks with movement of the graphics plane.

6. The method of claim 5, wherein the graphical plane is contoured such that a graphical representation linked to coordinates on the graphical plane includes two or more three dimensional positions with respect to the portion of the face.

7. The method of claim 5, wherein the graphics plane is transparent, extends outwardly from the face, and has a set of coordinates distributed about the graphics plane and further comprising:

determining the input extends outwardly from the portion of the face;

determining one or more coordinates of the set of coordinates corresponding to a position of the input;

linking the input to the one or more coordinates on the graphical plane; and

rendering the graphical representation extending outwardly from the face on the graphical plane within one or more subsequent frames of the video stream.

8. The method of claim 7, wherein the graphical plane is a first graphical plane positioned at a first three dimensional position with respect to the portion of the face and the graphical representation is a first graphical representation, and further comprising:

generating a second graphics plane at a second three dimensional position with respect to the portion of the face;

determining a second input positioned within a frame of the video stream;

linking the second input to one or more coordinates on the second graphical plane;

generating a second graphical representation of the second input at the one or more coordinates on the second graphical plane; and

rendering the first graphical representation on the first graphical plane and the second graphical representation on the second graphical plane.

9. The method of claim 1 further comprises tracking the one or more positions on the portion of the face to render the graphical representation as a persistent graphical representation by:

identifying the one or more positions on the portion of the face in a first subsequent frame at a first location within the first subsequent frame;

rendering the graphical representation at the one or more positions on the portion of the face at the first location;

identifying the one or more positions on the portion of the face in a second subsequent frame at a second location within the second subsequent frame; and

rendering the graphical representation at the one or more positions on the portion of the face at the second location.

10. The method of claim 1 further comprising:

storing the graphical representation within a processor-readable storage device;

in response to initiating an application, detecting the portion of the face within a new video stream; and

in response to detecting the portion of the face within the new video stream, rendering the graphical representation within the set of frames of the new video stream.

11. The method of claim 1, wherein a touch screen device configured to be sensitive to a pressure of a touch provides the input and wherein receiving the input further comprises:

determining a touch pressure of the input;

identifying an at least one of an edge sharpness, a line color, or color intensity associated with the touch pressure; and

assigning the at least one of an edge sharpness, a line color, or color intensity to the graphical representation generated from the input.

12. The method of claim 1 further comprising:

retrieving a specified minimum face scale;

determining that a size of the face depicted in at least one of the frames is less than the specified minimum face scale;

overlaying on top of a frame of the video stream a message informing the user to position the face closer to the image capture device until the size of the face corresponds to the specified minimum face scale.

13. The method of claim 1 further comprising:

determining an amount of a frame of the video stream occupied by the portion of the face;

determining a relative distance of the portion of the face to the image capture device based on the amount of the frame occupied by the portion of the face; and

scaling a size of the graphical representation based on the relative distance of the portion of the face.

14. The method of claim 1, wherein the event is selected by the user and comprises a change from a first manner in which a facial landmark appears in first frame of the set of frames to a second manner in which the facial landmark appears in a second frame of the set of frames, the facial landmark being on the portion of the face.

15. The method of claim 1, wherein

rendering the graphical representation comprises causing presentation of the graphical representation according to the timing scheme.

16. A system, comprising:

one or more processors; and

a non-transitory processor-readable storage medium coupled to the one or more processors and storing processor executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

receiving, by one or more processors, a set of frames of a video stream, at least a portion of the set of frames depicting at least a portion of a face;

receiving input, during sequential presentation of a plurality of the set of frames of the video stream, depicting at least the portion of the face, the input representing a user selection of a timing scheme comprising a set period of time for presentation of a graphical representation of the input;

identifying one or more positions on the portion of the face corresponding to the input;

linking the input to the one or more positions on the portion of the face;

generating the graphical representation of the input, the graphical representation of the input linked to the one or more positions on the portion of the face; and

in response to detecting an event and based on the timing scheme, rendering the graphical representation on the portion of the face within one or more subsequent frames of the video stream, the graphical representation presented on the portion of the face at the one or more positions.

17. The system of claim 16, wherein receiving the input further comprises:

determining a first relative distance of the portion of the face to an image capture device in a frame of the video stream;

determining a size characteristic of the graphical representation based on the first relative distance of the portion of the face; and

scaling the size characteristic of the graphical representation based on movement of the portion of the face from the first relative distance to a second relative distance in a subsequent frame of the video stream.

18. The system of claim 17, wherein scaling the size characteristic of the graphical representation further comprises:

identifying an first line weight for the graphical representation at the first relative distance;

determining a first relative position of at least one point on the graphical representation and two or more facial landmarks depicted on the portion of the face;

determining a change in distance between the two or more facial landmarks at the second relative distance; and

modifying the first line weight for the graphical representation based on the change in distance between the two or more facial landmarks to generate a second line weight for the graphical representation.

19. The system of claim 16, wherein identifying the one or more positions on the portion of the face corresponding to the input further comprises:

determining a facial mesh on the portion of the face, the facial mesh identifying one or more facial landmarks depicted within the portion of the face;

identify one or more coordinates of the input; and

mapping the one or more coordinates to at least a portion of the one or more facial landmarks.

20. The system of claim 16, wherein the operations further comprise:

generating a graphics plane positioned in front of the portion of the face;

linking one or more points on the graphics plane to one or more facial landmarks depicted on the portion of the face; and

tracking movement of the one or more facial landmarks and matching the movement of the one or more facial landmarks with movement of the graphics plane.

21. A non-transitory processor-readable storage medium storing processor executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:

receiving, by one or more processors, a set of frames of a video stream, at least a portion of the set of frames depicting at least a portion of a face;

receiving input, during sequential presentation of a plurality of the set of frames of the video stream, depicting at least the portion of the face, the input representing a user selection of a timing scheme comprising a set period of time for presentation of a graphical representation of the input;

identifying one or more positions on the portion of the face corresponding to the input;

linking the input to the one or more positions on the portion of the face;

generating the graphical representation of the input, the graphical representation of the input linked to the one or more positions on the portion of the face; and

in response to detecting an event and based on the timing scheme, rendering the graphical representation on the portion of the face within one or more subsequent frames of the video stream, the graphical representation presented on the portion of the face at the one or more positions.

22. The non-transitory processor-readable storage medium of claim 21, wherein receiving the input further comprises:

determining a first relative distance of the portion of the face to an image capture device in a frame of the video stream;

determining a size characteristic of the graphical representation based on the first relative distance of the portion of the face; and

scaling the size characteristic of the graphical representation based on movement of the portion of the face from the first relative distance to a second relative distance in a subsequent frame of the video stream.

23. The non-transitory processor-readable storage medium of claim 22, wherein scaling the size characteristic of the graphical representation further comprises:

identifying an first line weight for the graphical representation at the first relative distance;

determining a first relative position of at least one point on the graphical representation and two or more facial landmarks depicted on the portion of the face;

determining a change in distance between the two or more facial landmarks at the second relative distance; and

modifying the first line weight for the graphical representation based on the change in distance between the two or more facial landmarks to generate a second line weight for the graphical representation.

24. The non-transitory processor-readable storage medium of claim 21, wherein identifying the one or more positions on the portion of the face corresponding to the input further comprises:

determining a facial mesh on the portion of the face, the facial mesh identifying one or more facial landmarks depicted within the portion of the face;

identify one or more coordinates of the input; and

mapping the one or more coordinates to at least a portion of the one or more facial landmarks.