Modifying multiple objects within a video stream

• US 10,445,938 B1
• Filed: 11/02/2017
• Issued: 10/15/2019
• Est. Priority Date: 11/09/2016
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Poliakov, Fedir, Pyshchenko, Oleksandr
• Primary Examiner(s): Liu, Gordon G
• Application Number:
  US15/801,814
• Time in Patent Office: 712 Days
• Field of Search: 345633
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

Systems, devices, media, and methods are presented for presentation of modified objects within a video stream. The systems and methods receive a set of images within a video stream and identify at least a portion of a face in a first subset of images. The systems and methods determine face characteristics by analyzing the portion of the face in the first subset of images. The systems and methods apply a graphical representation of glasses to the face based on the face characteristics and cause presentation of a modified video stream including the portion of the face with the graphical representation of the glasses in a second subset of images of the set of images while receiving the video stream.

Claims

1. A method comprising:

receiving a set of images within a video stream;

identifying, by one or more processors of a machine, at least a portion of a face in a first subset of images of the set of images;

determining, by the one or more processors, face characteristics h analyzing the portion of the face in the first subset of images;

identifying, by the one or more processors, an obstruction on the portion of the face in the first subset of images;

applying, by the one or more processors, a graphical representation of glasses to the face based on the face characteristics to obscure the identified obstruction, the graphical representation of the glasses applied in a second subset of images of the set of images occurring within the video stream as the video stream is being received;

detecting a change in the portion of the face between a first image in the first subset of the images and a second image in the first subset of the images;

in response to detecting the change in the portion of the face, modifying a visual attribute of the graphical representation of the glasses; and

causing, by the one or more processors, presentation of a modified video stream including the portion of the face with the graphical representation of the glasses in the second subset of images of the set of images while receiving the video stream.

2. The method of claim 1, further comprising:

in response to identifying the portion of the face in an image of the first subset of images, tracking, by the one or more processors, the portion of the face between one or more subsequent images of the first subset of images; and

in response to applying the graphical representation of the glasses, tracking, by the one or more processors, the portion of the face with the graphical representation of the glasses between one or more images of the second subset of images.

3. The method of claim 2, further comprising:

in response to tracking the portion of the face with the graphical representation of the glasses, detecting a position change between a first image and a second image of the second subset of images;

determining one or more visual effects for the graphical representation of the glasses based on the position change;

generating a modified graphical representation of the glasses based on the one or more visual effects and an initial characteristic of the graphical representation of the glasses; and

causing presentation of the modified video stream including the portion of the face with the modified graphical representation of the glasses in the second subset of images of the set of images while receiving the video stream.

4. The method of claim 1, wherein the identified obstruction comprises real-world glasses depicted in the first subset of images, further comprising editing out the real-world glasses from the second subset of images to remove the identified obstruction prior to applying the graphical representation of the glasses.

5. The method of claim 1, wherein determining the face characteristics further comprises:

generating a set of two-dimensional coordinates for the portion of the face depicted within the first subset of images of the video stream;

determining a face scale for the portion of the face, based at least in part on the set of two-dimensional coordinates;

determining, by the one or more processors, one or more normalization values for the set of two-dimensional coordinates and the face scale, the one or more normalization values determined based on an expected facial range; and

generating a set of normalized two-dimensional coordinates and a normalized face scale.

6. The method of claim 5, wherein generating the set of two-dimensional coordinates further comprises:

determining, by the one or more processors, one or more relative distances between facial features of a set of facial features depicted on the portion of the face depicted within the first subset of images of the video stream; and

generating, by the one or more processors, an estimated face size for the portion of the face depicted within the first subset of images of the video stream.

7. The method of claim 6, wherein the expected facial range comprises a first limit and a second limit, the first limit representing an upper portion of the expected facial range and the second limit representing a lower portion of the expected facial range, the method further comprising:

determining that the set of two-dimensional coordinates and the estimated face size correspond to a value less than the second limit of the facial range;

generating a set of correction values for the set of two-dimensional coordinates and the estimated face size; and

generating a set of corrected two-dimensional coordinates and a corrected face size based on the set of two-dimensional coordinates, the estimated face size, and the set of correction values.

8. The method of claim 5, wherein applying the graphical representation of the glasses further comprises:

identifying one or more first attachment points within the set of two-dimensional coordinates;

identifying one or more second attachment points within the graphical representation glasses; and

positioning at least one second attachment point of the graphical representation of the glasses proximate to at least one first attachment point of the set of two-dimensional coordinates.

9. The method of claim 1, wherein applying the graphical representation of the glasses to the face further comprises:

identifying one or more dimensions of the graphical representation of the glasses, the graphical representation of the glasses having a first size comprising the one or more dimensions; and

based on the face characteristics, modifying the one or more dimensions to scale the graphical representation of the glasses to fit the portion of the face, the modifying of the one or more dimensions maintaining a set of proportions of the graphical representation of the glasses.

10. The method of claim 1, further comprising:

receiving one or more selections corresponding to a set of glasses characteristics of the graphical representation of the glasses;

identifying one or more glasses characteristics from the set of glasses characteristics indicated by the one or more selections; and

rendering the graphical representation of the glasses depicting at least a portion of the one more glasses characteristics.

11. The method of claim 1, wherein the graphical representation of the glasses is a first graphical representation of first glasses, further comprising:

detecting that the face in the first subset of images is a first face of a plurality of faces depicted in the first subset of images;

applying a second graphical representation of second glasses to a second face of the plurality of faces depicted in the first subset of image, the second graphical representation applied in the second subset of images; and

simultaneously presenting in the modified video stream the portion of the first face with the first graphical representation and the second face with the second graphical representation.

12. The method of claim 1, wherein the portion of the face is a first face, the method further comprising:

identifying a set of faces within the first subset of images of the set of images, the first face being one of the set of faces;

determining a set of face characteristics for each face of the set of faces, each face having a distinct set of face characteristics;

applying a set of graphical representations of glasses to the set of faces, each graphical representation of glasses corresponding to a single face to which the graphical representation of glasses is applied; and

causing presentation of the modified video steam including the set of faces with the set of graphical representations of glasses in the second subset of images while receiving the video stream.

13. The method of claim 1 further comprising depicting a reflection of a real-world visual aspect depicted in the video stream on lenses of the graphical representation of the glasses.

14. The method of claim 1 further comprising:

detecting a first change in a first face characteristic between the first image in the first subset of the images and the second image in the first subset of the images;

detecting a second change in a second face characteristic between the first image in the first subset, of the images and the second image in the first subset of the images;

in response to detecting the first change, modifying a first visual attribute of the graphical representation of the glasses; and

in response to detecting the second change, modifying a second visual attribute, of graphical representation of the glasses.

15. The method of claim 14, wherein the first change corresponds to an eye blink and causes the first visual attribute comprising a color of the glasses to be modified, and wherein the second change corresponds to movement of an eyebrow or mouth position and causes the second visual attribute comprising a style of the glasses to be modified.

16. A device comprising:

one or more processors; and

a non-transitory processor-readable storage medium coupled to the one or more processors, the non-transitory processor-readable storage medium storing processor-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

receiving a set of images within a video stream;

identifying, by the or more processors, at least a portion of a face in a first subset of images of the set of images;

determining, by the one or more processors, face characteristics by analyzing the portion of the face in the first subset of images;

identifying, by the one or more processors, an obstruction on the portion of the face in the first subset of images;

applying, by the one or more processors, a graphical representation of glasses to the face based on the face characteristics to obscure the identified obstruction, the graphical representation of the glasses applied in a second subset of images of the set of images occurring within the video stream as the video stream is being received;

detecting a change in the portion of the face between a first image in the first subset of the images and a second image in the first subset of the images;

in response to detecting the change in the portion of the face, modifying a visual attribute of the graphical representation of the glasses; and

causing presentation of a modified video stream including the portion of the face with the graphical representation of the glasses in the second subset of images of the set of images while receiving the video stream.

17. A non-transitory processor-readable storage medium storing processor-executable instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

receiving a set of images within a video stream;

identifying, by the one or more processors, at least a portion of a face in a first subset of images of the set of images;

determining, by the one or more processors, face characteristics by analyzing the portion of the face in the first subset of images;

identifying, by the one or more processors, an obstruction on the portion of the face in the first subset of images;

applying, by the one or more processors, a graphical representation of glasses to the face based on the face characteristics to obscure the identified obstruction, the graphical representation of the glasses applied in a second subset of images of the set of images occurring within the video stream as the video stream is being received;

detecting a change in the portion of the face between a first image in the first subset of images and a second image in the first subset of the images;

in response to detecting the change in the portion of the face, modifying a visual attribute of the graphical representation of the glasses; and

causing presentation of a modified video stream including the portion of the face with the graphical representation of the glasses in the second subset of image s of the set of images while receiving the video stream.

18. The non-transitory processor-readable storage medium of claim 17, wherein determining the face characteristics further comprises:

generating a set of two-dimensional coordinates for the portion of the face depicted within the first subset of images of the video stream;

determining a face scale for the portion of the face based at least in part n the set of two-dimensional coordinates;

determining, by the one or more processors, one or more normalization values for the set of two-dimensional coordinates and the face scale, the one or more normalization values determined based on an expected facial range; and

generating a set of normalized two-dimensional coordinates and a normalized face scale.

19. The non-transitory processor-readable storage medium of claim 17, wherein applying the graphical representation of the glasses to the face further comprises:

identifying one or more dimensions of the graphical representation of the glasses, the graphical representation of the glasses having a first size comprising the one or more dimensions; and

based on the face characteristics, modifying the one or more dimensions to scale the graphical representation of the glasses to fit the portion of the face, the modifying of the one or more dimensions maintaining a set of proportions of the graphical representation of the glasses.