Method for real time video processing involving changing a color of an object on a human face in a video

• US 10,255,948 B2
• Filed: 07/13/2016
• Issued: 04/09/2019
• Est. Priority Date: 02/05/2014
• Status: Active Grant

• Current Assignee: Avatar Merger Sub II, LLC
• Original Assignee: Avatar Merger Sub II, LLC
• Inventors: Shaburova, Elena, Monastyrshyn, Yurii
• Primary Examiner(s): Niu, Feng
• Application Number:
  US15/208,973
• Publication Number:
  US 20160322079A1
• Time in Patent Office: 1,000 Days
• Field of Search: None
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/42 Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 10/56 : relating to colour
• CPC Class Codes: G06V 10/60 : relating to illumination properties, e.g. using a reflectance or lighting model
• CPC Class Codes: G06V 20/40 : in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/46 : Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 40/167 : using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/168 : Feature extraction; Face representation
• CPC Class Codes: G06V 40/171 : Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/174 : Facial expression recognition
• CPC Class Codes: G06V 40/20 : Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G11B 27/031 : Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: H04N 5/262 : Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
• CPC Class Codes: H04N 9/79 : Processing of colour television signals in connection with recording

Abstract

A computer-implemented method for real time video processing for changing a color of an object in a video, the method being performed in connection with a computerized system comprising a processing unit and a memory, the method comprising: providing an object in the video that at least partially and at least occasionally is presented in frames of the video; detecting the object in the video, wherein said detection comprises detecting feature reference points of the object; tracking the detected object in the video, wherein the tracking comprises creating a mesh that is based on the detected feature reference points of the object and aligning the mesh to the object in each frame; generating a set of node points on the created mesh based on a request for changing color, the set of node points defining an area the color of which is to be changed; and transforming the frames of the video in such way that the object'"'"'s color is changed within the defined area when the object is presented in frames of the video.

Claims

1. A computer-implemented method, comprising:

identifying, by one or more processors, a face within a set of images comprising a video stream;

identifying, by the one or more processors, a set of pixels for one or more objects depicted on the face;

identifying, by the one or more processors, one or more pixels to be recolored within the set of images, the one or more pixels included within the set of pixels for the one or more objects;

computing, by the one or more processors, a new pixel value, for the one or more pixels to be recolored in the set of images comprising the video stream, as a function of intensity values of a collection of pixels within the set of pixels that is within a given distance of the one or more pixels; and

transforming, by the one or more processors, at least a portion of the set of images of the video stream to generate a modified video stream by applying the new pixel value that has been computed to the one or more pixels to be recolored of each image of the video stream while the one or more objects, corresponding to the one or more pixels, are presented within the set of images of the video stream; and

wherein the set of pixels are within a region having a prespecified shape, and wherein determining that the one or more pixels to be recolored corresponds to one or more pixels of a target object to be recolored in response to determining that an intensity value of the one or more pixels, in the region, to be recolored is less than a fraction of a value of a pixel at a center of the region.

2. The method of claim 1, wherein the video stream comprises a plurality of frames, and wherein identifying the one or more pixels to be recolored comprises computing a probability that a target object to be recolored within the one or more pixels exists after a given frame number of the plurality of frames, and wherein the new pixel value is further computed as a function of the computed probability.

3. The method of claim 2 further comprising:

generating a set of node points on the mesh defining one or more areas containing the one or more pixels to be recolored, the new pixel value applied within the one or more areas.

4. The method of claim 1 further comprising:

tracking the face in the set of images; and

applying new color and new intensity value to the one or more pixels to be recolored of each image of the set of images in which the face is tracked.

5. The method of claim 4 further comprising:

detecting the pixels to be colored within each frame of the set of images in which the face is tracked based on a weighted sum of information from a current image and information from at least one previous image of the set of images.

6. The method of claim 1, wherein new color and new intensity value are calculated based on one or more color and intensity values of the face within the set of images, further comprising changing a color of a first portion of the face and maintaining a color of a second portion of the face.

7. A system, comprising:

one or more processors; and

a processor-readable storage device coupled to the one or more processors, the processor readable storage device storing processor executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

identifying a face within a set of images comprising a video stream;

identifying a set of pixels for one or more objects depicted on the face;

identifying one or more pixels to be recolored within the set of images, the one or more pixels included within the set of pixels for the one or more objects;

computing a new pixel value, for the one or more pixels to be recolored in the set of images comprising the video stream, as a function of intensity values of a collection of pixels within the set of pixels that is within a given distance of the one or more pixels; and

transforming at least a portion of the set of images of the video stream to generate a modified video stream by applying the new pixel value that has been computed to the one or more pixels to be recolored of each image of the video stream while the one or more objects, corresponding to the one or more pixels, are presented within the set of images of the video stream; and

wherein the set of pixels are within a region having a prespecified shape, and wherein determining that the one or more pixels to be recolored corresponds to one or more pixels of a target object to be recolored in response to determining that an intensity value of the one or more pixels, in the region, to be recolored is less than a fraction of a value of a pixel at a center of the region.

8. The system of claim 7, wherein the video stream comprises a plurality of frames, and wherein identifying the one or more pixels to be recolored comprises computing a probability that a target object to be recolored within the one or more pixels exists after a given frame number of the plurality of frames, and wherein the new pixel value is further computed as a function of the computed probability.

9. The system of claim 8, wherein the operations further comprise:

generating a set of node points on the mesh defining one or more areas containing the one or more pixels to be recolored, the new pixel value applied within the one or more areas.

10. The system of claim 7, wherein the operations further comprise:

tracking the face in the set of images; and

applying new color and new intensity value to the one or more pixels to be recolored of each image of the set of images in which the face is tracked.

11. The system of claim 10, wherein the operations further comprise:

detecting the pixels to be colored within each frame of the set of images in which the face is tracked based on a weighted sum of information from a current image and information from at least one previous image of the set of images.

12. The system of claim 7, wherein new color and new intensity value are calculated based on one or more color and intensity values of the face within the set of images, and wherein the operations further comprise changing a color of a first portion of the face and maintaining a color of a second portion of the face.

13. A processor-readable storage device storing processor executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:

identifying a face within a set of images comprising a video stream;

identifying a set of pixels for one or more objects depicted on the face;

identifying one or more pixels to be recolored within the set of images, the one or more pixels included within the set of pixels for the one or more objects;

computing a new pixel value, for the one or more pixels to be recolored in the set of images comprising the video stream, as a function of intensity values of a collection of pixels within the set of pixels that is within a given distance of the one or more pixels; and

transforming at least a portion of the set of images of the video stream to generate a modified video stream by applying the new pixel value that has been computed to the one or more pixels to be recolored of each image of the video stream while the one or more objects, corresponding to the one or more pixels, are presented within the set of images of the video stream; and

wherein the set of pixels are within a region having a prespecified shape, and wherein determining that the one or more pixels to be recolored corresponds to one or more pixels of a target object to be recolored in response to determining that an intensity value of the one or more pixels, in the region, to be recolored is less than a fraction of a value of a pixel at a center of the region.

14. The processor-readable storage device of claim 13, wherein the video stream comprises a plurality of frames, and wherein identifying the one or more pixels to be recolored comprises computing a probability that a target object to be recolored within the one or more pixels exists after a given frame number of the plurality of frames, and wherein the new pixel value is further computed as a function of the computed probability.

15. The processor-readable storage device of claim 13, wherein the operations further comprise:

tracking the face in the set of images; and

applying new color and new intensity value to the one or more pixels to be recolored of each image of the set of images in which the face is tracked.

16. The processor-readable storage device of claim 15, wherein the operations further comprise:

detecting the pixels to be colored within each frame of the set of images in which the face is tracked based on a weighted sum of information from a current image and information from at least one previous image of the set of images.

17. The processor-readable storage device of claim 13, wherein new color and new intensity value are calculated based on one or more color and intensity values of the face within the set of images, and wherein the operations further comprise changing a color of a first portion of the face and maintaining a color of a second portion of the face.