Method for real time video processing involving changing a color of an object on a human face in a video

• US 9,396,525 B2
• Filed: 07/08/2014
• Issued: 07/19/2016
• Est. Priority Date: 02/05/2014
• Status: Active Grant

• Current Assignee: Avatar Merger Sub II, LLC
• Original Assignee: Avatar Merger Sub II, LLC
• Inventors: Shaburova, Elena, Monastyrshin, Yuri
• Primary Examiner(s): Niu, Feng
• Application Number:
  US14/325,477
• Publication Number:
  US 20150221069A1
• Time in Patent Office: 742 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/42 Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 10/56 : relating to colour
• CPC Class Codes: G06V 10/60 : relating to illumination properties, e.g. using a reflectance or lighting model
• CPC Class Codes: G06V 20/40 : in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/46 : Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 40/167 : using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/168 : Feature extraction; Face representation
• CPC Class Codes: G06V 40/171 : Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/174 : Facial expression recognition
• CPC Class Codes: G06V 40/20 : Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G11B 27/031 : Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: H04N 5/262 : Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
• CPC Class Codes: H04N 9/79 : Processing of colour television signals in connection with recording

Abstract

A computer-implemented method for real time video processing for changing a color of an object in a video is presented. The method is performed in connection with a computerized system comprising a processing unit and a memory. The method comprises providing an object in the video that at least partially and at least occasionally is presented in frames of the video; detecting the object in the video; tracking the detected object in the video; generating a set of node points on the created mesh based on a request for changing color, the set of node points defining an area the color of which is to be changed; and transforming the frames of the video in such way that the object'"'"'s color is changed within the defined area when the object is presented in frames of the video.

Claims

1. A computer-implemented method of real time processing of a video for changing color of an object on a face in the video, the method being performed in connection with a computerized system comprising a processing unit and a memory, the method comprising:

forming a mesh of the face in the video based on distinguishable points present in most frames of the video;

aligning the mesh to the face image in each frame;

forming a binary mask providing the pixels to be recolored based on a weighted sum of information from the current frame and information from at least one previous frame;

aligning the binary mask to the mesh on each frame; and

applying a new colour and a new intensity value to the pixels of each frame that are to be recolored.

2. The computer-implemented method of claim 1, wherein the new colour is applied without full intensity so that the colour is partly opaque.

3. The computer-implemented method of claim 1, wherein the new colour and intensity value for the pixels of each frame that are to be recolored are calculated using color and intensity values of a neighbor pixel.

4. The computer-implemented method of claim 1, wherein aligning the binary mask to the mesh on each frame comprises:

making a projection of a mesh to a regular grid to separate the mesh into 100×

100 cells by the regular grid;

determining the mesh element to which a cell of the grid corresponds to, for each cell; and

determining the pixel corresponding to each of the determined mesh elements.

5. The computer-implemented method of claim 4, wherein determining the pixel corresponding to each of the determined mesh elements is performed using a breadth-first-search.

6. The computer-implemented method of claim 4, wherein making a projection of a mesh to the grid is performed once and steps of determining the mesh element and determining the pixel corresponding to each of the determined mesh elements are performed for each frame.

7. The computer-implemented method of claim 1, wherein forming a binary mask providing the pixels to be recolored comprises:

dividing the mesh into at least 1600 cells by a grid;

scanning each frame by a square pixel region, wherein the size of the square pixel region in pixels is determined from the width of an eye in pixels;

determining the mean gray-value intensity of each scanned square of the frame;

detecting pixels to be recolored by comparing the gray-value intensity of each pixel with the mean value of the square it belongs to and by marking the pixels that have the intensity at least 0.92 times of the mean intensity value as pixels to be recolored;

removing a region from the mesh around the pixels to be recolored, wherein the region is a circle having a center on a pixel marked as a pixel to be recolored and a radius as a linear function of the eye width;

applying binary morphological closing to the plurality of marked pixels, wherein the binary morphological closing comprises morphological dilation and morphological erosion applied with a radius determined from the width of eye in pixels; and

removing small and large regions of the pixels to be recolored from the mesh, wherein the regions with an area less than R²*0.2 are detected as small and the regions with an area more than R²*8 are detected as large, wherein R is the radius of binary morphological operations and each region is a four-connected component.

8. The computer-implemented method of claim 7, wherein removing small and large regions of the pixels to be recolored is performed using a breadth-first-search.

9. A mobile computerized system comprising a processing unit and a memory, the memory storing instructions for:

forming a mesh of the face in a video based on distinguishable points present in most frames of the video;

aligning the mesh to the face image in each frame;

forming a binary mask providing the pixels to be recolored based on a weighted sum of information from the current frame and information from at least one previous frame;

aligning the binary mask to the mesh on each frame; and

applying a new colour and a new intensity value to the pixels of each frame that are to be recolored.

10. The mobile computerized system of claim 9, wherein the new colour is applied without full intensity so that the colour is partly opaque.

11. The mobile computerized system of claim 9, wherein the new colour and intensity value for the pixels of each frame that are to be recolored are calculated using color and intensity values of the neighbor pixel.

12. The mobile computerized system of claim 9, wherein forming a mask for each frame by aligning the mask to the mesh on each frame comprises the following steps:

making a projection of a mesh to a regular grid to separate the mesh into 100×

100 cells by the regular grid;

determining the mesh element to which a cell of the grid corresponds to, for each cell; and

determining the pixel corresponding to each of the determined mesh elements.

13. The mobile computerized system of claim 12, wherein determining the pixel corresponding to each of the determined mesh elements s performed using a breadth-first-search.

14. The mobile computerized system of claim 12, wherein making a projection of a mesh to the grid is performed once and steps of determining the mesh element and determining the pixel corresponding to each of the determined mesh elements are performed for each frame.

15. The mobile computerized system of claim 9, wherein forming a binary mask providing the pixels to be recolored comprises:

dividing the mesh into at least 1600 cells by a grid;

scanning each of the frames by a square pixel region, wherein the size of the square pixel region in pixels is determined from the width of an eye in pixels;

determining the mean gray-value intensity of each scanned square of the frame;

detecting pixels to be recolored by comparing the gray-value intensity of each pixel with the mean value of the square it belongs to and by marking the pixels that have the intensity at least 0.92 times of the mean intensity value as pixels to be recolored;

removing a region from the mesh around the pixels to be recolored, wherein the region is a circle having a center on a pixel marked as a pixel to be recolored and a radius as a linear function of the eye width;

applying a binary morphological closing to the plurality of marked pixels, wherein the binary morphological closing comprises morphological dilation and morphological erosion applied with a radius determined from the width of eye in pixels;

removing small and large regions of the pixels to be recolored from the mesh, wherein the regions with an area less than R²*0.2 are detected as small and the regions with an area more than R²*8 are detected as large, wherein R is radius of binary morphological operations and each region is a four-connected component.

16. The mobile computerized system of claim 15, wherein removing small and large regions of the pixels to be recolored is performed using a breadth-first-search.

17. A device capable of video processing comprising a processing unit and a memory, the memory storing instructions for:

forming a mesh of the face in a video based on distinguishable points present in most frames of the video;

aligning the mesh to the face image in each frame;

forming a binary mask providing the pixels to be recolored based on a weighted sum of information from the current frame and information from at least one previous frame;

aligning the binary mask to the mesh on each frame; and

applying anew colour and a new intensity value to the pixels of each frame that are to be recolored.

18. The mobile computerized system of claim 17, wherein the new colour is applied without net-full intensity so that the colour is partly opaque.

19. The mobile computerized system of claim 17, wherein the new colour and intensity value for the pixels of each frame that are to be recolored are calculated using color and intensity values of the neighbor pixel.

20. The mobile computerized system of claim 17, wherein forming a mask for each frame by aligning the mask to the mesh on each frame comprises the following steps:

making a projection of a mesh to a regular grid to separate the mesh into 100×

100 cells by the regular grid;

determining the mesh element to which a cell of the grid corresponds to, for each cell; and

determining the pixel corresponding to each of the determined mesh elements.

21. The mobile computerized system of claim 20, wherein determining the pixel corresponding to each of the determined mesh elements s performed using a breadth-first-search.

22. The mobile computerized system of claim 20, wherein making a projection of a mesh to the grid is performed once and steps of determining the mesh element and determining the pixel corresponding to each of the determined mesh elements are performed for each frame.

23. The mobile computerized system of claim 17, wherein forming a binary mask providing the pixels to be recolored comprises:

dividing the mesh into at least 1600 cells by a grid;

scanning each of the frames by a square pixel region, wherein the size of the square pixel region in pixels is determined from the width of an eye in pixels;

determining the mean gray-value intensity of each scanned square of the frame;

detecting pixels to be recolored by comparing the gray-value intensity of each pixel with the mean value of the square it belongs to and by marking the pixels that have the intensity at least 0.92 times of the mean intensity value as pixels to be recolored;

removing a region from the mesh around the pixels to be recolored, wherein the region is a circle having a center on a pixel marked as a pixel to be recolored and a radius as a linear function of the eye width;

applying a binary morphological closing to the plurality of marked pixels, wherein the binary morphological closing comprises morphological dilation and morphological erosion applied with a radius determined from the width of eye in pixels;

removing small and large regions of the pixels to be recolored from the mesh, wherein the regions with an area less than R²*0.2 are detected as small and the regions with an area more than R²*8 are detected as large, wherein R is radius of binary morphological operations and each region is a four-connected component.

24. The mobile computerized system of claim 23, wherein removing small and large regions of the pixels to be recolored is performed using a breadth-first-search.