METHOD FOR REAL-TIME VIDEO PROCESSING INVOLVING RETOUCHING OF AN OBJECT IN THE VIDEO

• US 20150221136A1
• Filed: 06/25/2014
• Published: 08/06/2015
• Est. Priority Date: 02/05/2014
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Shaburova, Elena, Shaburov, Victor
• Granted Patent: US 10,438,631 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/42 Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 10/56 : relating to colour
• CPC Class Codes: G06V 10/60 : relating to illumination properties, e.g. using a reflectance or lighting model
• CPC Class Codes: G06V 20/40 : in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/46 : Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 40/167 : using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/168 : Feature extraction; Face representation
• CPC Class Codes: G06V 40/171 : Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/174 : Facial expression recognition
• CPC Class Codes: G06V 40/20 : Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G11B 27/031 : Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: H04N 5/262 : Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
• CPC Class Codes: H04N 9/79 : Processing of colour television signals in connection with recording

Abstract

Method for real time video processing for retouching an object in a video, comprising: providing an object in the video steam, the object being at least partially and at least occasionally presented in frames of the video; setting a degree of retouching; generating a list of at least one element of the object selected based on a request of retouching and the degree of retouching; detecting the at least one element of the object in the video and parameters of the at least one element; calculating new parameters of the at least one element according to the degree of retouching; detecting characteristic points for each of the at least one element of the object; generating a mesh based on the characteristic points for each of the at least one element of the object; tracking the at least one element of the object in the video, wherein the tracking comprises aligning the mesh for each of the at least one element with a position of the corresponding each of the at least one element; and transforming the frames of the video such that the at least one element of the object is modified by the application of the new parameters to the at least one element of the object using the mesh.

Claims

1. A computer-implemented method for real time video processing for retouching an object in a video, the method being performed in connection with a computerized system comprising a processing unit and a memory, the method comprising:

providing an object in the video steam, the object being at least partially and at least occasionally presented in frames of the video;

setting a degree of retouching;

generating a list of at least one element of the object selected based on a request of retouching and the degree of retouching;

detecting the at least one element of the object in the video and parameters of the at least one element;

calculating new parameters of the at least one element according to the degree of retouching;

detecting characteristic points for each of the at least one element of the object;

generating a mesh based on the characteristic points for each of the at least one element of the object;

tracking the at least one element of the object in the video, wherein the tracking comprises aligning the mesh for each of the at least one element with a position of the corresponding each of the at least one element; and

transforming the frames of the video such that the at least one element of the object is modified by the application of the new parameters to the at least one element of the object using the mesh.

2. The computer-implemented method of claim 1, wherein the parameters of the at least one element of the object comprise its location, shape, color, and properties of a covering layer.

3. The computer-implemented method of claim 1, wherein applying the new parameters to the at least one element of the object using the mesh comprises:

generating a set of first points on the mesh for each of the at least one element of the object based on the actual location of the at least one element;

generating a set of second points on the mesh based on the request for retouching;

changing location of the at least one element of the object in accordance with the set of first points and the set of second points; and

modifying the surrounding area which surrounds the at least one element to avoid distortion of the surrounding area.

4. The computer-implemented method of claim 1, wherein applying the new parameters to the at least one element of the object using the mesh comprises:

generating a set of first points on the mesh for each of the at least one element of the object based on the degree of retouching;

generating a set of second points on the mesh based on the set of first points and the degree of retouching; and

changing proportions of the at least one element of the object in accordance with the degree of retouching, the set of first points and the set of second points, wherein, for each of the at least one element of the object, the set of first points comes into the set of second points using the mesh.

5. The computer-implemented method of claim 1, wherein applying the new parameters to the at least one element of the object using the mesh comprises:

generating a set of first points on the mesh for each of the at least one element of the object based on the degree of retouching;

generating at least one area based on the set of first points for each of the at least one element of the object; and

changing color of the at least one area based on the degree of retouching.

6. The computer-implemented method of claim 1, wherein applying the new parameters to the at least one element of the object using the mesh comprises:

generating a set of first points on the mesh for each of the at least one element of the object based on the degree of retouching;

generating a covering layer based on the set of first points and the calculated new parameters of the at least one element for each of the at least one element of the object; and

applying the generated covering layer to each of the at least one element of the object based on the degree of retouching.

7. The computer-implemented method of claim 1, wherein transforming the frames of the video comprises:

scanning the at least one element of the object in at least two successive frames;

generating a ratio of change of the at least one element of the object; and

adjusting modification of the at least one element based on the ratio of change of the at least one element.

8. The computer implemented method of claim 1, wherein the object in the video to be retouched is a human face.

9. The computer-implemented method of claim 8, the method comprising:

detecting parts of the human face which are selected based on the request for retouching; and

calculating new parameters for the selected parts of the human face to make the parts of the human face be symmetrical relative to the central axis of the human face.

10. The computer-implemented method of claim 8, further comprising:

creating a square grid associated with a background of the human face in the video; and

transforming the background of the human face using the square grid to achieve blurring of the background.

11. The computer implemented method of claim 1, wherein the processed video comprises a video stream.

12. A computer-implemented method of real time video processing for retouching an object in a video, the method being performed in connection with a computerized system comprising a processing unit and a memory, the method comprising:

receiving a request for retouching of an object, the object being at least partially and at least occasionally presented in frames of the video;

setting a degree of retouching;

generating an average image of the object based on a plurality of frames of the video and detecting feature reference points of the object;

generating a list of at least one element of the object selected based on the request of retouching and the degree of retouching;

detecting the at least one element of the object in the video and parameters of the at least one element;

calculating new parameters of the at least one element according to the degree of retouching;

calculating characteristic points for each of the at least one element of the object;

generating a first mesh based on the characteristic points for each of the at least one element of the object;

generating a covering layer based on the average image of the object and the calculated new parameters of the at least one element, wherein the covering layer comprises the average image with the at least one element of the object modified by the application of the calculated new parameters using the first mesh;

tracking the feature reference points of the object in the video, wherein the tracking comprises creating a second mesh that is based on the positions of the feature reference points of the object, and aligning the second mesh to the object in each frame;

transforming the covering layer based on the second mesh in each frame;

aligning the covering layer with the object using the second mesh; and

transforming the frames of the video such that the object is covered by the covering layer in each frame.

13. A mobile computerized system comprising a central processing unit and a memory, the memory storing instructions for:

providing an object in the video steam, the object being at least partially and at least occasionally presented in frames of the video;

setting a degree of retouching;

generating a list of at least one element of the object selected based on a request of retouching and the degree of retouching;

detecting the at least one element of the object in the video and parameters of the at least one element;

calculating new parameters of the at least one element according to the degree of retouching;

detecting characteristic points for each of the at least one element of the object;

generating a mesh based on the characteristic points for each of the at least one element of the object;

tracking the at least one element of the object in the video, wherein the tracking comprises aligning the mesh for each of the at least one element with a position of the corresponding each of the at least one element; and

transforming the frames of the video such that the at least one element of the object is modified by the application of the new parameters to the at least one element of the object using the mesh.

14. The mobile computerized system of claim 13, wherein the parameters of the at least one element of the object include its location, shape, color, and properties of a covering layer.

15. The mobile computerized system of claim 13, wherein applying the new parameters to the at least one element of the object using the mesh comprises:

generating a set of first points on the mesh for each of the at least one element of the object based on the actual location of the at least one element;

generating a set of second points on the mesh based on the request for retouching;

changing location of the at least one element of the object in accordance with the set of first points and the set of second points; and

modifying the surrounding area which surrounds the at least one element to avoid distortion of the surrounding area.

16. The mobile computerized system of claim 13, wherein applying the new parameters to the at least one element of the object using the mesh comprises:

generating a set of first points on the mesh for each of the at least one element of the object based on the degree of retouching;

generating a set of second points on the mesh based on the set of first points and the degree of retouching; and

changing proportions of the at least one element of the object in accordance with the degree of retouching, the set of first points and the set of second points, wherein, for each of the at least one element of the object, the set of first points comes into the set of second points using the mesh.

17. The mobile computerized system of claim 13, wherein applying the new parameters to the at least one element of the object using the mesh comprises:

generating a set of first points on the mesh for each of the at least one element of the object based on the degree of retouching;

generating at least one area based on the set of first points for each of the at least one element of the object; and

changing color of the at least one area based on the degree of retouching.

18. The mobile computerized system of claim 13, wherein applying the new parameters to the at least one element of the object using the mesh comprises:

generating a set of first points on the mesh for each of the at least one element of the object based on the degree of retouching;

generating a covering layer based on the set of first points and the calculated new parameters of the at least one element for each of the at least one element of the object; and

applying the generated covering layer to each of the at least one element of the object based on the degree of retouching.

19. The mobile computerized system of claim 13, wherein transforming the frames of the video comprises:

scanning the at least one element of the object in at least two successive frames;

generating a ratio of change of the at least one element of the object; and

adjusting modification of the at least one element based on the ratio of change of the at least one element.

20. The mobile computerized system of claim 13, wherein the object in the video stream to be retouched is a human face.

21. The mobile computerized system of claim 20, wherein the memory storing instructions for:

detecting parts of the human face which are selected based on the request for retouching; and

calculating new parameters for the selected parts of the human face to make the parts of the human face be symmetrical relative to the central axis of the human face.

22. The mobile computerized system of claim 20, wherein the memory further stores instructions for:

creating a square grid associated with a background of the human face in the video; and

transforming the background of the human face using the square grid to achieve blurring of the background.