Method for real-time video processing involving changing features of an object in the video

• US 9,928,874 B2
• Filed: 06/25/2014
• Issued: 03/27/2018
• Est. Priority Date: 02/05/2014
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Shaburova, Elena
• Primary Examiner(s): Niu, Feng
• Application Number:
  US14/314,324
• Publication Number:
  US 20150222821A1
• Time in Patent Office: 1,371 Days
• Field of Search: None
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/42 Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 10/56 : relating to colour
• CPC Class Codes: G06V 10/60 : relating to illumination properties, e.g. using a reflectance or lighting model
• CPC Class Codes: G06V 20/40 : in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/46 : Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 40/167 : using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/168 : Feature extraction; Face representation
• CPC Class Codes: G06V 40/171 : Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/174 : Facial expression recognition
• CPC Class Codes: G06V 40/20 : Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G11B 27/031 : Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: H04N 5/262 : Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
• CPC Class Codes: H04N 9/79 : Processing of colour television signals in connection with recording

Abstract

A method for real-time video processing for changing features of an object in a video is presented. The method comprises providing an object in the video, where the object is at least partially and at least occasionally presented in frames of the video. The method detects the object in the video and generates a list of at least one element of the object. The list is based on the object'"'"'s features to be changed according to a request for modification. The at least one element of the object is detected and tracked. The method transforms the frames of the video such that the at least one element of the object is modified according to the request for modification.

Claims

1. A computer-implemented method for real-time video processing for changing features of an object in a video, the method being performed in connection with a computerized system comprising a processing unit and a memory, the method comprising:

providing the object in the video, the object being at least partially and at least occasionally presented in frames of the video;

detecting the object in the video;

generating a list of at least one element of the object, the list being based on the object'"'"'s features to be changed according to a request for modification;

detecting the at least one element of the object in the video;

tracking the at least one element of the object in the video;

transforming the frames of the video to generate modified frames such that the at least one element of the object is modified within the video in which the object is detected while the video is provided at a computing device and the object is detected in the video, the frames transformed according to the request for modification resulting in at least one modified element, the at least one modified element being a version of the at least one elements detected and tracked within the video; and

providing the modified frames including the at least one modified element within the video.

2. The computer-implemented method of claim 1, wherein transforming the frames of the video comprises:

calculating characteristic points for each of the at least one element of the object;

generating a mesh based on the calculated characteristic points for each of the at least one element of the object;

tracking the at least one element of the object in the video, wherein tracking comprises aligning the mesh for each of the at least one element with a position of the corresponding each of the at least one element from frame to frame;

generating a set of first points on the mesh for each of the at least one element of the object based on the request for modification;

generating a set of second points on the mesh based on the set of first points and the request for modification; and

transforming the frames of the video such that the at least one element of the object is modified, wherein, for each of the at least one element of the object, the set of first points comes into the set of second points using the mesh.

3. The computer-implemented method of claim 2, further comprising:

generating a square grid associated with the background of the object in the video; and

transforming the background of the object using the square grid in accordance with the modification of the at least one element of the object.

4. The computer-implemented method of claim 2, further comprising:

generating at least one square grid associated with regions of the object that are adjacent to the modified at least one element of the object; and

modifying the regions of the object that are adjacent to the modified at least one element of the object in accordance with the modification of the at least one element of the object using the at least one square grid.

5. The computer-implemented method of claim 1, wherein the detecting of the object in the video is implemented with the use of Viola-Jones method.

6. The computer-implemented method of claim 2, wherein calculating of the object'"'"'s characteristic points is implemented with the use of an Active Shape Model (ASM).

7. The computer-implemented method of claim 1, wherein transforming the frames of the video comprises:

calculating characteristic points for each of the at least one element of the object;

generating a mesh based on the calculated characteristic points for each of the at least one element of the object;

generating a set of first points on the mesh for each of the at least one element of the object based on the request for modification;

generating at least one area based on the set of first points for each of the at least one element of the object;

tracking the at least one element of the object in the video, wherein the tracking comprises aligning the at least one area of each of the at least one element with a position of the corresponding each of the at least one element from frame to frame; and

transforming the frames of the video such that the properties of the at least one area are modified based on the request for modification.

8. The computer-implemented method of claim 7, wherein modification of the properties of the at least one area includes changing color of the at least one area.

9. The computer-implemented method of claim 7, wherein modification of the properties of the at least one area includes removing at least part of the at least one area from the frames of the video.

10. The computer-implemented method of claim 7, wherein modification of the properties of at least one area includes adding at least one new object to the at least one area, the at least one new object being based on the request for modification.

11. The computer-implemented method of claim 1, wherein objects to be modified include a human'"'"'s face.

12. The computer implemented method of claim 1, wherein the processed video comprises a video stream.

13. A mobile computerized system comprising a processing unit and a memory, the memory storing instructions for:

providing an object in the video, the object being at least partially and at least occasionally presented in frames of the video;

detecting the object in the video;

generating a list of at least one element of the object, the list being based on the object'"'"'s features to be changed according to a request for modification;

detecting the at least one element of the object in the video;

tracking the at least one element of the object in the video;

transforming the frames of the video to generate modified frames such that the at least one element of the object is modified within the video in which the object is detected while the video is provided at a computing device and the object is detected in the video, the frames transformed according to the request for modification resulting in at least one modified element, the at least one modified element being a version of the at least one elements detected and tracked within the video; and

providing the modified frames including the at least one modified element within the video.

14. The mobile computerized system of claim 13, wherein transforming the frames of the video comprises:

calculating characteristic points for each of the at least one element of the object;

generating a mesh based on the calculated characteristic points for each of the at least one element of the object;

tracking the at least one element of the object in the video, wherein tracking comprises aligning the mesh for each of the at least one element with a position of the corresponding each of the at least one element from frame to frame;

generating a set of first points on the mesh for each of the at least one element of the object based on the request for modification;

generating a set of second points on the mesh based on the set of first points and the request for modification; and

transforming the frames of the video such that the at least one element of the object is modified, wherein, for each of the at least one element of the object, the set of first points comes into the set of second points using the mesh.

15. The mobile computerized system of claim 14, further comprising:

generating a square grid associated with the background of the object in the video; and

transforming the background of the object using the square grid in accordance with the modification of the at least one element of the object.

16. The mobile computerized system of claim 14, further comprising:

generating at least one square grid associated with regions of the object that are adjacent to the modified at least one element of the object; and

modifying the regions of the object that are adjacent to the modified at least one element of the object in accordance with the modification of the at least one element of the object using the at least one square grid.

17. The mobile computerized system of claim 14, wherein calculating of the object'"'"'s characteristic points is implemented with the use of an Active Shape Model (ASM).

18. The mobile computerized system of claim 13, wherein the detecting of the object in the video is implemented with the use of Viola-Jones method.

19. The mobile computerized system of claim 13, wherein transforming the frames of the video comprises:

calculating characteristic points for each of the at least one element of the object;

generating a mesh based on the calculated characteristic points for each of the at least one element of the object;

generating a set of first points on the mesh for each of the at least one element of the object based on the request for modification;

generating at least one area based on the set of first points for each of the at least one element of the object;

tracking the at least one element of the object in the video, wherein the tracking comprises aligning the at least one area of each of the at least one element with a position of the corresponding each of the at least one element from frame to frame; and

transforming the frames of the video such that the properties of the at least one area are modified based on the request for modification.

20. The mobile computerized system of claim 19, wherein modification of the properties of the at least one area includes changing color of the at least one area.

21. The mobile computerized system of claim 19, wherein modification of the properties of the at least one area includes removing at least part of the at least one area from the frames of the video.

22. The mobile computerized system of claim 19, wherein modification of the properties of at least one area includes adding at least one new object to the at least one area, the at least one new object is based on the request for modification.

23. The mobile computerized system of claim 13, wherein objects to be modified include a human'"'"'s face.