METHOD FOR REAL TIME VIDEO PROCESSING FOR CHANGING PROPORTIONS OF AN OBJECT IN THE VIDEO

• US 20150221118A1
• Filed: 06/25/2014
• Published: 08/06/2015
• Est. Priority Date: 02/05/2014
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Shaburova, Elena
• Granted Patent: US 10,586,570 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/42 Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 10/56 : relating to colour
• CPC Class Codes: G06V 10/60 : relating to illumination properties, e.g. using a reflectance or lighting model
• CPC Class Codes: G06V 20/40 : in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/46 : Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 40/167 : using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/168 : Feature extraction; Face representation
• CPC Class Codes: G06V 40/171 : Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/174 : Facial expression recognition
• CPC Class Codes: G06V 40/20 : Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G11B 27/031 : Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: H04N 5/262 : Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
• CPC Class Codes: H04N 9/79 : Processing of colour television signals in connection with recording

Abstract

Method involving: providing an object in the video that at least partially and at least occasionally is presented in frames of a video; detecting the object in the video, wherein said detection comprises detecting feature reference points of the object; tracking the detected object in the video, wherein the tracking comprises creating a mesh that is based on the detected feature reference points of the object and aligning the mesh to the object in each frame; generating a first set of node points on the created mesh based on a request for changing proportions; generating a second set of node points based on the first set of node points; and transforming the frames of the video in such way that the object'"'"'s proportions are transformed in accordance with the second set of the node points using the mesh.

Claims

1. A computer implemented method for real time video processing for changing proportions of an object in the video, the method comprising:

providing an object in the video that at least partially and at least occasionally is presented in frames of the video;

detecting the object in the video, wherein said detection comprises detecting feature reference points of the object;

tracking the detected object in the video, wherein the tracking comprises creating a mesh that is based on the detected feature reference points of the object and aligning the mesh to the object in each frame;

generating a first set of node points on the created mesh based on a request for changing proportions;

generating a second set of node points based on the first set of node points; and

transforming the frames of the video in such way that the object'"'"'s proportions are transformed in accordance with the second set of the node points using the mesh.

2. The computer implemented method of claim 1, further comprising:

creating a square grid associated with a background of the object in the video; and

transforming the background of the object using the square grid to avoid the background distortion.

3. The computer implemented method of claim 1, wherein the object in the video to be detected is a human face.

4. The computer implemented method of claim 3, wherein the object'"'"'s feature reference points are at least one of the points indicating eyebrows vertical position, eyes vertical position, eyes width, eyes height, eye separation distance, nose vertical position, nose pointing up, mouth vertical position, mouth width, chin width, upper lip raiser, jaw drop, lip stretcher, left brow lowerer, right brow lowerer, lip corner depressor, and outer brow raiser.

5. The computer implemented method of claim 1, wherein the method further comprises:

indicating a presence of an object from a list of objects in frames of the video, wherein the list further comprises rules for changing proportions of each object from the list; and

generating a request for changing proportions of the object which presence in frames of the video is indicated.

6. The computer implemented method of claim 1, wherein the method further comprises:

defining an object to be changed in frames of the video and rules for changing proportions of the object by a user; and

generating a request for changing proportions of the object defined by the user.

7. The computer implemented method of claim 6, wherein the method further comprises:

defining by a user a frame area of the video to be processed,

wherein the frame area to be processed sets a frame area of the video such that only proportions of those objects or their parts which are positioned in the frame area to be processed are changed.

8. The computer implemented method of claim 1, wherein the method further comprises:

randomly selecting at least one object to be changed in frames of the video out of the objects in frames of the video and randomly selecting at least one rule for changing proportions of the selected object out of a list of rules; and

generating a request for changing proportions of the randomly selected object based on the randomly selected rules.

9. The computer implemented method of claim 1, wherein the detecting of the object in the video is implemented with the use of Viola-Jones method.

10. The computer implemented method of claim 1, wherein the detecting of the object'"'"'s feature points is implemented with the use of an Active Shape Model (ASM).

11. The computer implemented method of claim 1, wherein the processed video comprises a video stream.

12. A mobile computerized system comprising a central processing unit and a memory, the memory storing instructions for:

providing an object in the video that at least partially and at least occasionally is presented in frames of a video;

detecting the object in the video, wherein said detection comprises detecting feature reference points of the object;

tracking the detected object in the video, wherein the tracking comprises creating a mesh that is based on the detected feature reference points of the object and aligning the mesh to the object in each frame;

generating a first set of node points on the created mesh based on a request for changing proportions;

generating a second set of node points based on the first set of node points; and

transforming the frames of the video in such way that the object'"'"'s proportions are transformed in accordance with the second set of the node points using the mesh.

13. The mobile computerized system of claim 12, wherein the memory further stores instructions for:

creating a square grid associated with a background of the object in the video; and

transforming the background of the object using the square grid to avoid the background distortion.

14. The mobile computerized system of claim 12, wherein the object in the video to be detected is a human face.

15. The mobile computerized system of claim 14, wherein the object'"'"'s feature reference points are at least one of the points indicating eyebrows vertical position, eyes vertical position, eyes width, eyes height, eye separation distance, nose vertical position, nose pointing up, mouth vertical position, mouth width, chin width, upper lip raiser, jaw drop, lip stretcher, left brow lowerer, right brow lowerer, lip corner depressor, and outer brow raiser.

16. The mobile computerized system of claim 12, wherein the memory storing further instructions for:

indicating a presence of an object from a list of objects in frames of the video, wherein the list further comprises rules for changing proportions of each object from the list; and

generating a request for changing proportions of the object which presence in frames of the video is indicated.

17. The mobile computerized system of claim 12, wherein the memory storing further instructions for:

defining an object to be changed in frames of the video and rules for changing proportions of the object by a user; and

generating a request for changing proportions of the object defined by the user.

18. The mobile computerized system of claim 17, wherein the memory storing further instructions for:

defining by a user a frame area of the video to be processed,

wherein the frame area to be processed sets a frame area of the video such that only proportions of those objects or their parts which are positioned in the frame area to be processed are changed.

19. The mobile computerized system of claim 12, wherein the memory storing further instructions for:

randomly selecting at least one object to be changed in frames of the video out of the objects in frames of the video and randomly selecting at least one rule for changing proportions of the selected object out of a list of rules; and

generating a request for changing proportions of the randomly selected object based on the randomly selected rules.

20. The mobile computerized system of claim 12, wherein the detecting of the object in the video is implemented with the use of Viola-Jones method.

21. The mobile computerized system of claim 12, wherein the detecting of the object'"'"'s feature points is implemented with the use of an Active Shape Model (ASM).