Subject stabilisation based on the precisely detected face position in the visual input and computer systems and computer-implemented methods for implementing thereof

• US 10,129,476 B1
• Filed: 04/25/2018
• Issued: 11/13/2018
• Est. Priority Date: 04/26/2017
• Status: Active Grant

• Current Assignee: Banuba Ltd.
• Original Assignee: Banuba Ltd.
• Inventors: Hushchyn, Yury, Sakolski, Aliaksei
• Primary Examiner(s): Haskins, Twyler
• Assistant Examiner(s): Bhuiyan, Fayez
• Application Number:
  US15/962,347
• Publication Number:
  US 20180316860A1
• Time in Patent Office: 202 Days
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory

Abstract

In some embodiments, the present invention provides for an exemplary computer system that may include: a camera component configured to acquire a visual content, wherein the visual content having a plurality of frames with a visual representation of a face of a person; a processor configured to: apply, for each frame, a multi-dimensional face detection regressor for fitting at least one meta-parameter to detect or to track a plurality of multi-dimensional landmarks representative of a face; apply a face movement detection algorithm to identify each displacement of each respective multi-dimensional landmark between frames; and apply a face movement compensation algorithm to generate a face movement compensated output that stabilizes the visual representation of the face.

Claims

1. A computer-implemented method, comprising:

obtaining, by at least one processor, a plurality of frames having a visual representation of a face of at least one person;

applying, by the at least one processor, for each frame, at least one multi-dimensional face detection regressor for fitting at least one meta-parameter to detect or to track a plurality of multi-dimensional landmarks that are representative of a presence of a face of at least one person in each respective frame;

separating, by the at least one processor, for each frame in the plurality of frames, the face of the at least one person from a background based on utilizing at least one deep learning algorithm;

applying, by the at least one processor, for each frame in the plurality of frames, at least one face movement detection algorithm to identify each displacement of each respective multi-dimensional landmark of the plurality of multi-dimensional landmarks between frames;

applying, by the at least one processor, for each two sequential frames in the plurality of frames, at least one face movement compensation algorithm that is configured to at least;

i) determine that a current displacement value of at least one respective multi-dimensional landmark of the plurality of multi-dimensional landmarks between two sequential frames exceeds a pre-determined threshold value, and

ii) re-draw the face of the at least one person for a particular frame of the two sequential frames, in which the current displacement value exceeds the pre-determined threshold value, to reduce the current displacement value of the at least one respective multi-dimensional landmark to an updated displacement value that is less than the pre-determined threshold value to generate a re-drawn face of the at least one person;

wherein the pre-determined threshold value is between 1 and 20 Hz; and

combining, by the at least one processor, the re-drawn face of the at least one person in the particular frame of the two sequential frames with the background to generate a face movement compensated output that stabilizes the visual representation of the face of the at least one person between the two sequential frames of the plurality of frames.

2. The method of claim 1, wherein the plurality of frames is part of a video stream.

3. The method of claim 2, wherein the video stream is a real-time video stream.

4. The method of claim 3, wherein the real-time video stream is a live video stream.

5. The method of claim 1, wherein the pre-determined threshold value is between 10 and 20 Hz.

6. The method of claim 1, wherein the method further comprising:

applying, by the at least one processor, at least one visual encoding algorithm to transform the plurality of face movement compensated frames into a visual encoded output.

7. The method of claim 6, wherein the at least one visual encoding algorithm comprises a perceptual coding compression based on a human visual system model to remove a perceptual redundancy.

8. The method of claim 1, wherein the plurality of frames is obtained by a camera of a portable electronic device and wherein the at least one processor is a processor of the portable electronic device.

9. A system, comprising:

a camera component, wherein the camera component is configured to acquire a visual content, wherein the visual content comprises a plurality of frames having a visual representation of a face of at least one person;

at least one processor;

a non-transitory computer memory, storing a computer program that, when executed by the at least one processor, causes the at least one processor to;

apply, for each frame of the plurality of frames, at least one multi-dimensional face detection regressor for fitting at least one meta-parameter to detect or to track a plurality of multi-dimensional landmarks that are representative of a presence of a face of at least one person in each respective frame;

separate, for each frame in the plurality of frames, the face of the at least one person from a background based on utilizing at least one deep learning algorithm;

apply, for each frame in the plurality of frames, at least one face movement detection algorithm to identify each displacement of each respective multi-dimensional landmark of the plurality of multi-dimensional landmarks between frames;

apply, for each two sequential frames in the plurality of frames, at least one face movement compensation algorithm that is configured to at least;

i) determine that a current displacement value of at least one respective multi-dimensional landmark of the plurality of multi-dimensional landmarks between two sequential frames exceeds a pre-determined threshold value, and

ii) re-draw the face of the at least one person for a particular frame of the two sequential frames, in which the current displacement value exceeds the pre-determined threshold value, to reduce the current displacement value of the at least one respective multi-dimensional landmark to an updated displacement value that is less than the pre-determined threshold value to generate a re-drawn face of the at least one person;

wherein the pre-determined threshold value is between 1 and 20 Hz; and

combine the re-drawn face of the at least one person in the particular frame of the two sequential frames with the background to generate a face movement compensated output that stabilizes the visual representation of the face of the at least one person between the two sequential frames of the plurality of frames.

10. The system of claim 9, wherein the plurality of frames is part of a video stream.

11. The system of claim 10, wherein the video stream is a real-time video stream.

12. The system of claim 11, wherein the real-time video stream is a live video stream.

13. The system of claim 9, wherein the pre-determined threshold value is between 10 and 20 Hz.

14. The system of claim 9, wherein the at least one processor is further configured to:

apply at least one visual encoding algorithm to transform the plurality of face movement compensated frames into a visual encoded output.

15. The system of claim 14, wherein the at least one visual encoding algorithm comprises a perceptual coding compression based on a human visual system model to remove a perceptual redundancy.

16. The system of claim 9, wherein the plurality of frames is obtained by a camera of a portable electronic device and wherein the at least one processor is a processor of the portable electronic device.