Computer systems and computer-implemented methods configured to track multiple eye-gaze and heartrate related parameters during users' interaction with electronic computing devices

• US 10,552,986 B1
• Filed: 07/18/2019
• Issued: 02/04/2020
• Est. Priority Date: 07/20/2018
• Status: Active Grant

• Current Assignee: Banuba Ltd.
• Original Assignee: Banuba Ltd.
• Inventors: Arkhipau, Viachaslau, Hushchyn, Yury, Sakolski, Aliaksei, Kachatkou, Dzmitry
• Primary Examiner(s): Liu, Li
• Application Number:
  US16/516,042
• Publication Number:
  US 20200027245A1
• Time in Patent Office: 201 Days
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/20 : Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G09B 5/02 : with visual presentation of the material to be studied, e.g. using film strip

Abstract

Embodiments of the present disclosure include receiving a sequence of images of a face of a user. A three-dimensional (3D) model of the face is generated and 3D facial points associated with flat facial surfaces are determined. The 3D facial points are projected onto a screen coordinate plane to produce a two-dimensional (2D) facial points. A hue is determined for each pixel associated with each of the 2D facial points in each image. A mean hue value is determined for each image. A spectral representation of a variation in each mean hue value across the sequence of images is determined. A frequency of a main hue is determined based on a largest weight of the variation in each mean hue value. A heart rate of the user is determined based on facial blood circulation according to the frequency of the main hue, and an activity recommendation is displayed.

Claims

1. A method, comprising:

receiving, by at least one processor, a sequence of images of a face of a user from at least one image capture device;

generating, by the at least one processor, a three-dimensional (3D) model of the face in the sequence of images according to a face recognition algorithm;

determining, by the at least one processor, a plurality of 3D facial points associated with one or more flat facial surfaces on the 3D model of the face;

projecting, by the at least one processor, the plurality of 3D facial points onto a screen coordinate plane to produce a plurality of two-dimensional (2D) facial points;

wherein the screen coordinate plane comprises a set of coordinates relative to a screen in communication with the processor;

determining, by the at least one processor, a hue of each pixel associated with each of the plurality of 2D facial points in each image of the sequence of images;

determining, by the at least one processor, a mean hue value associated with the face based on an averaging of the hue of each pixel of each of the plurality of 2D facial points in each image of the sequence of images;

wherein the mean hue value is determined with a moving window;

determining, by the at least one processor, a spectral representation of a variation in the mean hue value across the sequence of images;

determining, by the at least one processor, a frequency of a main hue based on a largest weight of the variation in the mean hue value for each of the plurality of 2D facial points in each image of the sequence of images;

determining, by the at least one processor, a heart rate of the user based on a blood circulation to facial blood vessels according to the frequency of the main hue; and

causing, by the at least one processor, to display at a screen of a computing device at least one activity recommendation based at least in part on the heart rate.

2. The method of claim 1, further comprising filtering, by the at least one processor, the mean hue value for each of the plurality of 2D facial points using a Butterworth filter.

3. The method of claim 1, further comprising:

determining, by the at least one processor, a color value associated with each pixel of the plurality of 2D facial points in each image of the sequence of images;

determining, by the at least one processor, a separation in a brightness and a color from each color value; and

determining, by the at least one processor, the hue based on the color of each pixel.

4. The method claim 1, wherein the moving window is one of a Hann window or a Gaussian window centered around zero.

5. The method of claim 1, wherein the moving window is a predetermined time moving window.

6. The method of claim 1, wherein the sequence of images comprises a real-time feed of images captured by image capture device.

7. The method of claim 1, wherein the image capture device is a camera of a mobile computing device.

8. The method of claim 1, wherein the plurality of 3D facial points comprises between 10 and 100 3D facial points.

9. The method of claim 1, wherein the 3D model comprises 3308 points.

10. A method, comprising:

receiving, by at least one memory device, a sequence of images of a face of a user from an image capture device;

generating, by at least one processor, a three-dimensional (3D) model of the face in the sequence of images according to a face recognition algorithm;

determining, by the at least one processor, a plurality of 3D facial points associated with one or more flat facial surfaces on the 3D model of the face;

projecting, by the at least one processor, the plurality of 3D facial points onto a screen coordinate plane to produce a plurality of two-dimensional (2D) facial points;

wherein the screen coordinate plane comprises a set of coordinates relative to a screen in communication with the processor;

determining, by the at least one processor, a hue of each pixel associated with each of the plurality of 2D facial points in each image of the sequence of images;

determining, by the at least one processor, a mean hue value associated with each of the plurality of 2D facial points in each image of the sequence of images;

wherein the mean hue value is determined with a moving window;

determining, by the at least one processor, a spectral representation of a variation in the mean hue value for each of the plurality of 2D facial points across the sequence of images;

determining, by the at least one processor, a frequency of a main hue based on a largest weight of the variation in the mean hue value for each of the plurality of 2D facial points across the sequence of images;

determining, by the at least one processor, a heart rate based blood circulation to facial blood vessels according to the frequency of the main hue;

determining, by the at least one processor, a phase of the heart rate based on the frequency of the main hue;

determining, by the at least one processor, an average period of the heart rate;

determining, by the at least one processor, phase differences based on a change in the phase across the sequence of images;

determining, by the at least one processor, an absolute difference from the average period based on the phase differences to estimate heart rate variability of the user; and

causing, by the at least one processor, to display with a screen an activity recommendation based at least on the heart rate variability.

11. The method of claim 10, further comprising:

determining, by the at least one processor, a power associated with each hue frequency in the spectral representation based on a convolution of a sine of the mean hue value and on a convolution of a cosine of the mean hue value;

wherein each hue frequency is associated with a distinct hue; and

determining, by the at least one processor, the frequency having a maximum power in the spectral representation;

wherein the phase is calculated across the sequence of images based on the convolution of the sine of the mean hue value and on the convolution of the cosine of the mean hue value for the frequency with the maximum power.

12. The method of claim 10, further comprising:

determining, by the at least one processor, a phase associated with each image in the sequence of images;

wherein the phase is based on an arc angle of a convolution of a sine of the mean hue value and on a convolution of a cosine of the mean hue value for the frequency of the main hue in the spectral representation; and

determining, by the at least one processor, the phase differences between each image in the sequence of images based on a difference between the phase associated with each image in the sequence of images.

13. The method of claim 10, further comprising:

determining, by the at least one processor, phase change intervals based on intervals of time between adjacent images in the sequence of images having threshold phase differences; and

wherein the threshold phase differences comprise the phase differences greater than about pi.

14. The method of claim 13, further comprising:

determining, by the at least one processor, relevant threshold phase change intervals based on the phase change intervals and the average period;

wherein the relevant threshold phase change intervals comprise intervals of the threshold phase change intervals having a duration between about one quarter of the average period and about twice the average period; and

determining, by the at least one processor, the absolute difference between the relevant threshold phase change intervals and the average period.

15. The method of claim 10, further comprising:

determining, by the at least one processor, a color value associated with each pixel of the plurality of 2D facial points in each image of the sequence of images;

determining, by the at least one processor, a separation between a brightness and a color from each color value; and

determining, by the processor, the hue based on the color of each pixel.

16. The method claim 10, wherein the moving window comprises one of a Hann window or a Gaussian window centered around zero.

17. The method of claim 10, wherein the image capture device is a camera of a mobile computing device.

18. The method of claim 10, wherein the plurality of 3D facial points comprise between about 10 and about 100 3D facial points.

19. A system, comprising:

a memory device;

a video capture device; and

a processing device;

wherein the processing device is configured to perform instructions stored in the memory device, the instructions comprising;

receiving, by a memory device, a sequence of images of a face of a user from the video capture device;

generating, by a processor, a three-dimensional (3D) model of the face in the sequence of images according to a face recognition algorithm;

determining a plurality of 3D facial points associated with one or more flat facial surfaces on the 3D model of the face;

projecting the plurality of 3D facial points onto a screen coordinate plane to produce a plurality of two-dimensional (2D) facial points;

wherein the screen coordinate plane comprises a set of coordinates relative to a screen in communication with the processor;

determining a hue of each pixel associated with each of the plurality of 2D facial points in each image of the sequence of images;

determining a mean hue value associated with each of the plurality of 2D facial points in each image of the sequence of images;

wherein the mean hue value is determined with a moving window;

determining a spectral representation of a variation in the mean hue value for each of the plurality of 2D facial points across the sequence of images;

determining a frequency of a main hue based on a largest weight of the variation in the mean hue value for each of the plurality of 2D facial points across the sequence of images;

determining a heart rate based blood circulation to facial blood vessels according to the frequency of the main hue;

determining a phase of the heart rate based on the frequency of the main hue;

determining an average period of the heart rate;

determining phase differences based on a change in the phase across the sequence of images;

determining an absolute difference from the average period based on the phase differences to estimate heart rate variability of the user; and

causing to display with a screen an activity recommendation based at least on the heart rate variability.

20. The system of claim 19, wherein the video capture device is a camera of a mobile computing device.