METHOD FOR DETECTING FACIAL EXPRESSIONS AND EMOTIONS OF USERS

• US 20190138096A1
• Filed: 08/22/2018
• Published: 05/09/2019
• Est. Priority Date: 08/22/2017
• Status: Abandoned Application

• Current Assignee: Silicon Algebra Incorporated
• Original Assignee: Silicon Algebra Incorporated
• Inventors: Lee, Kevin, DeWolf, Quentin, Pella, John P., Reyes, Ivan Roberto
• Application Number:
  US16/109,614
• Publication Number:
  US 20190138096A1
• Time in Patent Office: Days
• CPC Class Codes: A61B 5/163 by tracking eye movement, gaze, or pupil change
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/316 Modalities, i.e. specific diagnostic methods
• CPC Class Codes: A61B 5/397 Analysis of electromyograms
• CPC Class Codes: A61B 5/6803 Head-worn items, e.g. helmets, masks, headphones or goggles
• CPC Class Codes: A61B 5/7264 Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G10L 25/18 the extracted parameters being spectral information of each sub-band
• CPC Class Codes: G10L 25/51 for comparison or discrimination
• CPC Class Codes: G16H 50/20 for computer-aided diagnosis, e.g. based on medical expert systems

Abstract

A method for detecting facial emotions includes: recording a set of electromyograph signals through a set of sense electrodes arranged about a viewing window in a virtual reality headset; deducting a reference signal from each electromyograph signal in the set of electromyograph signals to generate a set of composite signals; for each composite signal in the set of composite signals, transforming the composite signal into a spectrum of electromyograph components; for each facial action unit in a set of facial action units, calculating a score indicating presence of the facial action unit in the user'"'"'s facial musculature during the sampling interval based on the spectrum of electromyograph components; and mapping scores for the set of facial action units to a facial expression of the user during the sampling; transforming the facial expression of the user to an emotion of the user based on an emotion model.

Claims

1. A method for detecting a facial expression of a user comprises:

during a sampling interval, recording a set of electromyograph signals through a set of sense electrodes arranged about a viewing window in a virtual reality headset worn by a user;

deducting a reference signal from each electromyograph signal in the set of electromyograph signals to generate a set of composite signals;

for each composite signal in the set of composite signals, transforming the composite signal into a spectrum of oscillating electromyograph components within a frequency range of interest;

for each facial action unit in a set of facial action units, calculating a score indicating presence of the facial action unit in the user'"'"'s facial musculature during the sampling interval based on the spectrum of oscillating electromyograph components; and

mapping scores for the set of facial action units to a facial expression of the user during the sampling interval;

transforming the facial expression of the user to an emotion of the user based on an emotion model; and

outputting an identifier of the emotion to a device.

2. The method of claim 1:

wherein, for each facial action unit in the set of facial action units, calculating a score indicating presence of the facial action unit further comprises, for each facial action unit in the set of facial action units;

calculating a confidence level associated with the facial action unit that the user presented the facial action unit during the sampling interval based on a facial action unit model and the spectrum of oscillating electromyograph components; and

in response to the confidence level associated with the facial action unit exceeding a confidence threshold, identifying the facial action unit as a component facial action unit in a set of component facial action units;

wherein mapping scores for the set of facial action units to a facial expression of the user during the sampling interval further comprises, mapping the set of component facial action units to the facial expression of the user during the sampling interval.

3. The method of claim 2, further comprising:

identifying in the set of component facial action units a set of mutually exclusive facial action units based on an anatomical model;

identifying in the set of mutually exclusive facial action units the facial action unit associated with a maximum confidence level; and

removing the set of mutually exclusive facial action units from the set of component facial action units with the exception of the facial action unit associated with the maximum confidence level.

4. The method of claim 1, further comprising:

accessing a previous set of component facial action units identified during a previous sampling interval;

identifying a set of temporally incoherent facial action units based on the previous set of component facial action units, a time elapsed between the previous sampling interval and the sampling interval, and a facial motion model; and

removing, from the set of facial action units, the set of temporally incoherent facial action units.

5. The method of claim 1:

wherein each facial action unit in the set of facial action units comprises a set of discrete intensity levels of the facial action unit;

wherein, for each facial action unit in the set of facial action units, calculating a score indicating presence of the facial action unit further comprises, for each facial action unit in the set of facial action units;

for each discrete intensity level in the set of discrete intensity levels of the facial action unit;

calculating a confidence level associated with the facial action unit and the discrete intensity level that the user presented the facial action unit at the discrete intensity level during the sampling interval based on the facial action unit model and the spectrum of oscillating electromyograph components; and

in response to the confidence level associated with the facial action unit and the discrete intensity level exceeding the confidence threshold, identifying the facial action unit at the discrete intensity level as a component facial action unit in a set of component facial action units; and

wherein mapping scores for the set of facial action units to a facial expression of the user during the sampling interval further comprises, mapping the set of component facial action units to the facial expression of the user during the sampling interval.

6. The method of claim 1:

further comprising, during the sampling interval;

recording a galvanic skin response of the user through the set of sense electrodes;

recording a heartrate of the user through a heartrate monitor; and

recording a heartrate variability of the user through the heartrate monitor; and

wherein outputting the identifier of the emotion of the user further comprises, outputting the identifier of the emotion of the user based on the emotion model, the facial expression of the user, the galvanic skin response of the user, the heartrate of the user, and the heartrate variability of the user.

7. The method of claim 1, wherein outputting the identifier of the emotion of the user further comprises, outputting the identifier of the emotion of the user and an identifier of the expression of the user.

8. The method of claim 1, further comprising:

responsive to detecting a change between a previous identifier of an emotion of the user and the identifier of the emotion of the user during the sampling interval; and

recording content displayed on the viewing window of the virtual reality headset during the sampling interval.

9. A method for detecting a facial expression of a user comprises:

during a sampling interval, recording a set of electromyograph signals through a set of sense electrodes arranged about a viewing window in a virtual reality headset worn by a user;

deducting a reference signal from each electromyograph signal in the set of electromyograph signals to generate a set of composite signals;

for each composite signal in the set of composite signals, transforming the composite signal into a spectrum of oscillating electromyograph components within a frequency range of interest;

for each facial action unit in a set of facial action units;

calculating a confidence level associated with the facial action unit, based on the spectrum of oscillating electromyograph components; and

in response to the confidence level associated with the facial action unit exceeding a confidence threshold, identifying the facial action unit as a component facial action unit in a set of component facial action units;

mapping the set of component facial action units to a facial expression of the user during the sampling interval;

outputting an identifier of the facial expression of the user to a device.

10. The method of claim 9, further comprising:

during the sampling interval, recording an audio signal;

transforming the audio signal into a spectrum of oscillating audio components;

calculating a mouth position of the user based on the spectrum of oscillating audio components; and

outputting an identifier of the facial expression and a viseme value representing the mouth position of the user during the sampling interval to the device.

11. The method of claim 10, wherein outputting an identifier of the facial expression and the viseme value representing the mouth position further comprises, outputting an identifier for each of the set of component facial action units.

12. The method of claim 10, wherein for each facial action unit in the set of facial action units, calculating a confidence level associated with the facial action unit, based on the spectrum of oscillating electromyograph components, further comprises:

for each facial action unit in the set of facial action units, calculating the confidence level associated with the facial action unit, based on the spectrum of oscillating electromyograph components, the mouth position of the user, and an anatomical model.

13. The method of claim 9, further comprising:

during the sampling interval, recording a video of the lower face of the user;

identifying a mouth position of the user based on the video of the lower face of the user; and

outputting an identifier of the facial expression and a viseme value representing the mouth position of the user during the sampling interval.

14. The method of claim 9, wherein calculating a confidence level associated with the facial action unit further comprises:

accessing a previous facial expression of the user identified during a previous sampling interval;

identifying a set of temporally incoherent facial action units based on the previous facial expression, a time elapsed between the previous sampling interval and the sampling interval, and a facial motion model; and

removing, from the set of facial action units, the set of temporally incoherent facial action units.

15. A method for detecting a facial expression of a user comprises:

during a sampling interval, recording a set of electromyograph signals through a set of sense electrodes arranged about a viewing window in a virtual reality headset worn by a user;

deducting a reference signal from each electromyograph signal in the set of electromyograph signals to generate a set of composite signals;

for each composite signal in the set of composite signals, transforming the composite signal into a spectrum of oscillating electromyograph components within a frequency range of interest; and

for each facial action unit in a set of facial action units;

calculating a confidence level associated with the facial action unit, based on the spectrum of oscillating electromyograph components and an action unit model; and

in response to the confidence level associated with the facial action unit exceeding a confidence threshold, outputting an identifier associated with the facial action unit.

16. The method of claim 15, further comprising, during a calibration period prior to the sampling interval, the calibration period comprising a set of calibration intervals corresponding in number to a set of facial expressions:

during each calibration interval;

prompting the user to display a facial expression in the set of facial expressions;

recording a calibration set of electromyograph signals through the set of sense electrodes;

deducting a calibration reference signal from each electromyograph signal in the calibration set of electromyograph signals to generate a set of composite calibration signals;

for each composite calibration signal in the set of composite signals, transforming the composite calibration signal into a calibration spectrum of oscillating electromyograph components in a set of calibration spectra of oscillating electromyograph components within the frequency range of interest;

selecting the action unit based on the set of calibration spectra corresponding to the set of facial expressions.

17. The method of claim 15, further comprising, during a calibration period prior to the sampling interval, the calibration period comprising a set of calibration intervals corresponding in number to a set of facial expressions:

during each calibration interval;

displaying a media item in the viewing window of the virtual reality headset designed to induce the user to display a facial expression in the set of facial expressions;

recording a calibration set of electromyograph signals through the set of sense electrodes;

deducting a calibration reference signal from each electromyograph signal in the calibration set of electromyograph signals to generate a set of composite calibration signals;

for each composite calibration signal in the set of composite signals, transforming the composite calibration signal into a calibration spectrum of oscillating electromyograph components in a set of calibration spectra of oscillating electromyograph components within the frequency range of interest;

selecting the action unit model based on the set of calibration spectra corresponding to the set of facial expressions.

18. The method of claim 17, wherein selecting the action unit model further comprises:

performing a cluster analysis of the set of calibration spectra and previous sets of calibration spectra, each cluster in the cluster analysis corresponding to a profile of the user; and

selecting the action unit model corresponding to the profile of the user.

19. The method of claim 15, further comprising, based on the identifier associated with the facial action unit, updating a virtual face of an avatar within a virtual environment to display a virtual facial action unit corresponding to the facial action unit.

20. The method of claim 15, wherein calculating a confidence level associated with the facial action unit further comprises:

accessing a set of identifiers of previously detected facial action units within a time buffer; and

calculating the confidence level associated with the facial action unit, based on temporal coherence between the previously detected facial action units within the time buffer and the facial action unit, the spectrum of oscillating electromyograph components, and the action unit model.