Systems, methods, apparatuses and devices for detecting facial expression and for tracking movement and location in at least one of a virtual and augmented reality system

• US 10,521,014 B2
• Filed: 01/30/2019
• Issued: 12/31/2019
• Est. Priority Date: 01/19/2017
• Status: Active Grant

• Current Assignee: Mindmaze Group SA
• Original Assignee: MindMaze Holding SA
• Inventors: Tadi, Tej, Leeb, Robert, Bourdaud, Nicolas, Bolomey, Leandre, Garipelli, Gangadhar, Mensi, Skander, Pilet, Julien, Ngo, Dat, Lebrun, Yann, Giroux, Arthur, Cardin, Sylvain, Merlini, Nicolas
• Primary Examiner(s): Sitta, Grant
• Application Number:
  US16/261,693
• Publication Number:
  US 20190155386A1
• Time in Patent Office: 335 Days
• CPC Class Codes: G06F 18/21326 involving optimisations, e.g. using regularisation techniques
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/24155 Bayesian classification
• CPC Class Codes: G06F 18/2453 non-linear, e.g. polynomial classifier
• CPC Class Codes: G06F 18/254 of classification results, e.g. of results related to same input data
• CPC Class Codes: G06F 2218/02 Preprocessing
• CPC Class Codes: G06F 2218/04 Denoising
• CPC Class Codes: G06F 2218/08 Feature extraction
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/15 Biometric patterns based on physiological signals, e.g. heartbeat, blood flow
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/176 Dynamic expression

Abstract

Systems, methods, apparatuses and devices for detecting facial expressions according to EMG signals for a virtual and/or augmented reality (VR/AR) environment, in combination with a system for simultaneous location and mapping (SLAM), are presented herein.

Claims

1. A facial expression determination system for determining a facial expression on a face of a user comprising:

an apparatus comprising a plurality of EMG (electromyography) electrodes configured for contact with the face of the user, said apparatus comprising an electrode interface;

a mask which contacts an upper portion of the face of the user, said mask including an electrode plate attached to at least eight EMG electrodes and one reference electrode such that said EMG electrodes contact said upper portion of the face of the user, wherein said electrode interface is operatively coupled to said EMG electrodes and a hardware processor, said electrode interface for providing said EMG signals from said EMG electrodes to said hardware processor; and

a computational device configured to receive a plurality of EMG signals from said EMG electrodes, and comprising said hardware processor and a memory having instructions thereon operable by said hardware processor to cause the computational device to;

receive said EMG signals;

process said EMG signals to form processed EMG signals and to determine at least one feature of said EMG signals in said processed EMG signals;

determine a roughness of said processed EMG signals according to a defined window, said determining a roughness comprising calculating an EMG-dipole and determining a movement of said processed EMG signals according to said EMG-dipole, and performing a nonlinear transformation of said processed EMG signals to enhance high-frequency contents of said processed EMG signals;

classify, using a classifier, a facial expression of said user according to said at least one feature of said EMG signals and according to said roughness.

2. The system of claim 1, wherein said classifying comprises determining whether the facial expression corresponds to a neutral expression or a non-neutral expression and, upon determining a non-neutral expression, determining said non-neutral expression.

3. The system of claim 1, wherein said defined window occurs within 100 MS.

4. The system of claim 1, wherein said processing said EMG signals to form processed EMG signals further comprises removing electrical power line interference (PLI).

5. The system of claim 1, wherein said classifier includes at least one of (1) a discriminant analysis classifier;

(2) a Riemannian geometry classifier;

(3) Nave Bayes classifier, (4) a k-nearest neighbor classifier, (5) a RBF (radial basis function) classifier, (6) a Bagging classifier, (7) a SVM (support vector machine) classifier, (8) a node classifier (NC), (9) NCS (neural classifier system), (10) SCRLDA (Shrunken Centroid Regularized Linear Discriminate and Analysis), or (11) a Random Forest classifier.

6. The system of claim 5, wherein said discriminant analysis classifier is one of (1) LDA (linear discriminant analysis), (2) QDA (quadratic discriminant analysis), or (3) sQDA.

7. The system of claim 5, wherein said classifier is one of (1) Riemannian geometry, (2) QDA and (3) sQDA.

8. The system of claim 1, wherein said memory has instructions thereon to further cause the computational device to train, using a training system, said classifier, said training system configured to:

receive a plurality of sets of processed EMG signals from a plurality of training users, each set including a plurality of groups of processed EMG signals from each training user, each group of processed EMG signals corresponding to a classified facial expression of said training user;

determine a pattern of variance for each of said groups of processed EMG signals across said plurality of training users corresponding to each classified facial expression; and

compare said processed EMG signals of the user to said patterns of variance to adjust said classification of the facial expression of the user.

9. The system of claim 8, wherein said memory has instructions thereon to further cause the computational device to:

retrain said classifier on said processed EMG signals of the user to form a retrained classifier;

classify said expression according to said processed EMG signals by said retrained classifier to determine the facial expression.

10. The system of claim 1, wherein said at least eight electrodes comprise eight unipolar EMG electrodes.

11. The system of claim 10, wherein said processing said EMG signals to form processed EMG signals further comprises removing common mode interference of said unipolar electrodes.

12. The system of claim 4, wherein removing PLI comprises filtering said EMG signals with two series of Butterworth notch filters of order 1, a first series of filter at 50 Hz and all its harmonics up to the Nyquist frequency, and a second series of filter with cutoff frequency at 60 Hz and all its harmonics up to the Nyquist frequency.

13. The system of claim 8, wherein said training system is further configured to:

compute a similarity score for said sets of said classified facial expressions of said users,

fuse each plurality of said classified facial expressions having said similarity score above a threshold indicating excessive similarity, so as to reduce a number of said classified facial expressions; and

train said classifier on said reduced number of said classified facial expressions.

14. The system of claim 11, wherein said memory has instructions thereon to further cause the computational device to:

normalize said EMG signals after said determining said roughness, said normalizing said EMG signals including calculating a log normal of said EMG signals, normalizing a variance for each electrode, and calculating covariance across a plurality of users;

train, using a training system before said classifying the facial expression, said classifier on a plurality of sets of processed EMG signals from a plurality of users, each set comprising a plurality of groups of processed EMG signals from each user, each group of processed EMG signals corresponding to a classified facial expression of said user, said training said classifier comprises determining a pattern of covariances for each of said groups of processed EMG signals across said plurality of users corresponding to each classified facial expression; and

wherein said classifying comprises comparing said normalized EMG signals of the user to said patterns of covariance to adjust said classification of the facial expression of the user.

15. The system of claim 14, wherein said memory has instructions thereon to further cause the computational device to:

determine a pattern of variance of each of said groups of processed EMG signals across said plurality of users corresponding to each classified facial expression, wherein said classifying comprises comparing said processed EMG signals of the user to said patterns of variance to classify the facial expression of the user.

16. The system of claim 15, wherein said training comprises:

assessing a similarity score for said classified facial expressions of said users; and

fusing together each plurality of said classified facial expressions having said similarity score above a threshold indicating excessive similarity, to form a reduced number of said classified facial expressions wherein said training said classifier comprises training on said reduced number of said classified facial expressions.

17. The system of claim 16, wherein said training further comprises determining a pattern of variance for each of said groups of processed EMG signals across said plurality of users corresponding to each classified facial expression;

and

wherein said classifying comprises comparing said processed EMG signals of the user to said patterns of variance to adjust said classification of the facial expression of the user.

18. The system of claim 14, wherein:

said memory has instructions thereon to further cause the computational device to determine a level of said facial expression according to a standard deviation of said roughness;

said at least one feature of said EMG signals further comprising said level of said facial expression, said roughness of said EMG-dipole being determined by determining a movement of said signals according to said EMG-dipole, said facial expression being determined at least partially according to said movement of said processed EMG signals.

19. The system of claim 18, further comprising a display for displaying an avatar, wherein a facial expression of said avatar is determined according to said classified facial expression from said classification.

20. The system of claim 19, wherein said memory has instructions thereon to further cause the computational device to:

predict, using a weight prediction module, a weight of said at least one feature; and

model, using an avatar modeler, said avatar according to a blend-shape;

wherein said blend-shape is determined according to said weight prediction.