System, method and apparatus for detecting facial expression in a virtual reality system

• US 10,515,474 B2
• Filed: 01/19/2018
• Issued: 12/24/2019
• Est. Priority Date: 01/19/2017
• Status: Active Grant

• Current Assignee: Mindmaze Group SA
• Original Assignee: MindMaze Holding SA
• Inventors: Tadi, Tej, Leeb, Robert, Bourdaud, Nicolas, Garipelli, Gangadhar, Mensi, Skander, Merlini, Nicolas, Lebrun, Yann, Giroux, Arthur, Cardin, Sylvain
• Primary Examiner(s): Chin, Michelle
• Application Number:
  US15/875,306
• Publication Number:
  US 20180240261A1
• Time in Patent Office: 704 Days
• Field of Search: None
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A system, method and apparatus for detecting facial expressions according to EMG signals for a VR environment.

Claims

1. A system for determining a facial expression on a face of a subject in a VR (virtual reality) environment, comprising

an apparatus comprising a plurality of EMG (electromyography) electrodes arranged so as to be in contact with at least a portion of the face of a subject when the apparatus is worn by the subject;

a computational device configured to receive a plurality of EMG signals from said EMG electrodes and configured with;

a first set of instructions operating thereon to cause the computational device to;

process said EMG signals to form processed EMG signals;

and

classifying a facial expression according to said processed EMG signals using a classifier according to at least one feature of data from said processed EMG signals; and

a second set of instructions operating thereon comprising a VR application for receiving said classified facial expression;

and

a display for displaying a VR environment according to the VR application.

2. The system of claim 1, wherein classifying comprises determining whether the facial expression corresponds to a neutral expression or a non-neutral expression.

3. The system of claim 1, wherein said VR application is configured to receive user input for one or more commands for operating and/or controlling said computational device and/or computer instructions operating thereon.

4. The system of claim 1, wherein said classifying said facial expression according to said processed EMG signals further comprises analyzing said processed EMG signals to determine a range of capabilities of muscles of the face of the subject;

and classifying said facial expression according to said range of capabilities.

5. The system of claim 1, wherein said feature of said data from said processed EMG signals comprises one or more of roughness, roughness of EMG dipole, or a direction of movement of said EMG signals of said EMG-dipole.

6. The system of claim 5, wherein said roughness comprises a filtered squared second derivative of said data.

7. A wearable apparatus for determining a facial expression on a face of a subject for a VR (virtual reality) environment, comprising:

a facemask including a plurality of EMG (electromyography) electrodes arranged thereon so as to be in contact with at least a portion of the face of the subject upon the subject wearing the facemask;

a computational device configured to receive a plurality of EMG signals from said EMG electrodes and configured with;

a first set of instructions operating thereon to cause the computational device to;

process said EMG signals to form processed EMG signals;

and

classifying a facial expression according to said processed EMG using a classifier; and

a second set of instructions operating thereon comprising a VR application for receiving said classified facial expression;

a display for displaying a VR environment to the user; and

a mount, wherein at least one of said facemask, said computational device and said display are physically connected to said mount or to one-another.

8. The apparatus of claim 7, wherein classifying comprises determining whether the facial expression corresponds to a neutral expression or a non-neutral expression.

9. The apparatus of claim 8, wherein said computational device comprises a hardware processor configured to perform a defined set of basic operations in response to receiving a corresponding basic instruction selected from a defined native instruction set of codes;

and memory;

said computational device comprising a first set of machine codes selected from the native instruction set for receiving said EMG data, a second set of machine codes selected from the native instruction set for processing said EMG data to determine at least one feature of said EMG data and a third set of machine codes selected from the native instruction set for determining a facial expression according to said at least one feature of said EMG data;

wherein each of the first, second and third sets of machine code is stored in the memory.

10. The apparatus of claim 7, wherein said classifying said facial expression from said processed EMG signals comprises classifying said facial expression according to at least one feature of data from said EMG signals.

11. The apparatus of claim 10, wherein said feature of said data from said processed EMG signals comprises one or more of roughness, roughness of EMG dipole, or a direction of movement of said EMG signals of said EMG-dipole.

12. The apparatus of claim 11, wherein said roughness comprises a filtered squared second derivative of said data.

13. A method for providing feedback to a user in a VR (virtual reality) environment, comprising:

providing the wearable device of claim 7;

receiving a plurality of EMG signals from said EMG electrodes;

determining the facial expression according to said processed EMG signals;

and

providing feedback to the user regarding the facial expression.

14. The method of claim 13, further comprising:

displaying the VR environment to the user, wherein the VR environment includes a system avatar;

displaying a facial expression by the system avatar;

analyzing a facial expression of the user, wherein the facial expression of the user corresponds to the facial expression displayed by the system avatar; and

providing feedback to the user regarding the correspondence of the facial expression of the user with the facial expression displayed by the system avatar.

15. The method of claim 14, further comprising:

displaying another avatar for representing a facial expression of the user; and

displaying said corresponding facial expression by the user by said other avatar.

16. The method of claim 14, further comprising:

displaying the VR environment to the user, wherein the VR environment includes a system avatar;

communicating between the user and said system avatar;

analyzing a communication style of the user, including a facial expression; and

providing feedback on said communication style.

17. A method for an interaction between a plurality of users of a VR system in a VR environment, comprising:

providing the wearable device of claim 7 for each of a plurality of users;

displaying the VR environment to said plurality of users;

receiving a plurality of EMG signals from said EMG electrodes for said plurality of users;

determining facial expressions according to said processed EMG signals for each of said plurality of users; and

displaying the facial expressions for each of said plurality of users through the VR environment.

18. The method of claim 17, further comprising performing a negotiation process between the plurality of users, wherein communication for said negotiation process occurs through the VR environment.

19. The method of claim 18, further comprising providing feedback to at least one user of the plurality of users regarding a state of said negotiation process according to the facial expressions.

20. The method of claim 17 comprising a plurality of other avatars, in addition to the user'"'"'s own avatar.

21. A method for calibration of facial expression recognition of a user, comprising:

providing the wearable device of claim 7;

receiving a plurality of EMG signals from said EMG electrodes;

determining the facial expression according to said processed EMG signals;

correlating the facial expression according to said processed EMG signals with a previously determined classification of the facial expression of the user; and

calibrating facial expression recognition of the user according to said correlation.