Techniques for anonymizing neuromuscular signal data

• US 10,504,286 B2
• Filed: 01/25/2019
• Issued: 12/10/2019
• Est. Priority Date: 01/25/2018
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: CTRL-Labs Corp. (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Kaifosh, Patrick, Giurgica-Tiron, Tudor, Kober, Steven, Al-natsheh, Adam, Barachant, Alexandre
• Primary Examiner(s): Nguyen, Phu K
• Application Number:
  US16/258,442
• Publication Number:
  US 20190228579A1
• Time in Patent Office: 319 Days
• Field of Search: 345518
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics

Abstract

Methods and apparatus for anonymizing neuromuscular signals used to generate a musculoskeletal representation. The method comprises recording, using a plurality of neuromuscular sensors arranged on one or more wearable devices, a plurality of neuromuscular signals from a user, providing as input to a trained statistical model, the plurality of neuromuscular signals and/or information based on the plurality of neuromuscular signals; and generating, the musculoskeletal representation based, at least in part, on an output of the trained statistical model, wherein the musculoskeletal representation is an anonymized musculoskeletal representation from which at least one personal characteristic of the user has been removed.

Claims

1. A computerized system for anonymizing neuromuscular signals used to generate a musculoskeletal representation, the system comprising:

a plurality of neuromuscular sensors configured to continuously record a plurality of neuromuscular signals from a user, wherein the plurality of neuromuscular sensors are arranged on one or more wearable devices; and

at least one computer processor programmed to;

provide as input to a statistical model, the plurality of neuromuscular signals and/or information based on the plurality of neuromuscular signals, wherein;

the plurality of neuromuscular signals and/or information based on the plurality of neuromuscular signals provided as input to the statistical model includes a first personal characteristic of the user, and

the statistical model is trained to remove the first personal characteristic of the user from the plurality of neuromuscular signals and/or information based on the plurality of neuromuscular signals; and

generate the musculoskeletal representation based, at least in part, on position estimates and/or force estimates output from the trained statistical model, wherein;

the musculoskeletal representation is an anonymized musculoskeletal representation from which the first personal characteristic of the user has been removed, and

the statistical model has been trained using an adversarial training approach to remove the first personal characteristic of the user while predicting the position estimates and/or the force estimates associated with the musculoskeletal representation.

2. The computerized system of claim 1, wherein the at least one computer processor is further programmed to process the plurality of neuromuscular signals and/or the information based on the plurality of neuromuscular signals to remove a second personal characteristic of the user.

3. The computerized system of claim 2, wherein the second personal characteristic of the user is removed from the plurality of neuromuscular signals and/or the information based on the plurality of neuromuscular signals prior to providing the plurality of neuromuscular signals and/or the information based on the plurality of neuromuscular signals as input to the statistical model.

4. The computerized system of claim 2, wherein the at least one computer processor is further programmed to:

store the plurality of neuromuscular signals and/or the information based on the plurality of neuromuscular signals in a storage device,

wherein the second personal characteristic of the user is removed from the plurality of neuromuscular signals and/or the information based on the plurality of neuromuscular signals prior to storage of the plurality of neuromuscular signals and/or the information based on the plurality of neuromuscular signals in the storage device.

5. The computerized system of claim 2, wherein the second personal characteristic of the user comprises at least one movement pattern associated with a neurological disorder or a non-pathological idiosyncrasy, and the at least one computer processor is further programmed to process the plurality of neuromuscular signals and/or the information based on the plurality of neuromuscular signals to:

identify the at least one movement pattern in the plurality of neuromuscular signals and/or the information based on the plurality of neuromuscular signals; and

remove the identified at least one movement pattern from the plurality of neuromuscular signals and/or the information based on the plurality of neuromuscular signals.

6. The computerized system of claim 1, wherein the first personal characteristic of the user relates to at least one of muscle fatigue, muscle activity level, muscle frequency content, and muscle spiking pattern.

7. The computerized system of claim 1, wherein the first personal characteristic of the user relates to a body mass index or body fat percentage of the user.

8. The computerized system of claim 1, wherein the first personal characteristic of the user comprises at least one movement pattern associated with a neurological disorder or a non-pathological idiosyncrasy.

9. The computerized system of claim 1, wherein the adversarial training approach comprises using an adversarial network that is configured to concurrently remove the first personal characteristic of the user and predict the position estimates and/or the force estimates associated with at least one segment of the musculoskeletal representation.

10. The computerized system of claim 1, wherein the at least one computer processor is further programmed to process the position estimates and/or the force estimates output from the statistical model to remove a second personal characteristic of the user.

11. The computerized system of claim 10, wherein processing the position estimates and/or the force estimates output from the statistical model comprises averaging the position estimates and/or the force estimates output from the statistical model over time to generate a smoothed output of the statistical model.

12. The computerized system of claim 10, wherein processing the position estimates and/or the force estimates output from the statistical model comprises mapping the position estimates and/or the force estimates output from the statistical model to at least one state of a plurality of discrete states, wherein each state of the plurality of discrete states represents a generic movement pattern.

13. The computerized system of claim 1, wherein the at least one computer processor is further programmed to:

render a visual representation based on the musculoskeletal representation, wherein a second personal characteristic of the user is removed during rendering of the visual representation.

14. A computerized system for anonymizing neuromuscular signals used to generate a musculoskeletal representation, the system comprising:

a plurality of neuromuscular sensors configured to continuously record a plurality of neuromuscular signals from a user, wherein the plurality of neuromuscular sensors are arranged on one or more wearable devices; and

at least one computer processor programmed to;

provide as input to a trained statistical model, the plurality of neuromuscular signals and/or information based on the plurality of neuromuscular signals;

generate the musculoskeletal representation based, at least in part, on an output of the trained statistical model, wherein the output of the trained statistical model comprises;

position estimates and/or force estimates associated with the musculoskeletal representation, wherein the position estimates and/or the force estimates include at least one personal characteristic of the user; and

render a visual representation based on the musculoskeletal representation, wherein;

the musculoskeletal representation includes the at least one personal characteristic of the user, and

rendering the visual representation comprises removing the at least one personal characteristic of the user during rendering of the visual representation.

15. The computerized system of claim 14, wherein the at least one personal characteristic of the user is removed by altering a dimension of one or more fingers of a hand during rendering of the visual representation of the hand.

16. The computerized system of claim 14, wherein the at least one personal characteristic of the user is removed by mapping the visual representation to at least one state of a plurality of discrete states, wherein each state of the plurality of discrete states represents a generic visual representation.

17. The computerized system of claim 14, wherein the at least one computer processor is further programmed to:

receive user input indicating whether to render an anonymized visual representation.

18. The computerized system of claim 17, wherein the at least one computer processor is further programmed to:

in response to user input indicating that the anonymized visual representation is to be rendered, render the visual representation with the at least one personal characteristic removed from the visual representation.

19. A method for anonymizing neuromuscular signals used to generate a musculoskeletal representation, the method comprising:

recording, using a plurality of neuromuscular sensors arranged on one or more wearable devices, a plurality of neuromuscular signals from a user;

providing as input to a statistical model, the plurality of neuromuscular signals and/or information based on the plurality of neuromuscular signals, wherein;

the plurality of neuromuscular signals and/or information based on the plurality of neuromuscular signals provided as input to the statistical model includes at least one personal characteristic of the user, and

the statistical model is trained to remove the at least one personal characteristic of the user from the plurality of neuromuscular signals and/or information based on the plurality of neuromuscular signals; and

generating, the musculoskeletal representation based, at least in part, on position estimates and/or force estimates output from the trained statistical model, wherein;

the musculoskeletal representation is an anonymized musculoskeletal representation from which the at least one personal characteristic of the user has been removed, and

the statistical model has been trained using an adversarial training approach to remove the at least one personal characteristic of the user while predicting the position estimates and/or the force estimates associated with the musculoskeletal representation.

20. The method of claim 19, wherein the adversarial training approach comprises using an adversarial network that is configured to concurrently remove the at least one personal characteristic of the user and predict the position estimates and/or the force estimates associated with at least one segment of the musculoskeletal representation.

21. A method for anonymizing neuromuscular signals used to generate a musculoskeletal representation, the method comprising:

providing as input to a trained statistical model, a plurality of neuromuscular signals recorded from a user by a plurality of neuromuscular sensors and/or information based on the plurality of neuromuscular signals;

generating the musculoskeletal representation based, at least in part, on an output of the trained statistical model, wherein the output of the trained statistical model comprises;

position estimates and/or force estimates associated with the musculoskeletal representation, wherein the position estimates and/or the force estimates include at least one personal characteristic of the user; and

rendering a visual representation based on the musculoskeletal representation, wherein;

the musculoskeletal representation includes the at least one personal characteristic of the user, and

rendering the visual representation comprises removing the at least one personal characteristic of the user during rendering of the visual representation.

22. The method of claim 21, wherein the at least one personal characteristic of the user is removed by altering a dimension of one or more fingers of a hand during rendering of the visual representation of the hand.

23. The method of claim 21, wherein the at least one personal characteristic of the user is removed by mapping the visual representation to at least one state of a plurality of discrete states, wherein each state of the plurality of discrete states represents a generic visual representation.

24. The method of claim 21, wherein the method further comprises:

receiving user input indicating whether to render an anonymized visual representation.

25. The method of claim 24, wherein the method further comprises:

in response to user input indicating that the anonymized visual representation is to be rendered, rendering the visual representation with the at least one personal characteristic removed from the visual representation.