Real-time processing of handstate representation model estimates

• US 10,460,455 B2
• Filed: 01/25/2019
• Issued: 10/29/2019
• Est. Priority Date: 01/25/2018
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: CTRL-Labs Corp. (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Giurgica-Tiron, Tudor, Al-natsheh, Adam, Danielson, Nathan
• Primary Examiner(s): Osifade, Idowu O
• Application Number:
  US16/258,232
• Publication Number:
  US 20190228533A1
• Time in Patent Office: 277 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/251 involving models

Abstract

System and methods are provided for providing a dynamically-updated musculoskeletal representation of a hand. The system includes a plurality of neuromuscular sensors configured to continuously record a plurality of neuromuscular signals from a user, and at least one computer processor programmed to provide as input to a trained statistical model, the plurality of neuromuscular signals and temporally smooth in real-time an output of the trained statistical model. The system is also programmed to determine, based on the smoothed output of the trained statistical model, position information describing a spatial relationship between two or more connected segments of the musculoskeletal representation, force information describing a force exerted by at least one segment of the musculoskeletal representation, and update the musculoskeletal representation of the hand based, at least in part, on the position information and the force information.

Claims

1. A computerized system for providing a dynamically-updated musculoskeletal representation of a hand, the system comprising:

a plurality of neuromuscular sensors configured to continuously record a plurality of neuromuscular signals from a user; and

at least one computer processor programmed to;

provide as input to a trained statistical model, the plurality of neuromuscular signals;

temporally smooth in real-time an output of the trained statistical model;

determine, based on the smoothed output of the trained statistical model;

position information describing a spatial relationship between two or more connected segments of the musculoskeletal representation;

force information describing a force exerted by at least one segment of the musculoskeletal representation; and

update the musculoskeletal representation of the hand based, at least in part, on the position information and the force information.

2. The computerized system of claim 1, wherein temporally smoothing the output of the trained statistical model comprises processing the output of the trained statistical model using at least one filter.

3. The computerized system of claim 2, wherein the at least one filter comprises at least one filter from a group comprising an exponential filter, a Kalman filter, a non-linear Kalman filter, a particle filter and a Bayesian filter.

4. The computerized system of claim 1, wherein the at least one computer processor is programmed to determine an accuracy of the trained statistical model, and temporally smooth, responsive to the determined accuracy of the trained statistical model, the output of the trained statistical model in real-time.

5. The computerized system of claim 4, wherein the at least one computer processor is programmed to determine at least one of an input accuracy and an output accuracy of the trained model.

6. The computerized system of claim 1, wherein temporally smoothing the output of the trained statistical model comprises including in the trained statistical model at least one temporal smoothing characteristic.

7. The computerized system of claim 6, wherein the trained statistical model that includes at least one temporal smoothing characteristic comprises training the statistical model with a penalization term to promote smoothness in the model outputs.

8. The computerized system of claim 6, wherein the trained statistical model that includes at least one temporal smoothing characteristic comprises training the statistical model with an error function that includes an error in estimates of time derivatives of the output of the statistical model.

9. The computerized system of claim 6, wherein the trained statistical model that includes at least one temporal smoothing characteristic comprises including in the trained statistical model a prior on the smoothness of the output of the statistical model.

10. The computerized system of claim 9, wherein temporally smoothing the output of the trained statistical model further comprises generating the output of the trained statistical model for a first timepoint based on first neuromuscular signals recorded at the first timepoint and second neuromuscular signals recorded at at least one second timepoint after the first timepoint.

11. The computerized system of claim 1, wherein the at least one computer processor is further programmed to:

determine a quality of the smoothed output of the trained statistical model at a first timepoint; and

adjust at least one smoothing parameter based on the determined quality; and

temporally smooth output of the trained statistical model at a second timepoint based on the at least one adjusted smoothing parameter.

12. The computerized system of claim 1, wherein the at least one computer processor is further programmed to temporally smooth the output of the trained statistical model based, at least in part, on at least one characteristic of a motion of the user during recording of the plurality of neuromuscular signals.

13. The computerized system of claim 12, further comprising:

at least one inertial measurement unit configured to continuously record a plurality of inertial measurement unit signals, wherein the at least one characteristic of a motion of the user is determined based, at least in part, on the plurality of inertial measurement unit signals.

14. The computerized system of claim 13, wherein the at least one characteristic of a motion of the user comprises a speed at which a user is moving a hand and/or arm of the user.

15. The computerized system of claim 14, wherein temporally smoothing the output of the trained statistical model comprises increasing an amount of temporal smoothing of the output of the trained statistical model when it is determined that the user is moving a hand and/or arm of the user at a speed greater than a threshold value.

16. The computerized system of claim 1, wherein the at least one computer processor is further programmed to update a loss function of the trained statistical model based, at least in part, on the smoothed output of the trained statistical model.

17. The computerized system of claim 1, wherein the at least one computer processor is further programmed to:

execute an application that provides a virtual reality environment;

render, based on the updated musculoskeletal model of the hand, a visual representation of a hand within the virtual reality environment; and

modulate an amount of temporal smoothing of the output of the trained statistical model based, at least in part, on a spatial relationship between the visual representation of the hand and one or more objects within the virtual reality environment.

18. The computerized system of claim 17, wherein modulating an amount of temporal smoothing of the output of the trained statistical model comprises at least one of decreasing and increasing the amount of temporal smoothing when the visual representation of the hand approaches an object in the virtual reality environment.

19. The computerized system according to claim 18, wherein the amount of temporal smoothing is either increased or decreased in a selected degree of freedom.

20. A method for providing a dynamically-updated musculoskeletal representation of a hand, the method comprising:

receiving, from a plurality of neuromuscular sensors, a plurality of neuromuscular signals from a user;

executing a trained statistical model representing handstate;

providing, as input to the executed statistical model, the plurality of neuromuscular signals;

temporally smoothing in real-time an output of the trained statistical model;

determining, based on the smoothed output of the trained statistical model;

position information describing a spatial relationship between two or more connected segments of the musculoskeletal representation; and

force information describing a force exerted by at least one segment of the musculoskeletal representation; and

updating the musculoskeletal representation of the hand based, at least in part, on the position information and the force information.

21. A non-transitory computer-readable medium encoded with a plurality of instructions that, when executed by at least one computer processor, performs a method for providing a dynamically-updated musculoskeletal representation of a hand, the method comprising:

receiving, from a plurality of neuromuscular sensors, a plurality of neuromuscular signals from a user;

executing a trained statistical model representing handstate;

providing, as input to the executed statistical model, the plurality of neuromuscular signals;

temporally smoothing in real-time an output of the trained statistical model;

determining, based on the smoothed output of the trained statistical model;

position information describing a spatial relationship between two or more connected segments of the musculoskeletal representation; and

force information describing a force exerted by at least one segment of the musculoskeletal representation; and

updating the musculoskeletal representation of the hand based, at least in part, on the position information and the force information.