Calibration techniques for handstate representation modeling using neuromuscular signals

• US 10,496,168 B2
• Filed: 01/25/2019
• Issued: 12/03/2019
• Est. Priority Date: 01/25/2018
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: CTRL-Labs Corp. (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Kaifosh, Patrick, Giurgica-Tiron, Tudor, Berenzweig, Adam, Kober, Steven, Al-natsheh, Adam, Barachant, Alexandre, Wang, Zhuo
• Primary Examiner(s): He, Yingchun
• Application Number:
  US16/257,979
• Publication Number:
  US 20190227627A1
• Time in Patent Office: 312 Days
• Field of Search: None
• CPC Class Codes: A61B 2560/0238 Means for recording calibration data
• CPC Class Codes: A61B 5/1107 Measuring contraction of parts of the body, e.g. organ, muscle apparatus for measuring work or force in general G01L
• CPC Class Codes: A61B 5/1112 Global tracking of patients, e.g. by using GPS
• CPC Class Codes: A61B 5/1114 Tracking parts of the body
• CPC Class Codes: A61B 5/1122 of movement trajectories
• CPC Class Codes: A61B 5/1128 using image analysis A61B5/1127 takes precedence
• CPC Class Codes: A61B 5/296 for electromyography [EMG]
• CPC Class Codes: A61B 5/30 Input circuits therefor
• CPC Class Codes: A61B 5/316 Modalities, i.e. specific diagnostic methods
• CPC Class Codes: A61B 5/389 Electromyography [EMG]
• CPC Class Codes: A61B 5/6824 Arm or wrist
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors

Abstract

Methods and apparatus for calibrating performance of one or more statistical models used to generate a musculoskeletal representation. The method comprises controlling presentation of instructions via a user interface to instruct the user to perform the at least one gesture and updating at least one parameter of the one or more statistical models based, at least in part on a plurality of neuromuscular signals recorded by a plurality of neuromuscular sensors during performance of the at least one gesture by the user.

Claims

1. A computerized system configured to calibrate performance of one or more statistical models used to generate a musculoskeletal representation, the system comprising:

a plurality of neuromuscular sensors configured to continuously record a plurality of neuromuscular signals, wherein the plurality of neuromuscular sensors are arranged on at least one wearable device; and

at least one computer processor programmed to;

process at least some of the plurality of neuromuscular signals using a statistical model to generate based, at least in part, on joint angle estimates and/or force estimates output from the statistical model, the musculoskeletal representation;

determine based, at least in part, on at least one aspect of the musculoskeletal representation, that calibration of the statistical model used to generate the musculoskeletal representation is needed;

initiate a calibration session in response to determining that calibration is needed;

update a statistical model configuration based, at least in part, on a plurality of neuromuscular signals recorded by the plurality of neuromuscular sensors and ground-truth data representing position information and/or force information recorded during the calibration session to produce an updated statistical model;

process at least some of the plurality of neuromuscular signals using the updated statistical model to generate an updated musculoskeletal representation;

determine, based, at least in part, on the at least one aspect of the updated musculoskeletal representation whether further calibration of the statistical model is needed; and

end the calibration session in response to determining that further calibration is not needed.

2. The computerized system of claim 1, wherein updating the statistical model configuration comprises selecting a different statistical model to generate the updated musculoskeletal representation than the statistical model used to generate to the musculoskeletal representation.

3. The computerized system of claim 1, further comprising:

a user interface configured to instruct a user to perform at least one gesture during the configuration session,

wherein the at least one computer processor is further programmed to control presentation of instructions via the user interface to instruct the user to perform the at least one gesture, and

wherein updating the statistical model configuration is based, at least in part, on a plurality of neuromuscular signals recorded by the plurality of neuromuscular sensors during performance of the at least one gesture by the user.

4. The computerized system of claim 3, wherein the at least one computer processor is further programmed to update the statistical model configuration based, at least in part, on stored estimates of one or more of position information, force information, and neuromuscular information for the at least one gesture.

5. The computerized system of claim 3, wherein instructing the user to perform at least one gesture comprises instructing the user to perform a predetermined set of gestures.

6. The computerized system of claim 5 wherein controlling presentation of the instructions via the user interface comprises instructing the user to perform the gestures of the predetermined set in succession.

7. The computerized system of claim 1, wherein the at least one computer processor is further programmed to control at least one external device to capture the ground-truth data.

8. The computerized system of claim 7, wherein the at least one external device comprises an imaging device configured to capture at least one image, and wherein the ground-truth data is determined based, at least in part, on the at least one image.

9. The computerized system of claim 7, wherein the at least one external device comprises an inertial measurement unit.

10. The computerized system of claim 7, wherein the at least one external device comprises a plurality of external devices, each of which is configured to capture the position information or the force information of the ground-truth data.

11. The computerized system of claim 1, wherein updating the statistical model configuration comprises training the statistical model based, at least in part, on the plurality of neuromuscular signals recorded during the calibration session.

12. The computerized system of claim 1, wherein the statistical model is a trained statistical model, and wherein updating the statistical model configuration comprises adjusting one or more parameters provided as input to the trained statistical model without retraining the trained statistical model.

13. The computerized system of claim 1, wherein the at least one computer processor is further programmed to provide feedback to a user during the calibration session, wherein the feedback relates to the extent of a match between the updated musculoskeletal representation and the ground-truth data recorded during the calibration session.

14. The computerized system of claim 13, wherein providing feedback to the user comprises providing visual feedback.

15. The computerized system of claim 1, wherein determining that calibration is needed comprises identifying, based on the at least one aspect of the musculoskeletal representation, at least one gesture characteristic that the statistical model is poor at estimating.

16. The computerized system of claim 1, wherein the at least one computer processor is further programmed to render a visual representation associated with the hand based on the musculoskeletal representation and/or the updated musculoskeletal representation.

17. The computerized system of claim 16, wherein the visual representation is rendered during the calibration session.

18. The computerized system of claim 1, wherein the at least one computer processor is further programmed to:

continuously track a calibration of a performance of the statistical model during usage of the statistical model following training; and

initiate a calibration session when the calibration of the performance deviates from an acceptable calibration threshold by a particular amount.

19. The computerized system of claim 1, wherein updating the statistical model configuration comprises updating a plurality of model parameters, and wherein the at least one computer processor is further programmed to:

store a user profile that includes user-specific calibration parameters, wherein the user-specific calibration parameters includes a subset of the plurality of model parameters.

20. The computerized system of claim 19, wherein the user-specific calibration parameters include at least one joint stiffness parameter.

21. A method of calibrating performance of one or more statistical models used to generate a musculoskeletal representation, the method comprising:

recording a plurality of neuromuscular signals using a plurality of neuromuscular sensors arranged on at least one wearable device;

processing at least some of the plurality of neuromuscular signals using a statistical model to generate based, at least in part, on joint angle estimates and/or force estimates output from the statistical model, the musculoskeletal representation;

determining based, at least in part, on at least one aspect of the musculoskeletal representation, that calibration of the statistical model used to generate the musculoskeletal representation is needed;

initiating a calibration session in response to determining that calibration is needed;

updating a statistical model configuration based, at least in part on a plurality of neuromuscular signals recorded by the plurality of neuromuscular sensors and ground-truth data representing position information and/or force information recorded during the calibration session to produce an updated statistical model;

processing at least some of the plurality of neuromuscular signals using the updated statistical model to generate an updated musculoskeletal representation;

determining, based, at least in part, on the at least one aspect of the updated musculoskeletal representation whether further calibration of the statistical model is needed; and

ending the calibration session in response to determining that further calibration is not needed.

22. A computerized system configured to calibrate performance of one or more statistical models used to generate a musculoskeletal representation, the system comprising:

a user interface configured to instruct a user to perform at least one gesture while wearing a wearable device having a plurality of neuromuscular sensors arranged thereon; and

at least one computer processor programmed to;

control presentation of instructions via the user interface to instruct the user to perform the at least one gesture;

update at least one parameter of the one or more statistical models based, at least in part, on a plurality of neuromuscular signals recorded by the plurality of neuromuscular sensors during performance of the at least one gesture by the user;

identify, based on at least one aspect of the musculoskeletal representation generated based, at least in part, on joint angle estimates and/or force estimates output from the one or more statistical models, a plurality of gesture characteristics that the one or more statistical models is poor at estimating;

select a new gesture for the user to perform that includes the identified plurality of gesture characteristics;

control presentation of instructions via the user interface to instruct the user to perform the new gesture; and

update at least one parameter of the one or more statistical models based, at least in part, on the plurality of neuromuscular signals recorded by the neuromuscular sensors during performance of the new gesture by the user.

23. The computerized system of claim 22, wherein the at least one computer processor is further programmed to update at least one parameter of the one or more statistical models based, at least in part, on stored estimates of one or more of position information, force information, and neuromuscular information for the at least one gesture.

24. The computerized system of claim 22, wherein the at least one computer processor is further programmed to:

control at least one external device to capture at least one measurement of position and/or force information during performance of the at least one gesture; and

update the at least one parameter of the one or more statistical models based, at least in part, on the at least one measurement of position and/or force information.

25. The computerized system of claim 22, wherein updating the at least one parameter of the one or more statistical models comprises training the one or more statistical models based, at least in part, on the plurality of neuromuscular signals recorded by the neuromuscular sensors during performance of the at least one gesture by the user.

26. The computerized system of claim 22, wherein the one or more statistical models includes a trained statistical model, and wherein updating the at least one parameter of the one or more statistical models comprises adjusting one or more parameters provided as input to the trained statistical model without retraining the trained statistical model.

27. The computerized system of claim 22, wherein the at least one computer processor is further programmed to provide feedback to the user, wherein the feedback indicates to the user whether the performance of the at least one gesture has been performed as instructed.

28. A method of calibrating performance of one or more statistical models used to generate a musculoskeletal representation, the method comprising:

instructing, via a user interface, a user to perform at least one gesture while wearing a wearable device having a plurality of neuromuscular sensors arranged thereon;

controlling presentation of instructions via the user interface to instruct the user to perform the at least one gesture;

updating at least one parameter of the one or more statistical models based, at least in part, on a plurality of neuromuscular signals recorded by the plurality of neuromuscular sensors during performance of the at least one gesture by the user;

identifying, based on at least one aspect of the musculoskeletal representation generated based, at least in part, on joint angle estimates and/or force estimates output from the one or more statistical models, a plurality of gesture characteristics that the one or more statistical models is poor at estimating;

selecting a new gesture for the user to perform that includes the identified plurality of gesture characteristics;

controlling presentation of instructions via the user interface to instruct the user to perform the new gesture; and

updating at least one parameter of the one or more statistical models based, at least in part, on the plurality of neuromuscular signals recorded by the neuromuscular sensors during performance of the new gesture by the user.