User-controlled tuning of handstate representation model parameters

• US 10,489,986 B2
• Filed: 01/25/2019
• Issued: 11/26/2019
• Est. Priority Date: 01/25/2018
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: CTRL-Labs Corp. (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Kaifosh, Patrick, Berenzweig, Adam, Al-natsheh, Adam
• Primary Examiner(s): Zalalee, Sultana M
• Application Number:
  US16/258,409
• Publication Number:
  US 20190228590A1
• Time in Patent Office: 305 Days
• Field of Search: None
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/2021 Shape modification

Abstract

Methods and apparatus for enabling a user or third-party to select or adjust parameters of one or more statistical models used to generate a musculoskeletal representation. The method comprises providing as input to the statistical model(s), a plurality of neuromuscular signals recorded by a plurality of neuromuscular sensors during performance of at least one gesture by a user, wherein the at least one gesture is performed by the user while wearing a wearable device having the plurality of neuromuscular sensors arranged thereon, rendering at least one visual representation based on an output of the statistical model(s), and receiving user or third-party input to adjust parameters of the statistical model(s) based on the rendered at least one visual representation, the user input including a selection of a particular statistical model of the statistical model(s) and/or an adjustment of parameters associated with the particular statistical model.

Claims

1. A computerized system configured to enable a person to select one or more statistical models used to generate a musculoskeletal representation, the one or more statistical models comprising a first statistical model and a second statistical model, the system comprising:

a user interface configured to instruct a user to perform at least one gesture while wearing a wearable device having a plurality of neuromuscular sensors arranged thereon; and

at least one computer processor programmed to;

provide as input to the first statistical model and the second statistical model, a plurality of neuromuscular signals recorded by the neuromuscular sensors during performance of the at least one gesture by the user;

generate a first musculoskeletal representation based at least in part by;

outputting from the first statistical model, first position estimates and/or first force estimates associated with the first musculoskeletal representation, and

updating the first musculoskeletal representation based on the first position estimates and/or the first force estimates;

generate a second musculoskeletal representation based at least in part by;

outputting from the second statistical model, second position estimates and/or second force estimates associated with the second musculoskeletal representation, and

updating the second musculoskeletal representation based on the second position estimates and/or the second force estimates;

render a first visual representation based on the first musculoskeletal representation and a second visual representation based on the second musculoskeletal representation;

repeatedly switch between displaying the rendered first visual representation and the rendered second visual representation; and

receive, subsequent to initiation of the displaying, input from the person comprising receiving a selection associated with the first visual representation or the second visual representation.

2. The computerized system of claim 1, wherein receiving input from the person comprises receiving input from the user.

3. The computerized system of claim 1, wherein receiving input from the person comprises receiving input from a third-party.

4. The computerized system of claim 1, wherein the at least one computer processor is further programmed to prompt the user to select a choice from each of a plurality of binary choices associated with different rendered visual representations, and

wherein receiving input from the person comprises receiving input associated with the plurality of binary choices.

5. The computerized system of claim 4, wherein the at least one computer processor is further programmed to:

aggregate the received input associated with the plurality of binary choices, and

adjust one or more parameters of the first statistical model or the second statistical model based on the aggregated input.

6. The computerized system of claim 1, wherein the selection associated with the first visual representation or the second visual representation is based on one or more anatomical characteristics of the user.

7. The computerized system of claim 1, wherein the at least one gesture comprises a discrete gesture.

8. The computerized system of claim 1, wherein receiving input from the person comprises receiving the input during display of the rendered first visual representation or second visual representation.

9. The computerized system of claim 1, wherein the at least one gesture comprises a continuous gesture, and wherein rendering the first visual representation or the second visual representation comprises generating at least one animation representing the at least one gesture, wherein the at least one animation comprises the first visual representation or the second visual representation rendered at a plurality of points in time.

10. The computerized system of claim 1, wherein the first musculoskeletal representation is a musculoskeletal representation associated with a hand, and

wherein the first position estimates and/or the first force estimates comprise handstate estimates.

11. The computerized system of claim 1, wherein the user interface is a graphical user interface configured to visually instruct the user to perform the at least one gesture.

12. The computerized system of claim 1, wherein the at least one computer processor is further programmed to display a likelihood or quality metric associated with the output from the first statistical model and/or the second statistical model.

13. A method for enabling a person to select one or more statistical models used to generate a musculoskeletal representation, the one or more statistical models comprising a first statistical model and a second statistical model, the method comprising:

providing as input to the first statistical model and the second statistical model, a plurality of neuromuscular signals recorded by neuromuscular sensors during performance of at least one gesture by a user;

generating a first musculoskeletal representation based at least in part by;

outputting from the first statistical model, first position estimates and/or first force estimates associated with the first musculoskeletal representation, and

updating the first musculoskeletal representation based on the first position estimates and/or the first force estimates;

generating a second musculoskeletal representation based at least in part by;

outputting from the second statistical model, second position estimates and/or second force estimates associated with the second musculoskeletal representation, and

updating the second musculoskeletal representation based on the second position estimates and/or the second force estimates;

rendering a first visual representation based on the first musculoskeletal representation and a second visual representation based on the second musculoskeletal representation;

repeatedly switching between displaying the rendered first visual representation and the rendered second visual representation; and

receiving, subsequent to initiation of the displaying, input from the person comprising receiving a selection associated with the first visual representation or the second visual representation.

14. The method of claim 13, wherein receiving input from the person comprises receiving input from the user.

15. The method of claim 13, wherein receiving input from the person comprises receiving input from a third-party.

16. The method of claim 13, wherein the first visual representation is rendered based on the first musculoskeletal representation that is dynamically updated in real-time.

17. A computerized system configured to enable a person to select one or more statistical models used to generate a musculoskeletal representation, the one or more statistical models comprising a first statistical model and a second statistical model, the system comprising:

a user interface configured to instruct a user to perform at least one gesture while wearing a wearable device having a plurality of neuromuscular sensors arranged thereon; and

at least one computer processor programmed to;

provide as input to the first statistical model and the second statistical model, a plurality of neuromuscular signals recorded by the neuromuscular sensors during performance of the at least one gesture by the user;

generate a first musculoskeletal representation based at least in part by;

outputting from the first statistical model, first position estimates and/or first force estimates associated with the first musculoskeletal representation, and

updating the first musculoskeletal representation based on the first position estimates and/or the first force estimates;

generate a second musculoskeletal representation based at least in part by;

outputting from the second statistical model, second position estimates and/or second force estimates associated with the second musculoskeletal representation, and

updating the second musculoskeletal representation based on the second position estimates and/or the second force estimates;

render a first visual representation based on the first musculoskeletal representation and a second visual representation based on the second musculoskeletal representation;

display the first visual representation and the second visual representation, at least in part, by overlaying the first visual representation with the second visual representation, wherein the first and second visual representations are rendered using different colors; and

receive input from the person comprising receiving a selection associated with the first visual representation or the second visual representation.

18. The computerized system of claim 17, wherein receiving input from the person comprises receiving input from the user.

19. The computerized system of claim 17, wherein receiving input from the person comprises receiving input from a third-party.

20. The computerized system of claim 17, wherein the at least one computer processor is further programmed to prompt the user to select a choice from each of a plurality of binary choices associated with different rendered visual representations, and

wherein receiving input from the person comprises receiving input associated with the plurality of binary choices.

21. The computerized system of claim 20, wherein the at least one computer processor is further programmed to:

aggregate the received input associated with the plurality of binary choices, and

adjust one or more parameters of the first statistical model or the second statistical model based on the aggregated input.

22. The computerized system of claim 17, wherein the selection associated with the first visual representation or the second visual representation is based on one or more anatomical characteristics of the user.

23. The computerized system of claim 17, wherein receiving input from the person comprises receiving the input during the display of the rendered first visual representation and second visual representation.

24. The computerized system of claim 17, wherein the at least one gesture comprises a discrete gesture.

25. The computerized system of claim 17, wherein the at least one gesture comprises a continuous gesture, and wherein rendering the first visual representation and the second visual representation comprises generating at least one animation representing the at least one gesture, wherein the at least one animation comprises the first visual representation or the second visual representation rendered at a plurality of points in time.

26. The computerized system of claim 17, wherein the first musculoskeletal representation is a musculoskeletal representation associated with a hand, and wherein the first position estimates and/or the first force estimates comprise handstate estimates.

27. The computerized system of claim 17, wherein the user interface is a graphical user interface configured to visually instruct the user to perform the at least one gesture.

28. A method for enabling a person to select one or more statistical models used to generate a musculoskeletal representation, the one or more statistical models comprising a first statistical model and a second statistical model, the method comprising:

providing as input to the first statistical model and the second statistical model, a plurality of neuromuscular signals recorded by neuromuscular sensors during performance of at least one gesture by a user;

generating a first musculoskeletal representation based at least in part by;

outputting from the first statistical model, first position estimates and/or first force estimates associated with the first musculoskeletal representation, and

updating the first musculoskeletal representation based on the first position estimates and/or the first force estimates;

generating a second musculoskeletal representation based at least in part by;

outputting from the second statistical model, second position estimates and/or second force estimates associated with the second musculoskeletal representation, and

updating the second musculoskeletal representation based on the second position estimates and/or the second force estimates;

rendering a first visual representation based on the first musculoskeletal representation and a second visual representation based on the second musculoskeletal representation;

display the first visual representation and the second visual representation, at least in part, by overlaying the first visual representation with the second visual representation, wherein the first and second visual representations are rendered using different colors; and

receive input from the person comprising receiving a selection associated with the first visual representation or the second visual representation.

29. The method of claim 28, wherein receiving input from the person comprises receiving input from the user.

30. The method of claim 28, wherein receiving input from the person comprises receiving input from a third-party.