Motion predictions of overlapping kinematic chains of a skeleton model used to control a computer system

• US 10,635,166 B2
• Filed: 08/06/2019
• Issued: 04/28/2020
• Est. Priority Date: 06/01/2018
• Status: Active Grant

• Current Assignee: Finchxr Limited
• Original Assignee: Finch Technologies Ltd.
• Inventors: Erivantcev, Viktor Vladimirovich, Lobanov, Alexander Sergeevich, Kartashov, Alexey Ivanovich, Goncharov, Daniil Olegovich
• Primary Examiner(s): Edun, Muhammad N
• Application Number:
  US16/532,880
• Publication Number:
  US 20190369715A1
• Time in Patent Office: 266 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

A system having sensor modules and a computing device. Each sensor module has an inertial measurement unit attached to a portion of a user to generate motion data identifying a sequence of orientations of the portion. The sensor modules include a first subset and a second subset that share a common sensor module. The computing device provides orientation measurements generated by the first subset as input to a first artificial neural network to obtain at least one first orientation measurement of the common module, provides orientation measurements generated by the second subset as input to a second artificial neural network to obtain at least one second orientation measurement of the common module, and generates, a predicted orientation measurement of the common module by combining the at least one first orientation measurement of the common module and the at least one second orientation measurement of the common module.

Claims

1. A system, comprising:

multiple sensor modules configured to be attached to multiple portions of a user connected by joints, wherein the multiple sensor modules include a first subset and a second subset that share a common sensor module; and

a computing device coupled to the plurality of sensor modules and configured to;

calculate, based on orientation measurements generated by the first subset, a first sequence of orientation measurements of a portion of the user on which the common sensor module is attached;

calculate, based on orientation measurements generated by the second subset, a second sequence of orientation measurements of the portion of the user on which the common sensor module is attached;

provide, as input to an artificial neural network, the first sequence of orientation measurements of the portion of the user and the second sequence of orientation measurements of the portion of the user; and

generate, using the artificial neural network based on the first and second sequences of orientation measurements, a third sequence of orientation measurements of the portion of the user.

2. The system of claim 1, wherein each respective sensor module in the multiple sensor modules has a micro-electromechanical system (MEMS) gyroscope configured to measure an orientation of the respective sensor module.

3. The system of claim 2, wherein the artificial neural network includes at least one bidirectional long short-term memory (BLSTM) unit.

4. The system of claim 3, wherein the computing device is configured to use a first artificial neural network having at least one long short-term memory (LSTM) units to calculate the first sequence of orientation measurements of the portion of the user based on the orientation measurements generated by the first subset.

5. The system of claim 4, wherein the computing device is configured to use a second artificial neural network having at least one long short-term memory (LSTM) units to calculate the second sequence of orientation measurements of the portion of the user based on the orientation measurements generated by the second subset.

6. The system of claim 4, wherein the first artificial neural network is configured to predict orientation measurements generated by an optical tracking system based on the orientation measurements generated by the first subset.

7. The system of claim 6, wherein the computing device is further configured to control, based on predictions generated from the first artificial neural network, movements of a kinematic chain of a skeleton model corresponding to a first kinematic chain of the user tracked using the first subset.

8. The system of claim 7, wherein orientations of a first part in the first kinematic chain of the user is not tracked using an inertial measurement unit;

and the first artificial neural network is configured to predict orientations of the first part from the orientation measurements generated by the first subset.

9. The system of claim 2, wherein the artificial neural network averages the first and second sequences of orientation measurements to generate the third sequence of orientation measurements of the portion of the user.

10. A method, comprising:

receiving, from multiple inertial measurement units attached to multiple portions of a user connected by joints, orientation measurements of the portions of the user as a function of time, wherein the multiple inertial measurement units include a first subset and a second subset, and wherein the first subset and the second subset share a common inertial measurement unit;

determining, based on orientation measurements generated by the first subset, a first sequence of orientation measurements of a portion of the user on which the common inertial measurement unit is attached;

determining, based on orientation measurements generated by the second subset, a second sequence of orientation measurements of the portion of the user on which the common inertial measurement unit is attached;

providing the first sequence of orientation measurements of the portion of the user and the second sequence of orientation measurements of the portion of the user as input to an artificial neural network; and

generating, by the artificial neural network based on the first and second sequences of orientation measurements, a third sequence of orientation measurements of the portion of the user.

11. The method of claim 10, wherein the determining of the first sequence of orientation measurements of the portion of the user is by a first artificial neural network processing the orientation measurements generated by the first subset.

12. The method of claim 11, wherein the determining of the second sequence of orientation measurements of the portion of the user is by a second artificial neural network processing the orientation measurements generated by the second subset.

13. The method of claim 12, wherein the first artificial neural network and the second artificial neural network include long short-term memory (LSTM) units.

14. The method of claim 10, wherein the artificial neural network includes a bidirectional long short-term memory (BLSTM) unit.

15. A method, comprising:

attaching multiple sensor modules to a person, including a first subset of the sensor modules to track orientations of a first kinematic chain of the person and a second subset of the sensor modules to track orientations of a second kinematic chain of the person, wherein the first kinematic chain and the second kinematic chain have a common part of the person;

measuring, using the multiple sensor modules, sequences of orientations of the sensor modules during the person performing sequences of motions;

measuring, using a separate tracking system, orientations of at least the common part of the person during performance of the sequences of motions; and

training an artificial neural network to predict the orientations measurement of the common part of the person as measured by the separate tracking system, based on the sequences of the orientations of the sensor modules measured by the multiple sensor modules.

16. The method of claim 15, wherein the artificial neural network includes a first artificial neural network trained to predict first sequences of orientations of the common part of the person based on measurements generated by the first subset of the sensor modules.

17. The method of claim 16, wherein the artificial neural network further includes a second artificial neural network trained to predict second sequences of orientations of the common part of the person based on measurements generated by the second subset of the sensor modules.

18. The method of claim 17, wherein the artificial neural network further includes a third artificial neural network configured to predict the orientations measurement of the common part of the person as measured by the separate tracking system, based on the first and second sequences of orientations of the common part of the person that are predicted by the first and second artificial neural networks respectively.

19. The method of claim 18, wherein the first artificial neural network and the second artificial neural network include long short-term memory (LSTM) units;

and the third artificial neural network includes a bidirectional long short-term memory (BLSTM) unit.

20. The method of claim 16, further comprising:

training the first artificial neural network to reduce differences between;

first orientations of the first kinematic chain predicted by the first artificial neural network based on orientations measured by the first subset during performance of the sequences of motions; and

second orientations of the first kinematic chain measured by the separate tracking system during performance of the sequences of motions.