Motion predictions of overlapping kinematic chains of a skeleton model used to control a computer system

• US 10,416,755 B1
• Filed: 06/01/2018
• Issued: 09/17/2019
• Est. Priority Date: 06/01/2018
• Status: Active Grant

• Current Assignee: Finchxr Limited
• Original Assignee: Finch Technologies Ltd.
• Inventors: Erivantcev, Viktor Vladimirovich, Lobanov, Alexander Sergeevich, Kartashov, Alexey Ivanovich, Goncharov, Daniil Olegovich
• Primary Examiner(s): Edun, Muhammad N
• Application Number:
  US15/996,389
• Time in Patent Office: 473 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

A system having sensor modules and a computing device. Each sensor module has an inertial measurement unit attached to a portion of a user to generate motion data identifying a sequence of orientations of the portion. The sensor modules include a first subset and a second subset that share a common sensor module. The computing device provides orientation measurements generated by the first subset as input to a first artificial neural network to obtain at least one first orientation measurement of the common module, provides orientation measurements generated by the second subset as input to a second artificial neural network to obtain at least one second orientation measurement of the common module, and generates, a predicted orientation measurement of the common module by combining the at least one first orientation measurement of the common module and the at least one second orientation measurement of the common module.

Claims

1. A system, comprising:

a plurality of sensor modules, each respective sensor module in the plurality of sensor modules having an inertial measurement unit and attached to a portion of a user to generate motion data identifying a sequence of orientations of the portion of the user, the plurality of sensor modules including a first subset and a second subset that share a common sensor module between the first subset and the second subset;

a computing device coupled to the plurality of sensor modules and configured to;

provide orientation measurements generated by the first subset as input to a first artificial neural network;

obtain, as output from the first artificial neural network, at least one first orientation measurement of a common part of the user on which the common sensor module is attached;

provide orientation measurements generated by the second subset as input to a second artificial neural network;

obtain, as output from the second artificial neural network, at least one second orientation measurement of the common part; and

generate a predicted orientation measurement of the common part from combining the at least one first orientation measurement of the common part and the at least one second orientation measurement of the common part.

2. The system of claim 1, wherein the first artificial neural network and the second artificial neural network contain long short-term memory (LSTM) units.

3. The system of claim 2, wherein the combining is performed using a third artificial neural network.

4. The system of claim 3, wherein the third artificial neural network contains a bidirectional long short-term memory (BLSTM) unit.

5. The system of claim 4, wherein the predicted orientation measurement of the common part is predicted for a first time instance;

the at least one first orientation measurement of the common part includes;

an orientation measurement, predicted by the first artificial neural network, of the common part at a second time instance before the first time instance; and

an orientation measurement, predicted by the first artificial neural network, of the common part at a third time instance after the first time instance; and

the at least one second orientation measurement of the common part includes;

an orientation measurement, predicted by the second artificial neural network, of the common part at the second time instance before the first time instance; and

an orientation measurement, predicted by the second artificial neural network, of the common part at the third time instance after the first time instance.

6. The system of claim 2, wherein the combining is performed via averaging the first orientation measurement of the common part and the second orientation measurement of the common part.

7. The system of claim 2, wherein the predicted orientation measurement of the common part is predicted to be measured using an optical tracking system.

8. The system of claim 7, wherein the first subset tracks a first kinematic chain of the user to control movements of a corresponding kinematic chain of a skeleton model in the system;

and the second subset tracks a second chain of the user to control movements of a corresponding kinematic chain of the skeleton model in the system.

9. The system of claim 8, wherein orientations of a first part in the first kinematic chain of the user is not tracked using an inertial measurement unit;

orientations of a second part in the second kinematic chain of the user is not tracked using an inertial measurement unit;

the first artificial neural network predicts orientations of the first part from the orientation measurements generated by the first subset; and

the second artificial neural network predicts orientations of the second part from the orientation measurements generated by the second subset.

10. The system of claim 1, wherein the inertial measurement unit includes a micro-electromechanical system (MEMS) gyroscope.

11. The system of claim 10, wherein the inertial measurement unit further includes a magnetometer and a MEMS accelerometer.

12. A method, comprising:

receiving, from a plurality of inertial measurement units attached to portions of a user connected by joints, motion data identifying sequences of orientations of the portions of the user, the plurality of inertial measurement units including a first subset and a second subset that share a common inertial measurement unit between the first subset and the second subset;

providing orientation measurements generated by the first subset as input to a first artificial neural network;

obtaining, as output from the first artificial neural network, at least one first orientation measurement of the common part of the user on which the common inertial measurement unit is attached;

providing orientation measurements generated by the second subset as input to a second artificial neural network;

obtaining, as output from the second artificial neural network, at least one second orientation measurement of the common part; and

generating, a predicted orientation measurement of the common part from combining the at least one first orientation measurement of the common part and the at least one second orientation measurement of the common part.

13. The method of claim 12, wherein the combining includes:

providing the at least one first orientation measurement of the common part and the at least one second orientation measurement of the common part as input to a third artificial neural network;

obtaining the predicted orientation measurement of the common part from as an output from the third artificial neural network.

14. The method of claim 13, wherein the first artificial neural network and the second artificial neural network include long short-term memory (LSTM) units;

and the third artificial neural network include a bidirectional long short-term memory (BLSTM) unit.

15. The method of claim 12, wherein the combining includes computing an average of the first orientation measurement of the common part and the second orientation measurement of the common part as the predicted orientation measurement of the common part.

16. A method, comprising:

attaching a plurality of sensor modules to a person, including a first subset of the sensor modules to track orientations of a first kinematic chain of the person and a second subset of the sensor modules to track orientations of a second kinematic chain of the person, wherein the first kinematic chain and the second kinematic chain have a common part of the person;

measuring, using the sensor modules, a plurality of sequences of orientations of the sensor modules during the person performing a plurality of sequence of motions;

measuring, independent of measurements of the sensor modules and using a separate tracking system, orientations of the sensor modules during the person performing a plurality of sequence of motions;

training a first artificial neural network using a supervised machine learning technique to predict orientation measurements of the first kinematic chain from the separate tracking system using orientation measurements from the first subset;

training a second artificial neural network using the supervised machine learning technique to predict orientation measurements of the second kinematic chain from the separate tracking system using orientation measurements from the second subset; and

training a third artificial neural network using the supervised machine learning technique to predict orientation measurements of the common part of the person from first predicted orientation measurements of the common part of the person generated from the first artificial neural network and second predicted orientation measurements of the common part of the person generated from the second artificial neural network.

17. The method of claim 16, wherein the first artificial neural network and the second artificial neural network include long short-term memory (LSTM) units;

and the third artificial neural network include a bidirectional long short-term memory (BLSTM) unit.

18. The method of claim 17, wherein to predict an orientation of the common part of the person at a first time instant, the third artificial neural network receives, as input,

an orientation of the common part of the person at a second time instant before the first time instant predicted by the first artificial neural network;

an orientation of the common part of the person at a third time instant after the first time instant predicted by the first artificial neural network;

an orientation of the common part of the person at the second time instant before the first time instant predicted by the second artificial neural network; and

an orientation of the common part of the person at the third time instant after the first time instant predicted by the second artificial neural network.

19. The method of claim 18, wherein the orientation of the common part of the person at the second time instant, predicted by the first artificial neural network, is updated using orientation measurements of the first kinematic chain at the first time instance from the first subset of the sensor modules;

and the orientation of the common part of the person at the second time instant, predicted by the first artificial neural network, is updated using orientation measurements of the second kinematic chain at the first time instance from the second subset of the sensor modules.

20. The method of claim 16, further comprising:

training the first artificial neural network and the second artificial neural network to reduce differences between;

the first predicted orientation measurements of the common part of the person generated from the first artificial neural network; and

the second predicted orientation measurements of the common part of the person generated from the second artificial neural network.