Markerless motion capture using machine learning and training with biomechanical data

• US 10,445,930 B1
• Filed: 05/17/2018
• Issued: 10/15/2019
• Est. Priority Date: 05/17/2018
• Status: Active Grant

• Current Assignee: Southwest Research Institute
• Original Assignee: Southwest Research Institute
• Inventors: Saylor, Kase J., Nicolella, Daniel P., Chambers, David R., Eliason, Travis D., Poole, Donald R.
• Primary Examiner(s): Wu, Yanna
• Application Number:
  US15/982,691
• Time in Patent Office: 516 Days
• Field of Search: None
• CPC Class Codes: A61B 5/1038 Measuring plantar pressure during gait
• CPC Class Codes: A61B 5/1128 using image analysis A61B5/1127 takes precedence
• CPC Class Codes: A61B 5/389 Electromyography [EMG]
• CPC Class Codes: A61B 5/7267 involving training the classification device
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 10/774 Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 40/23 : Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G16H 50/70 : for mining of medical data, e.g. analysing previous cases of other patients

Abstract

A method of using a learning machine to provide a biomechanical data representation of a subject based on markerless video motion capture. The learning machine is trained with both markerless video and marker-based (or other worn body sensor) data, with the marker-based or body worn sensor data being used to generate a full biomechanical model, which is the “ground truth” data. This ground truth data is combined with the markerless video data to generate a training dataset.

Claims

1. A method of training a learning machine to receive video data captured from an animate subject, and from the video data to generate biomechanical states of the animate subject, comprising:

placing markers on the animate subject;

using both marker-based motion capture camera(s) and markerless motion capture camera(s) to simultaneously acquire video sequences of the animate subject, thereby acquiring marker-based video data and markerless video data;

wherein the marker-based camera(s) detect the markers on the animate subject in a manner differently from detection of the rest of the animate subject;

fitting the marker-based video data to a kinematic model of the animate subject, thereby providing a ground truth dataset;

combining the ground truth dataset with the markerless video data, thereby providing a training dataset;

inputting the markerless video data to the learning machine;

comparing the output of the learning machine to the training dataset;

iteratively using the results of the comparing step to adjust operation of the learning machine; and

using the learning machine to generate at least one of the biomechanical states of the animate subject.

2. The method of claim 1, wherein the combining step is performed by calibrating the marker-based motion capture camera(s) and markerless motion capture camera(s) and using the results of the calibrating to generate the ground truth dataset.

3. The method of claim 1, wherein the placing, using, fitting, and combining steps are performed for multiple animate subjects.

4. The method of claim 1, wherein multiple animate subjects perform different activities.

5. The method of claim 1, further comprising the step of installing one or more biomechanical sensors on or near the animate subject, and wherein the output of the one or more biomechanical sensors is used to generate the kinematic model.

6. The method of claim 5, wherein the one or more biomechanical sensors are one or more of the following:

force plate, electromyographic sensor, accelerometer, or gyroscope.

7. A method of training a learning machine to receive video data captured from an animate subject, and from the video data to generate biomechanical states of the animate subject, comprising:

placing one or more biomechanical sensors on the animate subject;

using both sensor detector and markerless motion capture camera(s) to simultaneously acquire video sequences of the animate subject, thereby acquiring sensor detector data and markerless video data;

wherein the sensor detector data is data that acquired by detecting the one or more biomechanical sensors as the animate subject moves;

fitting the sensor detector data to a kinematic model of the animate subject, thereby providing a ground truth dataset;

combining the ground truth dataset with the markerless video data, thereby providing a training dataset;

inputting the markerless video data to the learning machine;

comparing the output of the learning machine to the training dataset; and

iteratively using the results of the comparing step to adjust operation of the learning machine; and

using the learning machine to generate at least one of the biomechanical states of the animate subject.

8. The method of claim 7, wherein the one or more biomechanical sensors are one or more of the following:

force plate, electromyographic sensor, accelerometer, or gyroscope.