FUSION OF INERTIAL AND DEPTH SENSORS FOR MOVEMENT MEASUREMENTS AND RECOGNITION

• US 20160292497A1
• Filed: 04/06/2016
• Published: 10/06/2016
• Est. Priority Date: 04/06/2015
• Status: Active Grant

• Current Assignee: Texas A&M University System
• Original Assignee: Texas A&M University System
• Inventors: KEHTARNAVAZ, Nasser, JARARI, Roozbeh, LIU, Kui, CHEN, Chen, WU, Jian
• Granted Patent: US 10,432,842 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/24155 Bayesian classification
• CPC Class Codes: G06F 18/254 of classification results, e.g. of results related to same input data
• CPC Class Codes: G06F 18/295 Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/809 of classification results, e.g. where the classifiers operate on the same input data
• CPC Class Codes: G06V 10/85 Markov-related models; Markov random fields
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A movement recognition system includes an inertial sensor, a depth sensor, and a processor. The inertial sensor is coupled to an object and configured to measure a first unit of inertia of the object. The depth sensor is configured to measure a three dimensional shape of the object using projected light patterns and a camera. The processor is configured to receive a signal representative of the measured first unit of inertia from the inertial sensor and a signal representative of the measured shape from the depth sensor and to determine a type of movement of the object based on the measured first unit of inertia and the measured shape utilizing a classification model.

Claims

1. A movement recognition system, comprising:

an inertial sensor coupled to an object and configured to measure a first unit of inertia of the object;

a depth sensor configured to measure a three dimensional shape of the object using projected light patterns and a camera; and

a processor configured to receive a signal representative of the measured first unit of inertia from the inertial sensor and a signal representative of the measured shape from the depth sensor and to determine a type of movement of the object based on the measured first unit of inertia and the measured shape utilizing a classification model.

2. The movement recognition system of claim 1, wherein the processor is further configured to determine the type of movement of the object by:

training a plurality of Hidden Markov models (HMMs), each of the plurality of HMMs corresponding to a particular type of movement;

calculating a likelihood of probability for each of the plurality of trained HMMs based on the signal representative of the measured first unit of inertia and the signal representative of the measured shape; and

selecting the type of movement corresponding to the trained HMM having the highest likelihood of probability.

3. The movement recognition system of claim 2, wherein the processor is further configured to train each of the plurality of HMMs by:

initializing HMM parameters including an HMM probability and a transition matrix;

determining an observation sequence of the particular type of movement for the particular HMM being trained;

calculating a probability of the observation sequence; and

performing a Baum Welch reestimation of the probability of the observation sequence to update the HMM.

4. The movement recognition system of claim 1, wherein:

the inertial sensor is further configured to measure a second unit of inertia of the object; and

the processor is further configured to receive a signal representative of the measured second unit of inertia from the inertial sensor and to determine the type of movement of the object based on the measured second unit of inertia.

5. The movement recognition system of claim 4, wherein the processor is configured to determine the type of movement of the object by:

training;

a first plurality of Hidden Markov models (HMMs), each of the first plurality of HMMs corresponding to a particular type of movement for the measured first unit of inertia;

a second plurality of HMMs, each of the second plurality of HMMs corresponding to the particular type of movement for the measured second unit of inertia; and

a third plurality of HMMs, each of the third plurality of HMMs corresponding to the particular type of movement for the measured shape;

calculating;

a first likelihood of probability for each of the first plurality of HMMs based on the signal representative of the measured first unit of inertia;

a second likelihood of probability for each of the second plurality of HMMs based on the signal representative of the measured second unit of inertia; and

a third likelihood of probability for each of the third plurality of HMMs based on the signal representative of the measured shape;

pooling together the first, second, and third likelihood of probabilities to generate an overall probability for each of the first, second, and third pluralities of HMMs, and

selecting the type of movement corresponding to the trained HMM having the highest overall probability.

6. The movement recognition system of claim 5, wherein the processor is further configured to pool the first, second, and third likelihood of probabilities by:

multiplying the first likelihood of probability by a first weight to generate a weighted first likelihood of probability, the second likelihood of probability by a second weight to generate a weighted second likelihood of probability, and the third likelihood of probability by a third weight to generate a weighted third likelihood of probability; and

adding the weighted first, weighted second, and weighted third likelihood of probabilities.

7. The movement recognition system of claim 1, the processor is further configured to determine the type of movement of the object by:

extracting a depth feature set from the signal representative of the measured shape;

extracting a inertial feature set from the signal representative of the measured first unit of inertia; and

fusing the depth feature and the inertial feature at a decision-level.

8. The movement recognition system of claim 7, wherein the processor is further configured to extract the depth feature from the signal representative of the measured shape by:

extracting a foreground containing the object from the signal representative of the measured shape utilizing a background subtraction algorithm to generate a foreground extracted depth image;

generating three two dimensional projected maps corresponding to a front, view of the foreground extracted depth image; and

accumulating a difference between two consecutive projected maps through an entire depth video sequence to generate a depth motion map (DMM).

9. The movement recognition system of claim 7, wherein the processor is further configured to fuse the depth feature and the inertial feature at a decision-level by:

applying a sparse representation classifier (SRC) or collaborative representation classifier (CRC) to the extracted depth feature set and the extracted inertial feature set to generate a first and second basic probability assignments (BPAs) respectively;

combining the first and second BPAs, and

selecting the type of movement.

10. A method of recognizing movement comprising:

measuring, by an inertial sensor, a first unit of inertia of an object;

measuring a three dimensional shape of the object;

receiving, by a processor, a signal representative of the measured first unit of inertia from the inertial sensor and a signal representative of the measured shape from the depth sensor;

determining a type of movement of the object based on the measured first unit of inertia and the measured shape utilizing a classification model.

11. The method of claim 10, wherein the determining the type of movement comprises:

initializing Hidden Markov model (HMM) parameters, including an HMM probability and a transition matrix, for a plurality of HMMs, each of the plurality of HMMs corresponding to a particular type of movement;

determining an observation sequence of the particular type of movement for the particular HMM being trained;

calculating a probability of the observation sequence;

performing a Baum Welch reestimation of the probability of the observation sequence to update the HMM;

calculating a likelihood of probability for each of the plurality of trained HMMs based on the signal representative of the measured first unit of inertia and the signal representative of the measured shape; and

selecting the type of movement corresponding to the trained HMM having the highest likelihood of probability.

12. The method of claim 10, wherein the determining the type of movement comprises:

training a first, second, and third plurality of Hidden Markov models (HMMs), each of the first plurality of HMMs corresponding to a particular type of movement for the measured first unit of inertia, each of the second plurality of HMMs corresponding to the particular type of movement for a measured second unit of inertia, and each of the third plurality of HMMs corresponding to the particular type of movement for the measured shape;

calculating a first for each of the first plurality of HMMs based on the signal representative of the measured first unit of inertia, second likelihood of probability for each of the second plurality of HMMs based on the signal representative of the measured second unit of inertia, and third likelihood of probability for each of the third plurality of HMMs based on the signal representative of the measured shape;

pooling together the first, second, and third likelihood of probabilities to generate an overall probability for each of the first, second, and third pluralities of HMMs, and

selecting the type of movement corresponding to the trained HMM having the highest overall probability.

13. The method of claim 10, wherein the determining the type of movement comprises:

extracting a depth feature set from the signal representative of the measured shape;

extracting a inertial feature set from the signal representative of the measured first unit of inertia; and

fusing the depth feature and the inertial feature at a decision-level.

14. The method of claim 13, wherein the extracting the depth feature comprises:

extracting a foreground containing the object from the signal representative of the measured shape utilizing a background subtraction algorithm to generate a foreground extracted depth image;

generating three two dimensional projected maps corresponding to a front, view of the foreground extracted depth image; and

accumulating a difference between two consecutive projected maps through an entire depth video sequence to generate a depth motion map (DMM).

15. The method of claim 13, wherein the fusing the depth feature and the inertial feature comprises:

applying a sparse representation classifier (SRC) or collaborative representation classifier (CRC) to the extracted depth feature set and the extracted inertial feature set to generate a first and second basic probability assignments (BPAs) respectively;

combining the first and second BPAs, and

selecting the type of movement.

16. A non-transitory computer-readable medium storing instructions that when executed on a computing system cause the computing system to:

receive a signal representative of a measured first unit of inertia from an inertial sensor coupled to an object and a signal representative of a measured shape of the object from a depth sensor; and

determine a type of movement of the object based on the measured first unit of inertia and the measured shape utilizing a classification model.

17. The computer-readable medium of claim 16, wherein the instructions further cause the computing system to:

train a plurality of Hidden Markov models (HMMs), each of the plurality of HMMs corresponding to a particular type of movement;

calculate a likelihood of probability for each of the plurality of trained HMMs based on the signal representative of the measured first unit of inertia and the signal representative of the measured shape; and

select the type of movement corresponding to the trained HMM having the highest likelihood of probability.

18. The computer-readable medium of claim 16, wherein the instructions further cause the computing system to:

train a first, second, and third plurality of Hidden Markov models (HMMs), each of the first plurality of HMMs corresponding to a particular type of movement for the measured first unit of inertia, each of the second plurality of HMMs corresponding to the particular type of movement for a measured second unit of inertia, and each of the third plurality of HMMs corresponding to the particular type of movement for the measured shape;

calculate a first for each of the first plurality of HMMs based on the signal representative of the measured first unit of inertia, second likelihood of probability for each of the second plurality of HMMs based on the signal representative of the measured second unit of inertia, and third likelihood of probability for each of the third plurality of HMMs based on the signal representative of the measured shape;

pool together the first, second, and third likelihood of probabilities to generate an overall probability for each of the first, second, and third pluralities of HMMs, and

select the type of movement corresponding to the trained HMM having the highest overall probability.

19. The computer-readable medium of claim 16, wherein the instructions further cause the computing system to:

extract a depth feature set from the signal representative of the measured shape;

extract a inertial feature set from the signal representative of the measured first unit of inertia; and

fuse the depth feature and the inertial feature at a decision-level.

20. The computer-readable medium of claim 16, wherein the measured first unit of inertia comprises acceleration data of the object.