System and method for activity recognition

• US 9,278,255 B2
• Filed: 12/20/2012
• Issued: 03/08/2016
• Est. Priority Date: 12/09/2012
• Status: Active Grant

• Current Assignee: Carnegie Mellon University, ARRIS Enterprises LLC (CommScope Holding Company, Inc.)
• Original Assignee: Carnegie Mellon University, ARRIS Enterprises LLC (CommScope Holding Company, Inc.)
• Inventors: Cheng, Heng-Tze, Davis, Paul C., Li, Jianguo, You, Di
• Primary Examiner(s): Mehta, Bhavesh
• Assistant Examiner(s): Mistry, Oneal R
• Application Number:
  US13/723,141
• Publication Number:
  US 20140161322A1
• Time in Patent Office: 1,174 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: A63B 24/00 Electric or electronic controls for exercising apparatus of preceding groups; Controlling or monitoring of exercises, sportive games, training or athletic performances
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 2218/00 Aspects of pattern recognition specially adapted for signal processing
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

A method for automatic recognition of human activity is provided and includes the steps of decomposing human activity into a plurality of fundamental component attributes needed to perform an activity and defining ontologies of fundamental component attributes from the plurality of the fundamental component attributes identified during the decomposing step for each of a plurality of different targeted activities. The method also includes the steps of converting a data stream captured during a performance of an activity performed by a human into a sequence of fundamental component attributes and classifying the performed activity as one of the plurality of different targeted activities based on a closest match of the sequence of fundamental component attributes obtained during the converting step to at least a part of one of the ontologies of fundamental component attributes defined during the defining step. A system for performing the method is also disclosed.

Claims

1. A method for automatic recognition of human activity, comprising the steps of:

decomposing a human activity into a plurality of fundamental component attributes needed to perform the human activity, wherein the human activity is included in a training set of activities;

defining ontologies of fundamental component attributes from the plurality of the fundamental component attributes identified during said decomposing step for each of a plurality of different targeted activities;

converting a data stream, the data stream captured during a performance by a human of a performed activity, into a sequence of fundamental component attributes; and

classifying the performed activity as one of the plurality of different targeted activities based on a closest match of the sequence of fundamental component attributes obtained during said converting step to at least a part of one of the ontologies of fundamental component attributes defined during said defining step, wherein the performed activity is not included in the training set of activities, the classifying comprising selecting only unseen classes in an attribute space;

wherein each of the fundamental component attributes is defined from a sequence of features, and further comprising the step of extracting features from the data stream with computations in at least one of time domain and frequency domain; and

wherein the data stream provides a time-sequence of features, and wherein, during said classifying step, a feature at each time slice of the data stream is compared to features of the fundamental component attributes at a corresponding time slice within the ontologies to determine a closest match.

2. The method according to claim 1, wherein the plurality of fundamental component attributes include attributes corresponding to at least one of a body motion and a body position.

3. The method according to claim 1, wherein the data stream is an electronic data stream including data captured by at least one motion sensor worn on the human during the performance of the performed activity.

4. The method according to claim 1, wherein the data stream is video data of the human during the performance of the performed activity.

5. The method according to claim 1, wherein said step of defining ontologies includes entry of a combination of fundamental component attributes in a database for each of the plurality of different targeted activities.

6. The method according to claim 1, wherein each of the different targeted activities includes a unique sequence and combination of fundamental component attributes.

7. A method of automatically recognizing a physical activity being performed by a human, comprising the steps of:

electronically decomposing training data obtained for each of a plurality of different physical activities within a training set of physical activities into a plurality of component attributes needed to perform the physical activities within the training set;

defining ontologies of component attributes from the plurality of component attributes identified during said decomposing step for each of a plurality of different physical activities within a targeted set of different physical activities, the targeted set being different from the training set, wherein the plurality of different physical activities within the targeted set includes at least one physical activity not included within the training set;

electronically capturing a data stream representing an actual physical activity performed;

electronically converting the data stream obtained during said capturing step into a plurality of component attributes; and

automatically classifying the actual physical activity being performed by comparing the plurality of component attributes obtained during said converting step to one of the ontologies of component attributes defined during said defining step;

wherein the automatically classifying further comprises classifying the actual physical activity for at least one class of physical activity wherein at least one of the component attributes has not been included within the training set, the classifying comprising selecting only unseen classes in an attribute space;

wherein each of the component attributes is defined from a sequence of features, and further comprising the step of extracting features from the data stream with computations in at least one of time domain and frequency domain; and

wherein the data stream provides a time-sequence of features, and wherein, during said classifying step, a feature at each time slice of the data stream is compared to features of the component attributes at a corresponding time slice within the ontologies to determine a closest match.

8. The method according to claim 7, wherein the plurality of different physical activities within the training set is a subset of the plurality of different physical activities within the targeted set.

9. The method according to claim 7, wherein the data stream includes data captured by at least one input source selected from a group consisting of a motion sensor, a video recorder, a sound recording device, a location sensor, a temperature sensor, a pressure sensor, an ambient light sensor, a heartrate sensor, and a proximity sensor.

10. The method according to claim 7, wherein the features are extracted by computations in at least one of time domain and frequency domain, and wherein the computations include at least one of mean, standard deviation, pair-wise correlation, cross correlation, slope, zero crossing rate, Fourier transform coefficients, and spectral flux.

11. The method according to claim 7, further comprising the steps of:

determining if there is a match of the features extracted from the data stream obtained during said capturing step to features produced by the training data for one of the plurality of different physical activities within the training set;

if there is a match to one of the plurality of different physical activities within the training set, a matched physical activity within the training set is recognized as the actual physical activity; and

if there is not a match to one of the plurality of different human physical activities within the training set, said converting and classifying steps are performed to predict a physical activity from the targeted set.

12. The method according to claim 7, wherein said step of defining ontologies includes entry of a combination of component attributes in a database for each physical activity without training data.

13. The method according to claim 7, wherein the component attributes include attributes corresponding to one of a body motion and body position, and wherein each of the different physical activities is composed of a unique sequence and combination of component attributes.

14. The method according to claim 7, wherein a sequence and combination of the different physical activities corresponds to a higher level activity, and further comprising the step of recognizing the higher level activity based on the sequence and combination of physical activities classified during said classifying step.

15. A system for automatically recognizing physical activity of a human, comprising:

a feature extractor configured to receive electronic input data captured by a sensor relative to a physical activity and to identify features from the input data; and

an attribute-based activity recognizer having;

an attribute detector for electronically determining an attribute as defined by a sequence of features, and

an activity classifier for classifying and outputting a prediction of the physical activity based on at least one of a sequence and combination of the attributes determined by the attribute detector;

wherein the activity recognizer has access to a database of a plurality of different attributes developed from training data for use in identifying attributes based on identified features, and is further configured to classify and output a prediction of the physical activity for at least one class of physical activity wherein at least one of the attributes determined by the attribute detector has not been included within the training data, the classifying comprising selecting only unseen classes in an attribute space;

wherein each of the attributes is defined from a sequence of features, and the activity recognizer is further configured to extract features from a data stream with computations in at least one of time domain and frequency domain; and

wherein the data stream provides a time-sequence of features, and wherein the activity recognizer is further configured to, during said classifying, compare a feature at each time slice of the data stream to features of the attributes at a corresponding time slice within the ontologies to determine a closest match.

16. A system according to claim 15, wherein at least some of the attributes define at least one of a body motion and a body position.

17. A system according to claim 16, wherein said feature extractor is configured to extract the features from the input data via computations in at least one of time domain and frequency domain, and wherein the computations include at least one of mean, standard deviation, pair-wise correlation, cross correlation, slope, zero crossing rate, Fourier transform coefficients, and spectral flux.

18. A system according to claim 16, wherein the physical activity is defined as a unique sequence and combination of attributes, and wherein the activity recognizer includes a database of a plurality of different physical activities defined as being composed of different sequences and combinations of attributes.

19. A system according to claim 16, further comprising a sensor for capturing data relative to the physical activity.