Method and apparatus for recognizing motion feature of user, using orthogonal semisupervised non-negative matrix factorization (OSSNMF)-based feature data

• US 10,337,868 B2
• Filed: 01/21/2014
• Issued: 07/02/2019
• Est. Priority Date: 01/18/2013
• Status: Active Grant

• Current Assignee: Postech Academy-Industry Foundation, Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Sung, Jae-Mo, Kong, Won-Keun, Kim, Sae-Hun, Choi, Seung-Jin
• Primary Examiner(s): Bloss, Stephanie E
• Application Number:
  US14/159,790
• Publication Number:
  US 20140207401A1
• Time in Patent Office: 1,988 Days
• CPC Class Codes: G01C 21/16 by integrating acceleration or speed, i.e. inertial navigation
• CPC Class Codes: G01C 25/005 initial alignment, calibration or starting-up of inertial devices
• CPC Class Codes: G06F 17/16 Matrix or vector computation , e.g. matrix-matrix or matrix-vector multiplication, matrix factorization matrix transposition G06F7/78

Abstract

An apparatus for recognizing a user'"'"'s motions based on sensor information and label information, a method for establishing an ONMF-based basis matrix, and a method for establishing an OSSNMF-based basis matrix are provided, where the basis matrices are used to extract motion features of the user. The apparatus for recognizing the user'"'"'s motions may include a feature vector extractor configured to multiply a transposed matrix of an orthogonalized basis matrix by a sensor data matrix of frequency domain sensor data acquired from sensors to extract an ONMF-based feature vector and a multi-class classifier configured to use the extracted ONMF-based feature vector to classify the user'"'"'s motion into a type from among types of a user'"'"'s motions.

Claims

1. A processor implemented method, performed by a mobile terminal, of recognizing a user'"'"'s motions using an Orthogonal Semi-Supervised Non-negative Matrix Factorization (OSSNMF)-based data factorization, the method comprising:

measuring, using a plurality of sensors by one or more processors of the mobile terminal, sensor data;

determining, by the one or more processors, label data from the measured sensor data;

generating, by the one or more processors, a spectrogram of the measured sensor data based on a conversion of the measured sensor data from a time domain to a frequency domain;

generating, by the one or more processors, first and second motion feature data associated with the user and extracted, using an orthogonal conversion, from the measured sensor data, based on the generated spectrogram of the measured sensor data and the determined label data;

repeatedly performing, by the one or more processors, the orthogonal conversion, associated with the measured sensor data, while the first and second motion feature data are updated every time when the orthogonal conversion is performed, wherein

for the repeatedly performing of the orthogonal conversion, the method further includes randomly initializing the first motion feature data, the second motion feature data, and an encoding feature data, and selectively updating the initialized first motion feature data, the initialized second motion feature data, and the initialized encoding feature data while requiring that each of the updated first motion feature data and the updated second feature data are orthogonal and without calculating a respective pseudo-inverse of the initialized first motion feature data and the initialized second motion feature data, so that the performing of the orthogonal conversion does not include performing an additional inversion of the measured sensor data, resulting in improving an accuracy of the extracting of motion features data and reducing resource requirement to perform the orthogonal conversion, thereby improving a performance of the recognizing of the user'"'"'s motions; and

performing the recognizing of, by the one or more processors, the user'"'"'s motions based on extracted motion feature data of the user'"'"'s motions and combined motion feature data of the first and second motion feature data.

2. The method of claim 1, wherein the repeatedly performing of the orthogonal conversion includes repeating the orthogonal conversion until a result of the performing of the orthogonal conversion is converged to a predetermined value, while an orthogonality between the measured sensor data and orthogonally-converted data of the measured sensor data is maintained, and

wherein, in response to the result of the performing of the orthogonal conversion being determined to be converged to the predetermined value, the method further comprises terminating the repeatedly performing of the orthogonal conversion and the updating of the first and second motion feature data.

3. A non-transitory computer readable medium including instructions, which when executed by any of one or more processors, cause the one or more processors to perform the method of claim 2.

4. The method of claim 1, wherein the updating of the first and second motion feature data further includes L2-normalizing of the first motion feature data.

5. A non-transitory computer readable medium including instructions, which when executed by any of one or more processors, cause the one or more processors to perform the method of claim 4.

6. The method of claim 1, wherein the first and second motion feature data are respectively represented in a vector form, and the generating of the first and second motion feature data includes extracting the first and second motion feature data using the Orthogonal Semi-Supervised Non-negative Matrix Factorization (OSSNMF)-based data factorization.

7. The method of claim 1, wherein the first and second motion feature data is represented in a vector form, and the combined motion feature data of the first and second motion feature data corresponds to an integrated feature vector including an OSSNMF-based feature vector, the integrated feature vector being generated in the generating of the first and second motion feature data based on the generated spectrogram of the measured sensor data and the determined label data.

8. The method of claim 1, wherein the generating of the first and second motion feature data includes extracting of the first and second motion feature data, further including:

performing an extracting of the first motion feature data, based on the generated spectrogram of the sensor data and orthogonally-converted data of the measured sensor data, without the additional inversion of the measured sensor data; and

performing an extracting of the second motion feature data, based on the label data and the measured sensor data.

9. The method of claim 1, wherein the acquiring of the sensor data further includes:

combining respective sensor information, derived from the plurality of sensors, based on a coordinate system; and

providing the sensor data that accumulates different types of the respective sensor information derived from the plurality of sensors.

10. The method of claim 9, wherein

the plurality of sensors for the terminal includes an acceleration sensor, a gyro sensor, and a magnetism sensor.

11. A non-transitory computer readable medium including instructions, which when executed by one or more processors, cause any of the one or more processors of a mobile terminal to perform a method of recognizing a user'"'"'s motions using an orthogonal semi-supervised non-negative data factorization, the method comprising:

measuring, using a plurality of sensors by one or more processors of the mobile terminal, sensor data;

determining, by the one or more processors, label data from the measured sensor data;

generating, by the one or more processors, a spectrogram of the measured sensor data based on a conversion of the measured sensor data from a time domain to a frequency domain;

generating, by the one or more processors, first and second motion feature data associated with the user and extracted, using an orthogonal conversion, from the measured sensor data, based on the generated spectrogram of the measured sensor data and the label data;

repeatedly performing, by the one or more processors, the orthogonal conversion, associated with the measured sensor data, while the first and second motion feature data are updated every time when the orthogonal conversion is performed, wherein

for the repeatedly performing of the orthogonal conversion, the method further includes randomly initializing the first motion feature data, the second motion feature data, and an encoding feature data, and selectively updating the initialized first motion feature data, the initialized second motion feature data and the initialized encoding feature data while requiring that each of the updated first motion feature data and the updated second feature data are orthogonal and without calculating a respective pseudo-inverse of the initialized first motion feature data and the initialized second motion feature data, so that the performing of the orthogonal conversion does not include performing an additional inversion of the measured sensor data, resulting in improving an accuracy of the extracting of motion features data and reducing resource requirement to perform the orthogonal conversion, thereby improving a performance of the recognizing of the user'"'"'s motions; and

performing the recognizing of, by the one or more processors, the user'"'"'s motions based on extracted motion feature data of the user'"'"'s motions and combined motion feature data of the first and second motion feature data.

12. A non-transitory computer readable medium including instructions, which when executed by one or more processors, cause any of the one or more processors of a mobile terminal to perform a method of recognizing a user'"'"'s motions using an orthogonal semi-supervised non-negative data factorization, the method comprising:

measuring, using a plurality of sensors by one or more processors of the mobile terminal, sensor data;

determining, by the one or more processors, label data from the measured sensor data;

generating, by the one or more processors, a spectrogram of the measured sensor data based on a conversion of the measured sensor data from a time domain to a frequency domain;

generating, by the one or more processors, first and second motion feature data associated with the user and extracted, using an orthogonal conversion, from the measured sensor data, based on the generated spectrogram of the measured sensor data and the label data, so that the orthogonal conversion does not include performing an additional inversion of the measured sensor data, resulting in improving an accuracy of the extracting of motion features data and reducing resource requirement to perform the orthogonal conversion, thereby improving a performance of the recognizing of the user'"'"'s motions;

repeatedly performing, by the one or more processors, the orthogonal conversion, associated with the measured sensor data, while the first and second motion feature data are updated every time when the orthogonal conversion is performed; and

performing the recognizing of, by the one or more processors, the user'"'"'s motions based on extracted motion feature data of the user'"'"'s motions and combined motion feature data of the first and second motion feature data.

13. The non-transitory computer readable medium of claim 12, wherein the repeatedly performing of the orthogonal conversion further includes:

randomly initializing the first motion feature data, the second motion feature data, and an encoding feature data; and

selectively updating the initialized first motion feature data, the initialized second motion feature data, and the initialized encoding feature data while requiring that each of the updated first motion feature data and the updated second feature data are orthogonal and without calculating a respective pseudo-inverse of the initialized first motion feature data and the initialized second motion feature data.

14. The non-transitory computer readable medium of claim 12, wherein the first and second motion feature data are respectively represented in a vector form, and the generating of the first and second motion feature data includes extracting the first and second motion feature data using the Orthogonal Semi-Supervised Non-negative Matrix Factorization (OSSNMF)-based data factorization.

15. The non-transitory computer readable medium of claim 12, wherein the first and second motion feature data are respectively represented in a vector form, and the combined motion feature data of the first and second motion feature data corresponds to an integrated feature vector including an OSSNMF-based feature vector, the integrated feature vector being generated based on the generated spectrogram of the measured sensor data and the determined label data.

16. The non-transitory computer readable medium of claim 12, wherein the generating of the first and second motion feature data includes extracting the first and second motion feature data, further including:

performing an extracting of the first motion feature data, based on the generated spectrogram of the sensor data and orthogonally-converted data of the measured sensor data, without the additional inversion of the measured sensor data; and

performing an extracting of the second motion feature data, based on the label data and the measured sensor data.

17. The non-transitory computer readable medium of claim 12, wherein the acquiring of the sensor data further includes:

combining respective sensor information, derived from the plurality of sensors, based on a coordinate system; and

providing the sensor data that accumulates different types of the respective sensor information derived from the plurality of sensors.