Apparatus and method for recognizing user activity

• US 9,460,398 B2
• Filed: 09/13/2012
• Issued: 10/04/2016
• Est. Priority Date: 03/29/2012
• Status: Active Grant

• Current Assignee: Postech Academy-Industry Foundation, Samsung Electronics Co. Ltd.
• Original Assignee: Postech Academy-Industry Foundation, Samsung Electronics Co. Ltd.
• Inventors: Sung, Jae-Mo, Lee, Sun-Jae, Choi, Seungjin
• Primary Examiner(s): Hill, Stanley K
• Assistant Examiner(s): Kim, David H
• Application Number:
  US13/614,546
• Publication Number:
  US 20130262352A1
• Time in Patent Office: 1,482 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

A user activity real-time recognition apparatus and method are provided and include a collector configured to collect a frequency-domain signal for each user activity and to generate learning data based on the frequency-domain signal. The apparatus and method also include an extractor configured to extract a user activity feature from the frequency-domain signal based on an activity feature extracting model. The activity feature extracting model is learned based on the learning data from the collector. The apparatus and method further include a classifier configured to analyze the user activity feature to classify a user activity pattern based on an activity pattern classifying model and configured to transmit the classified user activity pattern to an application device.

Claims

1. A user activity recognition apparatus, comprising:

a collector configured to collect a frequency-domain signal for a user activity and to generate learning data based on the frequency-domain signal;

an activity feature extracting model that is learned based on the learning data from the collector;

an extractor configured to extract a user activity feature from the frequency-domain signal based on the activity feature extracting mode;

a classifier configured to analyze the user activity feature to classify a user activity pattern based on an activity pattern classifying model and configured to transmit the classified user activity pattern to an application device;

a first sensor configured to detect a first signal to analyze a user activity; and

a second sensor configured to detect a second signal to correct the first signal;

wherein the learning data corresponds to user activity patterns related to user containment of a mobile terminal.

2. The user activity recognition apparatus of claim 1, further comprising:

an activity feature extracting model database configured to store the activity feature extracting model and configured to provide the activity feature extracting model to the extractor; and

an activity pattern classifying model database configured to store the activity feature classifying model and configured to provide the activity pattern classifying model to the classifier.

3. The user activity recognition apparatus of claim 2, further comprising:

a user activity recognizer comprising the extractor and the classifier and configured to extract the user activity feature with reference to the activity feature extracting model database and to classify the user activity pattern according to the user activity feature by searching the activity pattern classifying model database.

4. The user activity recognition apparatus of claim 2, wherein the recognizer further comprises a Fast Fourier transformer configured to transform a normalized signal unrelated to how the user contains a mobile terminal to the frequency-domain signal.

5. The user'"'"'s activity recognition apparatus of claim 1, further comprising:

a signal normalizer configured to normalize the first signal to a signal unrelated to how the user contains the mobile terminal using the second signal.

6. The user activity recognition apparatus of claim 1, wherein the classifier transmits information about the user activity pattern to the application device enabling recognition of an activity of a user.

7. The user activity recognition apparatus of claim 6, wherein the application device is a device controller.

8. The user activity recognition apparatus of claim 6, wherein the application device is a user activity analyzer.

9. The user activity recognition apparatus of claim 6, wherein the application device is a life logging device.

10. The user activity recognition apparatus of claim 6, wherein the application device is a user context recognizer.

11. The user activity recognition apparatus of claim 4, wherein the learning data collector collects the frequency-domain signal for each user activity, transformed by the Fast Fourier transformer.

12. The user activity recognition apparatus of claim 1, wherein the learning data collector transmits the learning data to the learning server.

13. The user activity recognition apparatus of claim 5, wherein the first sensor is a sensor including a device coordinate system related to how the user contains the mobile terminal.

14. The user activity recognition apparatus of claim 13, wherein the first sensor is a sensor including a global coordinate system unrelated to how the user contains the mobile terminal.

15. The user activity recognition apparatus of claim 14, wherein the signal normalizer is a Global Coordinate Transformer (GCT) configured to obtain normalized acceleration data unrelated to how the user contains the mobile terminal by rotating a 3-axis device coordinate detected by an acceleration sensor, wherein the acceleration sensor comprises a device coordinate system related to how the user contains the mobile terminal to a global coordinate detected by a magnetic field sensor, wherein the magnetic field sensor has a global coordinate system unrelated to how the user contains the mobile terminal.

16. The user activity recognition apparatus of claim 1, further comprising:

a learning server configured to learn the activity feature extracting model based on the learning data from the collector, wherein the learning server comprises an activity feature extracting model learning unit configured to learn activity feature extracting models using a group of learning data collected by the collector, to select an optimal activity feature extracting model from among the activity feature extracting models, and to store the selected optimal activity feature extracting model in an activity feature extracting model database.

17. The user activity recognition apparatus of claim 16, wherein the learning server further comprises an activity pattern classifying model learning unit configured to learn activity pattern classifying models using the activity feature extracting models learned by the activity feature extracting model learning unit, to select an optimal activity pattern classifying model from among the activity pattern classifying models, and to store the selected activity pattern classifying model in an activity pattern classifying model database.

24. The user activity recognition apparatus of claim 1, wherein a device coordinate system is rotated to a global coordinate system using the second signal detected by the second sensor to transform the first signal related to how the use contains the mobile terminal to acceleration data on the global coordinate system unrelated to how the user contains the mobile terminal.

18. A user activity recognition method comprising:

collecting, at a collector, a frequency-domain signal for a user activity;

generating learning data based on the frequency-domain signal;

extracting, at an extractor, a user activity feature from the frequency-domain signal based on an activity feature extracting model, wherein the activity feature extracting model is learned based on the learning data from the collector;

analyzing, at a classifier, the user activity feature to classify a user activity pattern based on an activity pattern classifying model;

detecting, by a first sensor, a first signal to analyze a user activity; and

detecting, by a second sensor, a second signal to correct the first signal,

wherein the learning data corresponds to user activity patterns related to user containment of a mobile terminal.

19. The user activity recognition method of claim 18, further comprising:

storing the activity feature extracting model in an activity feature extracting model database; and

storing the activity feature classifying model in an activity pattern classifying model database.

20. The user activity recognition method of claim 19, further comprising:

extracting the user activity feature with reference to the activity feature extracting model database; and

classifying the user activity pattern according to the user activity feature by searching the activity pattern classifying model database.

21. The user activity recognition method of claim 18, wherein the generating of the learning data comprises

collecting the frequency-domain signal for each user activity based on a Fast Fourier transformation,

generating the learning data based on the frequency-domain signal, and

transmitting the learning data to a learning server.

22. The user activity recognition method of claim 20, further comprising:

transmitting information about the user activity pattern to an application device to enable recognition of an activity of a user.

23. A computer program embodied on a non-transitory computer readable medium, the computer program configured to control a processor to perform the user activity recognition method of claim 18.