User terminal device and method for recognizing user'S location using sensor-based behavior recognition

• US 10,341,981 B2
• Filed: 07/12/2016
• Issued: 07/02/2019
• Est. Priority Date: 09/02/2015
• Status: Active Grant

• Current Assignee: Industry Academic Cooperation Foundation, Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Yu, Seung-hak, Lee, Seung-woo, Cha, Ho-jung, Ko, Yo-han, Kim, Soo-hwan, Kim, Hyun-choong, Shin, Jong-hoon, Lee, Kyoung-woo, Hahm, Seong-il
• Primary Examiner(s): Cai, Wayne H
• Application Number:
  US15/754,512
• Publication Number:
  US 20180249435A1
• Time in Patent Office: 1,085 Days
• CPC Class Codes: H04L 65/40 Support for services or applications
• CPC Class Codes: H04M 2250/02 including a Bluetooth interface
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion
• CPC Class Codes: H04W 4/023 using mutual or relative location information between multiple location based services [LBS] targets or of distance thresholds
• CPC Class Codes: H04W 4/80 Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication
• CPC Class Codes: H04W 64/00 Locating users or terminals or network equipment for network management purposes, e.g. mobility management

Abstract

Disclosed is a system for recognizing a user location using sensor-based activity recognition. The system includes a mobile device which includes a sensor module configured to sense information about at least one of a user state and surrounding environments, a first processor configured to extract activity information based on the sensed information, and a first communication module configured to transmit the extracted activity information; and a user terminal device which includes a second communication module configured to receive the activity information from the first communication module, and a processor configured to determine a user location in a user space corresponding to a user activity based on the received activity information. By determining a user'"'"'s location in a user space corresponding to a user activity, it is possible to control electronic devices corresponding to the user activity or provide a service of monitoring activities of the old and weak and the child.

Claims

1. A user terminal device for recognizing a user location using sensor-based activity recognition, the user terminal device comprising:

a communication interface circuitry (CIC) configured to receive activity information from a mobile device that senses information about at least one of a user state and surrounding environments, and extracts the activity information based on the sensed information, and configured to perform Bluetooth communication and Wi-Fi communication; and

a processor configured to;

identify a user location in a user space corresponding to a user activity based on the received activity information,

measure a distance between the mobile device and the user terminal device based on signal strength of the Bluetooth communication with the mobile device, and

based on a user and the user terminal device being identified to be in the same location when the measured distance is within a predetermined range, identifying a user terminal location obtained by a Wi-Fi-based positioning system (WPS) as a user location.

2. The user terminal device according to claim 1, wherein the mobile device comprises at least one first sensor comprising at least one of an acceleration sensor, a microphone, a heart-beat pulse sensor, a gyroscope, a geomagnetic sensor, a gravity sensor, an optical sensor, a digital compass, an odor sensor, or a proximity sensor.

3. The user terminal device according to claim 2, wherein the user terminal device comprises at least one second sensor comprising at least one of an acceleration sensor, a microphone, a gyroscope, a geomagnetic sensor, a gravity sensor, an optical sensor, a digital compass, an odor sensor, or a proximity sensor.

4. The user terminal device according to claim 3, wherein the mobile device extracts first activity information based on first sensed information sensed by the first sensor, and

the processor extracts second activity information based on second sensed information sensed by the second sensor, and identifies a user location based on the first activity information and the second activity information.

5. The user terminal device according to claim 2, wherein the activity information comprises an activity fingerprint generated based on raw sense data obtained from the first sensor.

6. The user terminal device according to claim 5, wherein the activity fingerprint is generated by a clustering algorithm based on features extracted from the raw sense data.

7. The user terminal device according to claim 5, wherein the processor identifies a user location using a statistics Markov model based on the activity fingerprint.

8. The user terminal device according to claim 7, further comprising a storage configured to store user activity history data in which user activity corresponding to the activity information is associated with activity time.

9. The user terminal device according to claim 1, wherein the processor further comprises a storage configured to store an activity fingerprint corresponding to the user location as a learning model in association with the user location.

10. A system for recognizing a user location using sensor-based activity recognition, the system comprising:

a mobile device which comprises at least one sensor configured to sense information about at least one of a user state and surrounding environments, a first processor configured to extract activity information based on the sensed information, and a first communication interface circuitry (CIC) configured to transmit the extracted activity information and to perform Bluetooth communication; and

a user terminal device which comprises a second CIC configured to receive the activity information from the first CIC, and to perform the Bluetooth communication and Wi-Fi communication, and a second processor configured to;

identify a user location in a user space corresponding to a user activity based on the received activity information,

measure a distance between the mobile device and the user terminal device based on signal strength of the Bluetooth communication with the mobile device, and

based on a user and the user terminal device being identified to be in the same location when the measured distance is within a predetermined range, identifying a user terminal location obtained by a Wi-Fi-based positioning system (WPS) as a user location.

11. A system for recognizing a user location using sensor-based activity recognition, the system comprising:

a first mobile device which comprises at least one sensor configured to sense information about at least one of a user state and surrounding environments, a first processor configured to extract activity information based on the sensed information, and a first communication interface circuitry (CIC) configured to transmit the extracted activity information and to perform Bluetooth communication;

a second mobile device which comprises a second CIC configured to perform Wi-Fi communication and Bluetooth communication, and a second processor configured to obtain its own location information through the Wi-Fi communication and obtain distance information from the first mobile device through the Bluetooth communication; and

a user terminal device which comprises a third CIC configured to receive the activity information from the first CIC and the location information and the distance information from the second CIC, and a third processor configured to;

identify the location information as a user location when the first mobile device and the second mobile device are within a predetermined distance based on the distance information,

identify a user location in a user space corresponding to a user activity based on the received activity information when the first mobile device and the second mobile device are beyond a predetermined distance,

measure a distance between the first mobile device and the user terminal device based on signal strength of the Bluetooth communication with the first mobile device, and

based on a user and the user terminal device being identified to be in the same location when the measured distance is within a predetermined range, identifying a user terminal location obtained by a Wi-Fi-based positioning system (WPS) as a user location,

wherein the user terminal device is the same device as the second mobile device or a device different from the second mobile device.

12. A method of recognizing a user location using sensor-based activity recognition, the method comprising:

by a mobile device, sensing information about at least one of a user state and surrounding environments

extracting activity information based on the sensed information;

transmitting the activity information to a user terminal device; and

identifying a user location in a user space corresponding to the activity information based on the received activity information,

wherein the identifying comprises;

measuring a distance between the mobile device and the user terminal device based on signal strength of Bluetooth communication with the mobile device, and

based on a user and the user terminal device being identified to be in the same location when the measured distance is within a predetermined range, identifying a user terminal location obtained by a Wi-Fi-based positioning system (WPS) as a user location.

13. A non-transitory recording medium for recording a program of identifying the user location in the user space corresponding to the user activity based on the activity information according to claim 12.