Information processing apparatus, information processing method, program for recognizing behavior

• US 8,566,272 B2
• Filed: 01/26/2010
• Issued: 10/22/2013
• Est. Priority Date: 01/28/2009
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Fukumoto, Yasutaka, Murata, Makoto, Kurata, Masatomo, Katsu, Masanori
• Primary Examiner(s): Gaffin, Jeffrey A
• Assistant Examiner(s): Hill, Stanley K
• Application Number:
  US12/693,888
• Publication Number:
  US 20100191697A1
• Time in Patent Office: 1,365 Days
• Field of Search: None
• US Class Current: 706/54
• CPC Class Codes: A61B 2562/0219 Inertial sensors, e.g. accelerometers, gyroscopes, tilt switches
• CPC Class Codes: A61B 5/1038 Measuring plantar pressure during gait
• CPC Class Codes: A61B 5/1116 Determining posture transitions
• CPC Class Codes: A61B 5/1123 Discriminating type of movement, e.g. walking or running A61B5/1116, A61B5/112 take precedence
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation
• CPC Class Codes: G06N 5/04 Inference or reasoning models

Abstract

There is provided an information processing apparatus including: a sensor data generator sensing a user behavior and generating sensor data corresponding to the user behavior; a behavior recognizing unit performing a predetermined threshold value process on the sensor data to recognize the behavior exhibited by the user and generating behavior information that is information indicating the behavior exhibited by the user; a behavior manager managing the behavior information generated by the behavior recognizing unit in correspondence with the time point at which the behavior corresponding to the behavior information is exhibited; and a behavior information post-processing unit performing a predetermined post-process on the behavior information managed by the behavior manager, wherein the behavior recognizing unit further includes a plurality of behavior determination units specified to specific behaviors exhibited by the user and generates the behavior information based on the determination results of the plurality of behavior determination units.

Claims

1. An information processing apparatus comprising:

a sensor data generator that senses a user behavior and generates sensor data corresponding to the user behavior;

a behavior recognizing unit that recognizes the behavior exhibited by the user by performing a predetermined threshold value process on the sensor data and generates behavior information that is information indicating the behavior exhibited by the user;

a behavior manager that manages the behavior information generated by the behavior recognizing unit in correspondence with the time point at which the behavior corresponding to the behavior information is exhibited; and

a behavior information post-processing unit that performs a predetermined post-process on the behavior information managed by the behavior manager,

wherein the behavior recognizing unit further includes a plurality of behavior determination units specified to specific behaviors exhibited by the user and generates the behavior information based on determination results of the plurality of behavior determination units, and

wherein the behavior recognizing unit at least includes;

a stopped state determination unit that determines whether or not the user is in a stopped state;

a walking/running state determination unit that determines whether or not the user is in a walking state or in a running state;

a jumping state determination unit that determines whether or not the user is in a jumping state;

a posture change determination unit that determines whether or not the user is in a sitting state or in a standing state;

an elevator boarding determination unit that determines whether or not the user is in a boarding-elevator state;

a train boarding determination unit that determines whether or not the user has boarded a train; and

a turning-to-the-right/turning-to-the-left determination unit that determines whether or not the user turns to the right or to the left.

2. The information processing apparatus according to claim 1, wherein the behavior recognizing unit further includes a vehicle boarding determination unit that determines by using the sensor data and a predetermined identification function set in advance whether or not the user has boarded a vehicle.

3. The information processing apparatus according to claim 1 or 2,

wherein the behavior recognizing unit further includes a sensor data processing unit having:

a sensor data storage unit that stores the sensor data in an FIFO manner; and

a sensor data calculation unit that performs a predetermined calculation by using the sensor data, and

wherein each of the behavior determination units determines the behavior exhibited by the user based on a calculation result output from the sensor data processing unit.

4. The information processing apparatus according to claim 1,

wherein the behavior information post-processing unit further includes an exclusive behavior information re-processing unit that detects whether or not an exclusive characteristic representing that the user exhibits exclusive behaviors that are difficult to simultaneously perform exists in the behavior information, and

wherein, in the case where two or more behaviors have the exclusive characteristic, the exclusive behavior information re-processing unit excludes the behavior information corresponding to at least one behavior among the two or more behaviors.

5. The information processing apparatus according to claim 1,

wherein the behavior information post-processing unit includes a false recognition behavior information re-processing unit that reprocesses the behavior information for each unit time based on a behavior information necessary condition necessary for the user to exhibit each behavior, and

wherein, in the case where the behavior information does not satisfy the behavior information necessary condition, the false recognition behavior information re-processing unit corrects the behavior information corresponding to the behavior.

6. The information processing apparatus according to claim 1, wherein the behavior information post-processing unit includes a behavior information real-time adjusting unit that determines that the user exhibits each behavior at a time point that is earlier than a time point at which the behavior recognizing unit transmits the behavior information to the behavior manager, by an amount of time necessary for the behavior recognizing unit to generate the behavior information corresponding to each behavior.

7. The information processing apparatus according to claim 1, wherein, in a case where two or more behaviors performed within a predetermined time are similar to each other, the behavior information post-processing unit includes a behavior comparison re-processing unit that compares summed times of the behavior information corresponding to the two or more behaviors and selects the behavior information having the longest summed time as the behavior information for the predetermined time.

8. The information processing apparatus according to claim 3,

wherein the sensor data includes first to third acceleration sensor data that are data regarding accelerations in predetermined coordinate axes,

wherein the sensor data calculation unit calculates variance values of the first to third acceleration sensor data in a predetermined time stored in the sensor data storage unit, and

wherein, in the case where the maximum variance value that is the largest variance value is smaller than a stop recognition value of recognizing that the user stops and a time when the maximum variance value is smaller than the stop recognition value continues to be longer than a stop recognition time when the user is recognized as stopping, the stopped state determination unit generates the behavior information indicating that the user stops, and in the case where the maximum variance value is smaller than the stop recognition value and the time when the maximum variance value is smaller than the stop recognition value does not continue to be longer than the stop recognition time, the stopped state determination unit generates the behavior information indicating that the user temporarily stops.

9. The information processing apparatus according to claim 3,

wherein the sensor data includes first to third acceleration sensor data that are data regarding accelerations in predetermined coordinate axes,

wherein the sensor data calculation unit calculates variance values of the first to third acceleration sensor data in a predetermined time stored in the sensor data storage unit and the maximum variance value that is the largest variance value, calculates an autocorrelation function of the acceleration sensor data stored in the sensor data storage unit and a maximum value of the autocorrelation function to calculate walking/running frequency data at the time when the user walks or runs, and calculates walking/running determination data for determining whether or not to walk or run by multiplying the walking/running frequency data with the maximum variance value, and

wherein, in the case where a value of the walking/running determination data is larger than a minimum walking recognition value that is a lower limit value for recognizing that the user walks and smaller than a maximum walking recognition value that is an upper limit value for recognizing that the user walks, the walking/running state determination unit generates the behavior information indicating that the user walks, and in the case where the value of the walking/running determination data is larger than the maximum walking recognition value, the walking/running state determination unit generates the behavior information indicating that the user runs.

10. The information processing apparatus according to claim 3,

wherein the sensor data includes first to third acceleration sensor data that are data regarding accelerations in predetermined coordinate axes,

wherein the sensor data calculation unit calculates a jumping state determination value by calculating magnitudes of jumping accelerations represented by magnitudes of the first to third acceleration sensor data, and

wherein, in the case where the jumping state determination value is larger than a minimum jumping recognition value that is a lower limit value for recognizing that the user jumps, the jumping state determination unit generates the behavior information indicating that the user jumps.

11. The information processing apparatus according to claim 3,

wherein the sensor data includes first to third acceleration sensor data that are data regarding accelerations in predetermined coordinate axes,

wherein the sensor data calculation unit calculates first to third gravity data in the predetermined coordinate axes based on the first to third acceleration sensor data, calculates a first gravity change data representing how much the first gravity data is changed from the first gravity data that are previously stored in the storage unit, a second gravity change data representing how much the second gravity data is changed from the second gravity data that are previously stored in the storage unit, and a third gravity change data representing how much the third gravity data is changed from the third gravity data that are previously stored in the storage unit, and calculates a posture change determination value of determining whether or not the user changes posture, which is represented by magnitudes of the first gravity change data, the second gravity change data, and the third gravity change data, and

wherein, in the case where the posture change determination value is larger than a minimum posture change recognition value that is a lower limit value for recognizing that the user changes posture and in the case where the user is already standing, the posture change determination unit generates the behavior information indicating that the user sits, and in the case where the posture change determination value is larger than the minimum posture change recognition value and in the case where the user is already sitting, the posture change determination unit generates the behavior information indicating that the user stands.

12. The information processing apparatus according to claim 3,

wherein the sensor data includes first to third acceleration sensor data that are data regarding accelerations in predetermined coordinate axes,

wherein the sensor data calculation unit calculates gravity direction acceleration sensor data that is information indicating the acceleration in the gravity direction based on the first acceleration sensor data, the second acceleration sensor data, and the third acceleration sensor data and calculates elevator rising determination data of determining whether or not the user boards the elevator by calculating a difference between the gravity direction acceleration sensor data and the gravity,

wherein in the case where the elevator rising determination data is larger than a predetermined value D_α

at first and, after that, smaller than a predetermined value D_β

, the elevator boarding determination unit generates the behavior information indicating that the user is rising in the elevator, and in the case where elevator rising determination data is smaller than the predetermined value D_β

at first and, after that, larger than the predetermined value D_α

, the elevator boarding determination unit generates the behavior information indicating that the user is descending in the elevator,

wherein the D_α

is a minimum elevator rising recognition value that is a lower limit value for recognizing that the user starts to rise in the elevator, and

wherein the D_β

is a maximum elevator descending recognition value that is an upper limit value for recognizing that the user starts to descend in the elevator.

13. The information processing apparatus according to claim 12,

wherein the sensor data calculation unit calculates gravity adjusting data represented by the magnitudes of the first acceleration sensor data, the second acceleration sensor data, and the third acceleration sensor data to allow the value of the gravity to be corrected, records the gravity adjusting data in the sensor data storage unit, and calculates a gravity adjusting variance value that is a variance value of the gravity adjusting data stored in the sensor data storage unit and a gravity adjusting average data that is an average value of the gravity adjusting data, and

wherein, in the case where the gravity adjusting variance value is smaller than a maximum allowable gravity adjusting variance value that is a maximum variance value for allowing the gravity to be adjusted, in the case where the gravity adjusting average data is larger than a minimum allowable gravity average value that is a minimum average value of allowing the gravity to be adjusted, and in the case where the gravity adjusting average data is smaller than a maximum allowable gravity average value that is a maximum average value of allowing the gravity to be adjusted, the sensor data calculation unit considers the value of the gravity adjusting average data as the gravity after correction and performs calculation by using the gravity after correction instead of the gravity.

14. The information processing apparatus according to claim 3,

wherein the sensor data includes first to third acceleration sensor data that are data regarding accelerations in predetermined coordinate axes,

wherein the sensor data calculation unit calculates horizontal direction acceleration sensor data and vertical direction acceleration sensor data based on the first to third acceleration sensor data, records the horizontal direction acceleration sensor data and the vertical direction acceleration sensor data in the sensor data storage unit, calculates a horizontal direction variance value based on the horizontal direction acceleration sensor data recorded in the sensor data storage unit, calculates a vertical direction variance value based on the vertical direction acceleration sensor data recorded in the sensor data storage unit, and calculates boarding determination data of determining whether or not the user has boarded the train by integrating the small variance value between the horizontal direction variance value and the vertical direction variance value, and

wherein, in the case where the train boarding determination data is larger than a minimum train boarding recognition value that is a lower limit value for recognizing that the user has boarded the train, the train boarding determination unit generates the behavior information indicating that the user has boarded the train.

15. The information processing apparatus according to claim 3,

wherein the sensor data includes first to third acceleration sensor data that are data regarding accelerations in predetermined coordinate axes and first to third gyro sensor data that are data regarding gyros in predetermined coordinate axes,

wherein the sensor data calculation unit calculates an angular velocity in the gravity direction based on the first to third acceleration sensor data, from which the frequency area is removed, and the first to third gyro sensor data, and

wherein, in the case where the angular velocity is smaller than a maximum turning-to-the-right recognition value that is an upper limit value for recognizing that the user turns to the right, the turning-to-the-right/turning- to-the-left determination unit generates the behavior information indicating that the user turns to the right, and in the case where angular velocity is larger than a minimum turning-to-the-left recognition value that is a lower limit value for recognizing that the user turns to the left, the turning-to-the-right/turning-to-the-left determination unit generates the behavior information indicating that the user turns to the left.

16. The information processing apparatus according to claim 3,

wherein the sensor data includes first to third acceleration sensor data that are data regarding accelerations in predetermined coordinate axes,

wherein the sensor data calculation unit calculates a value of the identification function by using a characteristic vector generated based on the first to third acceleration sensor data and the identification function set according to vehicle type, and

wherein, in the case where the calculated value of the identification function is larger than a vehicle boarding recognition value that is used to recognize whether or not the vehicle has been boarded corresponding to the identification function, the vehicle boarding determination unit generates the behavior information indicating that the user has boarded the vehicle corresponding to the identification function.

19. The information processing apparatus according to claim 1, wherein the behavior recognizing unit further determines whether or not the user is riding in a car.

20. The information processing apparatus according to claim 1, wherein when the user is detected to be leaving home, user-interest information is provided to the user.

21. The information processing apparatus according to claim 20, wherein when the provided user-interest information includes weather information.

22. The information processing apparatus according to claim 1, wherein when the user is detected to be in a train station, user-interest information is provided to the user.

23. The information processing apparatus according to claim 22, wherein when the provided user-interest information includes train schedule information.

17. An information processing method comprising the steps of:

sensing a user behavior and generating sensor data corresponding to the user behavior;

recognizing the behavior exhibited by the user by performing a predetermined threshold value process on the sensor data and generating behavior information that is information indicating the behavior exhibited by the user;

managing the behavior information generated by the behavior recognizing unit in correspondence with the time point at which the behavior corresponding to the behavior information is exhibited; and

performing a predetermined post-process on the behavior information managed by the behavior manager,

wherein, the recognizing the behavior further includes a plurality of the steps of determining behaviors specified to the behaviors exhibited by the user and generating the behavior information based on determination results of the plurality of the steps of determining the behaviors, and

wherein the recognizing the behavior at least includes;

determining whether or not the user is in a stopped state;

determining whether or not the user is in a walking state or in a running state;

determining whether or not the user is in a jumping state;

determining whether or not the user is in a sitting state or in a standing state;

determining whether or not the user is in a boarding-elevator state;

determining whether or not the user has boarded a train; and

determining whether or not the user turns to the right or to the left.

18. A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer, causes the computer to perform a method, the method comprising:

a sensor data generation function of sensing a user behavior and generates sensor data corresponding to the user behavior;

a behavior recognition function of recognizing the behavior exhibited by the user by performing a predetermined threshold value process on the sensor data and generating behavior information that is information indicating the behavior exhibited by the user;

a behavior managing function of managing the behavior information generated by the behavior recognizing unit in correspondence with the time point at which the behavior corresponding to the behavior information is exhibited; and

a behavior information post-process function of performing a predetermined post-process on the behavior information managed by the behavior manager,

wherein the behavior recognition function further includes a plurality of behavior determination functions specified to specific behaviors exhibited by the user and generates the behavior information based on determination results of the plurality of behavior determination functions, and

wherein the behavior recognizing function at least includes;

determining whether or not the user is in a stopped state;

determining whether or not the user is in a walking state or in a running state;

determining whether or not the user is in a jumping state;

determining whether or not the user is in a sitting state or in a standing state;

determining whether or not the user is in a boarding-elevator state;

determining whether or not the user has boarded a train; and

determining whether or not the user turns to the right or to the left.