Work recognition system, work recognition device, and work recognition method

• US 8,654,187 B2
• Filed: 05/21/2010
• Issued: 02/18/2014
• Est. Priority Date: 06/08/2009
• Status: Expired due to Fees

• Current Assignee: Sovereign Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Inventors: Mizutani, Kenji
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): Rahman, Mohammad J
• Application Number:
  US13/056,789
• Publication Number:
  US 20110128363A1
• Time in Patent Office: 1,369 Days
• Field of Search: 348/77
• US Class Current: 348/77
• CPC Class Codes: A61B 2505/07 Home care
• CPC Class Codes: A61B 5/1114 Tracking parts of the body
• CPC Class Codes: A61B 5/6824 Arm or wrist
• CPC Class Codes: A61B 5/746 Alarms related to a physiological condition, e.g. details of setting alarm thresholds or avoiding false alarms alarms per se G08B
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0383 Signal control means within the pointing device

Abstract

A work recognition device, which recognizes processes in a user'"'"'s work, includes a whole-state observation time determination unit that determines, as a whole-state observation time, from among observation times of chronological first observation values generated by a first sensor to represent movements of a user'"'"'s hand, an observation time of observing a first observation value representing a predetermined movement of the hand; and an observation value selection unit that selects a second observation value corresponding to the time determined by the determination unit from among chronological second observation values generated by a second sensor to represent scenes in the user'"'"'s work. Additionally, a process recognition unit specifies a process in the work, by comparing the second observation value selected by the selection unit to pieces of knowledge data associated with work processes in which the user performs predetermined movements of the hand.

Claims

1. A work recognition device, comprising:

a whole-state observation time determination unit configured to determine, as a whole-state observation time, from among observation times of a plurality of first observation values generated by a speed sensor to represent respective movements of a hand of a user in chronological order, an observation time of observing a first observation value representing a predetermined movement of the hand, based on magnitudes of the plurality of first observation values and an angular difference between the plurality of first observation values; and

an observation value selection unit configured to select a second observation value corresponding to the whole-state observation time determined by the whole-state observation time determination unit, from among a plurality of second observation values generated by an imaging unit to represent respective states in work of the user in chronological order,

wherein the whole-state observation time determination unit includes;

a magnitude estimation unit configured to estimate a magnitude of an acceleration that is obtained by subtracting a gravity acceleration from each of the accelerations generated by the speed sensor;

an angular difference calculation unit configured to calculate, for each of the accelerations generated by the speed sensor, an angle between (a) one acceleration at a corresponding observation time and (b) another acceleration at an observation time prior to the corresponding observation time; and

a time determination unit configured to determine, as the whole-state observation time, the corresponding observation time, when (a) a magnitude of the one acceleration which is estimated by the magnitude estimation unit and (b) the angle calculated by the angular difference calculation unit show a predetermined pattern.

2. The work recognition device according to claim 1, further comprising a process recognition unit configured to recognize a process in the work, by comparing the second observation value selected by the observation value selection unit to pieces of knowledge data associated with respective processes in the work, in each of the respective processes the user is performing the predetermined movement of the hand.

3. The work recognition device according to claim 1, wherein the whole-state observation time determination unit is configured to determine, as the whole-state observation time, a time having a local minimum value of an amount of a change between the plurality of first observation values.

4. The work recognition device according to claim 1, wherein the whole-state observation time determination unit is configured to determine the whole-state observation time based on magnitudes of at least a part of the plurality of first observation values, the at least the part being generated during a time period after a first time until a predetermined time, and the first time being a time having a local minimum value of an amount of a change between the plurality of the first observation values.

5. The work recognition device according to claim 4, wherein the whole-state observation time determination unit is configured to determine, as the whole-state observation time, a time closest to the first time from among times having local maximum values or maximum values of the magnitudes of the at least a part of the plurality of first observation values generated during the time period after the first time until the predetermined time.

6. The work recognition device according to claim 1, wherein the speed sensor is a triaxial acceleration sensor that generates accelerations in chronological order as the plurality of first observation values.

7. The work recognition device according to claim 6, wherein the user wears the speed sensor on a dominant hand of the user.

8. The work recognition device according to claim 6, wherein the whole-state observation time determination unit is configured to determine, as the whole-state observation time, based on the plurality of the first observation values generated in chronological order, an observation time having a local minimum value representing a change of a direction of the hand, when a magnitude of an acceleration of the hand becomes locally maximum after the observation time having the local minimum value.

9. The work recognition device according to claim 1, wherein the time determination unit is configured to determine, as the whole-state observation time, a time at which (b) the angle calculated by the angular difference calculation unit approaches asymptotically to 0 degrees, when (a) the magnitude of the one acceleration which is estimated by the magnitude estimation unit becomes locally maximum within a predetermined time period immediately after the time.

10. The work recognition device according to claim 9, wherein the time determination unit is configured to determine the time at which (a) the angle calculated by the angular difference calculation unit approaches asymptotically to 0 degrees, using a learning machine that learns angles at respective times at which the angles approaches asymptotically to 0 degrees.

11. The work recognition device according to claim 1, wherein the magnitude estimation unit is configured to approximate magnitudes of accelerations that are obtained by subtracting a magnitude of the gravity acceleration from each of the magnitudes of the accelerations generated by the speed sensor.

12. The work recognition device according to claim 1, wherein the imaging unit is a camera that captures an image of a range of a process in the work of the user.

13. A work recognition system that recognizes processes in work performed by a user, the work recognition system comprising:

a speed sensor that generates, in chronological order, a plurality of first observation values representing respective movements of a hand of the user performing the work;

an imaging unit configured to generate, in chronological order, a plurality of second observation values representing respective states in the work; and

a work recognition device that recognizes a process in the work based on the plurality of first observation values generated in chronological order and the plurality of second observation values generated in chronological order,

wherein the work recognition device includes;

a whole-state observation time determination unit configured to determine, as a whole-state observation time, from among observation times of a plurality of first observation values generated in chronological order, an observation time of observing a first observation value representing a predetermined movement of the hand, based on magnitudes of the plurality of first observation values and an angular difference between the plurality of first observation values;

an observation value selection unit configured to select a second observation value corresponding to the whole-state observation time determined by the whole-state observation time determination unit, from among a plurality of second observation values generated in chronological order; and

a process recognition unit configured to specify a process in the work, by comparing the second observation value selected by the observation value selection unit to pieces of knowledge data associated with respective processes in the work, in each of the respective processes the user performing the predetermined movement of the hand,

wherein the whole-state observation time determination unit includes;

a magnitude estimation unit configured to estimate a magnitude of an acceleration that is obtained by subtracting a gravity acceleration from each of the accelerations generated by the speed sensor;

an angular difference calculation unit configured to calculate, for each of the accelerations generated by the speed sensor, an angle between (a) one acceleration at a corresponding observation time and (b) another acceleration at an observation time prior to the corresponding observation time; and

a time determination unit configured to determine, as the whole-state observation time, the corresponding observation time, when (a) a magnitude of the one acceleration which is estimated by the magnitude estimation unit and (b) the angle calculated by the angular difference calculation unit show a predetermined pattern.

14. A work recognition method of recognizing, by using a computer, processes in work performed by a user, the work recognition method comprising:

determining, as a whole-state observation time, from among observation times of a plurality of first observation values generated by a speed sensor to represent respective movements of a hand of the user in chronological order, an observation time of observing a first observation value representing a predetermined movement of the hand, based on magnitudes of the plurality of first observation values and an angular difference between the plurality of the first observation values;

selecting a second observation value corresponding to the whole-state observation time determined in the determining, from among a plurality of second observation values generated by an imaging unit to represent respective states in work of the user; and

specifying a process in the work, by comparing the second observation value selected in the selecting to pieces of knowledge data associated with respective processes in the work, in each of the respective processes the user performing the predetermined movement of the hand,

wherein the determining step includes;

estimating a magnitude of an acceleration that is obtained by subtracting a gravity acceleration from each of the accelerations generated by the speed sensor;

calculating, for each of the accelerations generated by the speed sensor, an angle between (a) one acceleration at a corresponding observation time and (b) another acceleration at an observation time prior to the corresponding observation time; and

determining, as the whole-state observation time, the corresponding observation time, when (a) a magnitude of the one acceleration which is estimated in the estimating and (b) the angle calculated in the calculating show a predetermined pattern.

15. A computer program which is recorded on a non-transitory computer-readable recording medium for use in a computer and is used for recognizing processes in work performed by a user, the computer program causing the computer to execute:

determining, as a whole-state observation time, from among observation times of a plurality of first observation values generated by a speed sensor to represent respective movements of a hand of the user in chronological order, an observation time of observing a first observation value representing a predetermined movement of the hand, based on magnitudes of the plurality of first observation values and an angular difference between the plurality of first observation values;

selecting a second observation value corresponding to the whole-state observation time determined in the determining, from among a plurality of second observation values generated by an imaging unit to represent respective states in work of the user; and

specifying a process in the work, by comparing the second observation value selected in the selecting to pieces of knowledge data associated with respective processes in the work, in each of the respective processes the user performing the predetermined movement of the hand,

wherein the determining step includes;

estimating a magnitude of an acceleration that is obtained by subtracting a gravity acceleration from each of the accelerations generated by the speed sensor;

calculating, for each of the accelerations generated by the speed sensor, an angle between (a) one acceleration at a corresponding observation time and (b) another acceleration at an observation time prior to the corresponding observation time; and

determining, as the whole-state observation time, the corresponding observation time, when (a) a magnitude of the one acceleration which is estimated in the estimating and (b) the angle calculated in the calculating show a predetermined pattern.