Gesture recognition apparatus and method

• US 8,005,257 B2
• Filed: 07/23/2008
• Issued: 08/23/2011
• Est. Priority Date: 10/05/2006
• Status: Expired due to Fees

• Current Assignee: the united states of america as represented by the secretary of the navy
• Original Assignee: the united states of america as represented by the secretary of the navy
• Inventors: Venetsky, Larry, Tieman, Jeffrey W.
• Primary Examiner(s): Tucker; Wesley
• Assistant Examiner(s): Woldemariam; Aklilu
• Application Number:
  US12/177,913
• Publication Number:
  US 20100013944A1
• Time in Patent Office: 1,126 Days
• Field of Search: 382/209, 382/103, 382/107, 382/117, 382/118, 382/153, 348/95, 348/222.1
• US Class Current: 382/100
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A method of identifying a human gesture using a machine includes providing a time sequence of data related to the human gesture; transforming the time sequence of data into waveforms; extracting features from the waveforms; and identifying the human gesture based on the extracted features.

Claims

1. A method of identifying a human gesture, performed, at least in part, with one or more processors, comprising:

providing a time sequence of data related to the human gesture;

transforming the time sequence of data into waveforms comprising a right hand waveform and a left hand waveform;

extracting features from the waveforms, wherein the extracting step comprises extracting static features from the waveforms including hand position, and dynamic features from the waveforms, including phase and frequency from the right hand waveform and phase and frequency from the left hand waveform; and

identifying the human gesture based on the extracted features by comparing one or more of the hand position, phase, and frequency from the right hand waveform and the hand position, phase, and frequency from the left hand waveform to a rule that describes the human gesture,

the rule composed of the features or combinations of the features, that are used to identify uniquely the human gesture.

2. The method of claim 1 wherein the providing step includes providing a time sequence of pixel images of the human gesture using at least one video camera.

3. The method of claim 1 wherein the providing step includes providing the data using accelerometers attached to the human.

4. The method of claim 1 wherein the transforming step includes transforming the data into waveforms that are normalized to a cranial location.

5. The method of claim 1 further comprising classifying hand positions into vertical and horizontal ranges.

6. The method of claim 5 wherein the vertical and horizontal ranges are defined in terms of a characteristic length of a human performing the gesture.

7. The method of claim 6 wherein the characteristic length is a distance D from a cranial point to a torso point.

8. The method of claim 7 wherein the vertical ranges comprise five ranges, a VERY HIGH range being more than 0.25D above the cranial point, a HIGH range being from the cranial point to 0.25D above the cranial point, a MIDDLE range being from the cranial point to 0.6D below the cranial point, a LOW range being below 0.6D but above 0.9D, and a VERY LOW range being 9D or lower.

9. The method of claim 7 wherein the horizontal ranges comprise three ranges, an INSIDE range being from 0 to less than 0.30 from a vertical axis of the human, a MIDDLE range being from 0.3D to 0.7D from the vertical axis and an OUTSIDE range being greater than 0.7D from the vertical axis.

10. The method of claim 7 further comprising classifying a hand position as EXTENDED if a distance from the hand position to a point that is 0.3750 vertically below the cranial point is greater than 0.750.

11. The method of claim 1 further comprising classifying cyclic hand movement with respect to an axis as IN-PHASE if absolute values of maximum positions of the right and left hands from the origin, measured along the axis, are the same or approximately the same, and absolute values of minimum positions of the right and left hands from the origin, measured along the axis, are the same or approximately the same.

12. The method of claim 5 wherein hand position in the vertical range is determined using a fuzzy logic method.

13. The method of claim 1 wherein the gesture is MOVE AHEAD and the rule is a y axis position of both hands is MIDDLE or HIGH or VERY HIGH;

and x axis and y axis motion of the hands is IN-PHASE.

14. The method of claim 1 wherein the gesture is TURN TO LEFT and the rule is left hand is EXTENDED and y axis position of left hand is MIDDLE and left hand NO MOTION on all axes;

right hand y axis position is MIDDLE or HIGH or VERY HIGH and right hand has MOTION.

15. The method of claim 1 wherein the gesture is TURN TO RIGHT and the rule is right hand is EXTENDED and y axis position of right hand is MIDDLE and right hand NO MOTION on all axes;

left hand y axis position is MIDDLE or HIGH or VERY HIGH and left hand has MOTION.

16. The method of claim 1 wherein the gesture is PIVOT TO LEFT and the rule is left hand y axis position is VERY LOW and EXTENDED and NO MOTION;

right hand y axis position is MIDDLE or HIGH or VERY HIGH and right hand has MOTION.

17. The method of claim 1 wherein the gesture is PIVOT TO RIGHT and the rule is right hand y axis position is VERY LOW and right hand is EXTENDED and right hand has NO MOTION;

left hand y axis position is MIDDLE or HIGH or VERY HIGH and left hand has MOTION.

18. The method of claim 1 wherein the gesture is PASS CONTROL:

HAND OFF LEFT and the rule is both hands y axis position is MIDDLE and both hands x axis position is MIDDLE or OUTSIDE and both hands have NO MOTION and both hands x axis position is negative.

19. The method of claim 1 wherein the gesture is PASS CONTROL:

HAND OFF RIGHT and the rule is both hands y axis position is MIDDLE and both hands x axis position is MIDDLE or OUTSIDE and both hands have NO MOTION and both hands x axis position is positive.

20. The method of claim 1 wherein the gesture is I HAVE COMMAND and the rule is left hand y axis position is VERY HIGH, right hand y axis position is VERY LOW, both hands have NO MOTION, left hand x axis position is approximately equal to right hand x axis position.

21. The method of claim 1 wherein the gesture is SLOW DOWN and the rule is both hands y axis position is MIDDLE or LOW, both hands y axis position is approximately equal, both hands have IN PHASE y axis motion and both hands have NO MOTION on x axis.

22. The method of claim 1 wherein the gesture is STOP and the rule is both hands y axis position is HIGH or VERY HIGH, both hands x axis position is INSIDE, both hands have NO MOTION.

23. The method of claim 1 wherein the gesture is MOVE BACK and the rule is both hands y axis position is VERY LOW or LOW, both hands NO MOTION on x axis, both hands IN PHASE on y axis, both hands y axis position approximately equal, both hands IN PHASE on z axis.

24. The method of claim 1 wherein the gesture is SLOW DOWN ENGINE(S) ON LEFT INDICATED SIDE and the rule is right hand y axis position is VERY LOW, right hand has NO MOTION, left hand y axis position is MIDDLE or VERY LOW, left hand has y axis MOTION.

25. The method of claim 1 wherein the gesture is SLOW DOWN ENGINE(S) ON RIGHT INDICATED SIDE and the rule is left hand y axis position is VERY LOW, left hand has NO MOTION, right hand y axis position is MIDDLE or LOW, right hand has y axis MOTION.

26. The method of claim 1 wherein the gesture is ENGAGE NOSEGEAR STEERING RIGHT and the rule is left hand y axis position is MIDDLE, left hand x axis position is INSIDE, left hand NO MOTION, right hand y axis position is MIDDLE, right hand is EXTENDED, right hand x axis position is OUTSIDE, right hand has NO MOTION.

27. The method of claim 1 wherein the gesture is ENGAGE NOSEGEAR LEFT and the rule is right hand y axis position is MIDDLE, right hand x axis position is INSIDE, right hand has NO MOTION, left hand y axis position is MIDDLE, left hand is EXTENDED, left hand x axis position is OUTSIDE, left hand has NO MOTION.

28. The method of claim 1 wherein the gesture is DISENGAGE NOSEGEAR STEERING RIGHT and the rule is left hand y axis position is MIDDLE, left hand x axis position is INSIDE, left hand has NO MOTION, right hand y axis position is MIDDLE) right hand has y axis MOTION, right hand has NO MOTION on x axis.

29. The method of claim 1 wherein the gesture is DISENGAGE NOSEGEAR STEERING LEFT and the rule is right hand y axis position is MIDDLE, right hand x axis position is INSIDE, right hand has NO MOTION, left hand y axis position is MIDDLE, left hand has MOTION on y axis, left hand has NO MOTION on x axis.

30. The method of claim 1 wherein the gesture is LAUNCH BAR UP and the rule is a sequence of two distinct positions, 1) both hands x axis position is INSIDE, right hand y axis position is MIDDLE, left hand y axis position is VERY LOW, left hand is EXTENDED, both hands have NO MOTION, then, within about one second, 2) right hand y axis position is MIDDLE, both hands x axis position is INSIDE, left hand y axis position is HIGH, both hands have NO MOTION.

31. The method of claim 1 wherein the gesture is LAUNCH BAR DOWN and the rule is a sequence of two distinct positions, 1) right hand y axis position is MIDDLE, both hands x axis position is INSIDE, left hand y axis position is HIGH) both hands have NO MOTION, then, within about one second, 2) both hands x axis position is INSIDE, right hand y axis position is MIDDLE, left hand y axis position is VERY LOW, left hand is EXTENDED, both hands NO MOTION.

32. The method of claim 1 wherein the gesture is SPREAD WINGSIHELICOPER BLADES and the rule is a sequence of two static poses:

1) both hands y axis position is MIDDLE, both hands x axis position is INSIDE, both hands have NO MOTION, then, within about one second

     2) both hands y axis position is MIDDLE, both hands are EXTENDED, both hands have NO MOTION, both hands x axis position is OUTSIDE.

33. The method of claim 1 wherein the gesture is UP HOOK and the rule is a sequence of two static poses:

1) right hand y axis position is MIDDLE, right hand x axis position is INSIDE, right hand has NO MOTION, left hand x axis position is INSIDE, left hand has NO MOTION, left hand is EXTENDED, left hand y axis position is VERY LOW, then, within about one second,

     2) both hands y axis position is MIDDLE, both hands x axis position is INSIDE, both hands NO MOTION.

34. The method of claim 1 wherein the gesture is DOWN HOOK and the rule is a sequence of two static poses:

1) right hand y axis position is MIDDLE, right hand x axis position is INSIDE, right hand has NO MOTION, left hand x axis position is INSIDE, left hand has NO MOTION, left hand y axis position is VERY HIGH or HIGH, then within about one second, the next pose is

     2) both hands y axis position is MIDDLE, both hands x axis position is INSIDE, both hands have NO MOTION.

35. The method of claim 1 wherein the gesture is FOLD WINGS/HELICOPTER BLADES and the rule is a sequence of two static poses:

1) both hands y axis position is MIDDLE, both hands are EXTENDED, both hands have NO MOTION, both hands x axis position is OUTSIDE, then within about one second

     2) both hands y axis position is MIDDLE, both hands x axis position is INSIDE, both hands have NO MOTION.

36. A non-transitory computer readable medium containing a computer program for performing a method of identifying a human gesture, the method comprising transforming a time sequence of data related to the human gesture into waveforms comprising a right hand waveform and a left hand waveform;

extracting features from the waveforms, wherein the extracting step comprises extracting static features from the waveforms including hand position, and dynamic features from the waveforms, including phase and frequency from the right hand waveform and phase and frequency from the left hand waveform; and

identifying the human gesture based on the extracted features by comparing one or more of the hand position, phase, and frequency from the right hand waveform and the hand position, phase, and frequency from the left hand waveform to a rule that describes the human gesture, the rule composed of the features or combinations of the features, that are used to identify uniquely the human gesture.

37. An apparatus for identifying a human gesture, comprising:

means for providing a time sequence of data related to the human gesture;

means for transforming the time sequence of data into waveforms comprising a right hand waveform and a left hand waveform;

means for extracting features from the waveforms, wherein the extracting step comprises extracting static features from the waveforms including hand position, and dynamic features from the waveforms, including phase and frequency from the right hand waveform and phase and frequency from the left hand waveform; and

means for identifying the human gesture based on the extracted features by comparing one or more of the hand position, phase, and frequency from the right hand waveform and the hand position, phase, and frequency from the left hand waveform to a rule that describes the human gesture, the rule composed of the features or combinations of the features, that are used to identify uniquely the human gesture by reference to a set of rules comprising features that uniquely identify each human gesture.

38. The apparatus of claim 37 wherein the means for providing comprises accelerometers.

39. The apparatus of claim 37 wherein the means for providing comprises at least one video camera.

40. The apparatus of claim 37 wherein the means for transforming, the means for extracting and the means for identifying comprise at least one computer.