System and method for multidimensional gesture analysis

• US 8,280,732 B2
• Filed: 03/26/2009
• Issued: 10/02/2012
• Est. Priority Date: 03/27/2008
• Status: Active Grant

• Current Assignee: Microchip Technology Germany GmbH (Microchip Technology Incorporated)
• Original Assignee: Microchip Technology Germany GmbH (Microchip Technology Incorporated)
• Inventors: Richter, Wolfgang, Aubauer, Roland
• Primary Examiner(s): COLUCCI, MICHAEL C
• Application Number:
  US12/411,514
• Publication Number:
  US 20100063813A1
• Time in Patent Office: 1,286 Days
• Field of Search: 715/204, 715/863, 715/246, 704/260, 704/208, 704/275, 704/271, 704/209, 704/200, 340/541, 455/456.1, 381/380, 709/204, 706/11
• US Class Current: 704/231
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features

Abstract

Hand gestures are translated by first detecting the hand gestures with an electronic sensor and converting the detected gestures into respective electrical transfer signals in a frequency band corresponding to that of speech. These transfer signals are inputted in the audible-sound frequency band into a speech-recognition system where they are analyzed.

Claims

1. A method of translating hand gestures, the method comprising the steps of:

detecting the hand gestures with an electronic sensor and converting the detected gestures into respective electrical transfer signals in a frequency band corresponding to that of speech;

inputting the transfer signals in the audible-sound frequency band to a speech-recognition system; and

analyzing the inputted transfer signals with the speech-recognition system.

2. The method according to claim 1 wherein the input signals are generated by a tone-generator system.

3. The method according to claim 1 wherein the input signals are generated by a terminal circuit.

4. The method according to claim 1 wherein the transfer signals are generated in such a way that they map the spatial movement of the gesture.

5. The method according to claim 1 wherein the transfer signals are composed of three tones, with the composition of the individual tones changing due to the movement of a gesturing hand or a finger according to a movement-mapping rule.

6. The method according to claim 1 wherein, on movement away from a detection point, the pitch of the transfer signal for this reference point increases.

7. The method according to claim 1 wherein, on movement toward a detection point, the pitch of the transfer signal for this reference point decreases.

8. The method according to claim 1 wherein the transfer signals are structured such that they describe the change in distance from the gesturing hand relative to a reference system by means of at least three reference points.

9. The method according to claim 1 wherein the gestures are sensed by at least three e-field sensors producing respective sensor output signals and only those sensor output signals are recognized as gestures in which at least one sensor signal exceeds a predetermined level.

10. The method according to claim 9 wherein only those sensor signals that have a minimum alteration dynamic are recognized as gestures.

11. The method according to claim 10, further comprising the step of

determining mathematical derivatives of the sensor signals.

12. The method according to claim 11, further comprising the steps of

standardizing sensor signals by subtraction of an average time value or direct signal portion of all sensor signals from the respective individual sensor signals.

13. The method according to claim 9, further comprising the step of

conducting a coordinate transformation of the sensor signals such that the corresponding gestures are mapped in a spatial gesture reference plane.

14. The method according to claim 13 wherein the gesture reference plane is established such that it runs essentially parallel to a sensor-electrode surfaces and transverse to a sensor axis such that the corresponding gestures are always mapped at a certain distance from the sensors.

15. The method according to claim 14 wherein a center of a surface spanned by the gesture lies essentially on the axis of the sensor.

16. The method according to claim 9 wherein the sensor signals are converted to tone sequences that lie in a speech range of 50 Hz-3400 Hz and the various sensor signals are combined into one single gesture signal to be analyzed by the speech-recognition system.

17. The method according to claim 1 wherein the sensor signals are captured via a time multiplexer.

18. The method according to claim 1 wherein the sensor signals are transformed by means of VCO conversion into different frequency bands in the speech frequency range.

19. The method according to claim 1 wherein the gesture recognition is managed using a dynamic time warping pattern recognizer for recognizing total gestures, analogously to word recognition in speech recognition.

20. The method according to claim 1 wherein the gesture recognition is managed using a Hidden Markov Model recognizer known from speech recognition.

21. The method according to claim 1 wherein the gestures to be recognized are trained and stored as reference patterns.

22. The method according to claim 1 wherein gestures that are recognized during a training as being too different are rejected as unreliable.

23. The method according to claim 1 wherein similar gestures that are not rejected are combined by averaging the corresponding reference signals.

24. The method according to claim 1 wherein a gesture training process is supported by visual parameters or visual feedback.

25. A system for translating hand gestures, the system comprising:

for a gesture detection unit operable to detect the hand gestures and to convert the detected hand gestures into respective electrical transfer signals in a frequency band corresponding to that of speech;

a speech-recognition system

wherein the electrical transfer signals are fed to the speech-recognition system for analysis of the inputted transfer signals by the speech-recognition system.

26. The system according to claim 25 wherein the sensor array is integrated into a mouse, mouse pad, headset, keyboard, or monitor.

27. The system according to claim 25 wherein the sensor array is integrated into a household appliance or a machine for performing input operations.

28. The system according to claim 25 wherein the sensor array is integrated into a mobile communications device.

29. The system according to claim 25 wherein the sensor array is integrated into a motor-vehicle navigation system.

30. The system according to claim 25 wherein the sensor array is integrated into an electronic toy.

31. A system for translating hand gestures, the system comprising:

means including a sensor array for detecting the hand gestures and converting the detected gestures into respective electrical transfer signals in a frequency band corresponding to that of speech;

a speech-recognition system; and

means for inputting the transfer signals in the audible-sound frequency band to the speech-recognition system for analysis of the inputted transfer signals by the speech-recognition system.