Detecting User Gestures with a Personal Mobile Communication Device

• US 20090157206A1
• Filed: 07/23/2008
• Published: 06/18/2009
• Est. Priority Date: 12/13/2007
• Status: Active Grant

• Current Assignee: Georgia Tech Research Corporation (University System of Georgia)
• Original Assignee: Georgia Tech Research Corporation (University System of Georgia)
• Inventors: Jayaprakash, Jagadeeswaran, Weinberg, Gil
• Granted Patent: US 8,175,728 B2
• Time in Patent Office: Days
• US Class Current: 700/94
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/043 using propagating acoustic waves

Abstract

System and methods for recognizing user gestures by processing audio from a microphone in a personal mobile communication device (PMCD) are disclosed. One such method includes sensing audio using a microphone in the PMCD and detecting a user gesture from audio sensed by the microphone. One such system includes a microphone, memory for storing instructions, and a processor configured by the instructions to sense audio using the microphone and detecting user gestures from audio sensed by the microphone. Also disclosed is a computer-readable medium storing a computer program for use in a personal mobile communication device. The program comprises instructions to: sense audio using a microphone in the personal mobile communication device; record the audio sensed by the microphone; process the recorded audio to extract a feature; classify the feature of the recorded audio; and identify one of a plurality of user gesture types based on the classified feature.

Claims

1. A method for detecting user gestures with a personal mobile communication device, the method comprising:

sensing audio using a microphone in the personal mobile communication device; and

detecting a user gesture from audio sensed by the microphone.

2. The method of claim 1, further comprising:

recording the audio sensed by the microphone;

processing the recorded audio;

detecting a user gesture from the processed audio; and

responsive to the detection of the user gesture, generating a programmatic event.

3. The method of claim 2, further comprising determining the detected gesture to be one of a plurality of user gesture types.

4. The method of claim 3, wherein determining the gesture type further comprises:

evaluating energy content of the recorded signal;

evaluating a spectral centroid of the recorded signal;

classifying the recorded signal; and

identifying the classified recorded signal as one of a plurality of user gesture types.

5. The method of claim 4, wherein classifying the recorded signal further comprises classifying the recorded signal using a machine learning algorithm.

6. The method of claim 4, further comprising training a classifier to perform the classifying step.

7. The method of claim 6, wherein training the classifier further comprises:

receiving from the mobile communication device an audio stream that is recorded from the microphone;

calculating characteristic features of the audio stream;

forming a feature matrix record comprising the calculated characteristic features and a gesture type;

adding the feature matrix record to a training set database; and

providing the training set database to the classifier.

8. The method of claim 2, wherein generating a programmatic event responsive to the detection of the user gesture further comprises mapping the programmatic event to one of a plurality of user-defined actions.

9. The method of claim 2, wherein processing the recorded audio further comprises transmitting an audio file containing the recorded audio to an external host.

11. The system of claim 1, wherein the processor is further configured by the executable instructions to:

record the audio sensed by the microphone;

process the recorded audio;

detect a user gesture from the processed audio; and

responsive to the detection of the user gesture, generate a programmatic event.

12. The system of claim 11, wherein the processor is further configured by the executable instructions to:

determine the detected gesture to be one of a plurality of user gesture types.

13. The system of claim 11, wherein the processor is further configured by the executable instructions to determine the gesture type by:

evaluating energy content of the recorded signal;

evaluating a spectral centroid of the recorded signal;

classifying the recorded signal; and

identifying the classified recorded signal as one of a plurality of user gesture types.

14. The system of claim 13, wherein the processor is further configured by the executable instructions to detect the gesture by

determining whether the total energy in the frame equals or exceeds the current threshold value for gesture detection.

15. The system of claim 13, wherein the processor is further configured by the executable instructions to evaluate the energy content by:

calculating a total energy content of half-rectified samples of the frame;

updating a current value of the threshold for gesture detection by calculating a moving average of total energy content, wherein the moving average is calculated as the average of the total energy of the frame and total energies of a plurality of previous frames; and

replacing the current threshold value for gesture detection with the calculated moving average of total energy value.

10. A system for detecting user gestures with a personal mobile communication device, the system comprising:

a microphone;

memory for storing executable instructions; and

a processor configured by the executable instructions to sense audio using the microphone and to detect user gestures from audio sensed by the microphone.

16. A computer readable medium storing a computer program for use in a personal mobile communication device, the computer program comprising computer executable instructions configured to:

sense audio using a microphone in the personal mobile communication device;

record the audio sensed by the microphone;

process the recorded audio to extract a feature;

classify the feature of the recorded audio; and

identify one of a plurality of user gesture types based on the classified feature.

17. The computer readable medium of claim 16, wherein processing the recorded audio to extract a feature further comprises:

evaluating energy content of the recorded signal; and

evaluating a spectral centroid of the recorded signal;

18. The computer readable medium of claim 16, wherein processing the recorded audio to extract a feature further comprises:

evaluating energy content of the recorded audio; and

wherein classifying the feature further comprises determining a position of impact relative to the location of the microphone.

19. The computer readable medium of claim 16, wherein identifying one of a plurality of gesture types further comprises determining if a continuous user gesture has occurred.

20. The computer readable medium of claim 16, wherein identifying one of a plurality of gesture types further comprises:

determining a number of zero-crossings in a current frame of recorded audio;

comparing the number of zero-crossings to one or more characteristic values corresponding to one or more material types; and

responsive to the comparison of the number of zero-crossing with the characteristic values corresponding to the material types, determining whether an impact occurred with a material belonging to the one or more material types.