Identifying gestures using multiple sensors

• US 9,063,704 B2
• Filed: 05/04/2012
• Issued: 06/23/2015
• Est. Priority Date: 05/05/2011
• Status: Expired due to Fees

• Current Assignee: Wickr Inc. (Amazon.com, Inc.)
• Original Assignee: Net Power & Light, Inc (Amazon.com, Inc.)
• Inventors: Vonog, Stanislav, Surin, Nikolay, Dubovik, Gleb
• Primary Examiner(s): Patel, Kumar
• Assistant Examiner(s): Ahn, Sejoon
• Application Number:
  US13/464,152
• Publication Number:
  US 20120280905A1
• Time in Patent Office: 1,145 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry

Abstract

Systems and methods for recognizing human gestures are disclosed. In one embodiment, a method for recognizing a gesture made by an operator with a portable device, may comprise: obtaining a first sensor data profile associated with measurements made by the first sensor while the operator made a specific gesture involving the portable device; obtaining a second sensor data profile associated with measurements made by the second sensor while the operator made the specific gesture involving the portable device; and identifying the specific gesture by analyzing the first sensor data profile and the second sensor data profile.

Claims

1. A computer implemented method for recognizing a clapping gesture made by an operator with a portable device and generating an applause response, the portable device including at least a kinematic sensor and an audio sensor, the computer implemented method comprising:

obtaining a kinematic sensor data profile associated with measurements made by the kinematic sensor while the operator performs a specific gesture involving the portable device during a time window having a beginning and end;

wherein, the specific gesture includes the operator holding the portable device in a palm of a first hand and clapping the portable device into a palm of a second hand;

obtaining an audio sensor data profile associated with measurements made by the audio sensor while the operator performs the specific gesture involving the portable device;

wherein, the audio sensor data profile corresponds with a substantially identical time window as the kinematic sensor data profile;

analyzing the kinematic sensor data profile and the audio sensor data profile;

wherein the analyzing comprises;

normalizing the kinematic sensor data profile and audio sensor data profile;

comparing the normalized kinematic sensor data profile and normalized audio sensor data profile against a database of known sensor data profiles;

wherein, the database of known sensor data profiles comprises at least a first known sensor data profile known to correspond to a clapping gesture, the first known sensor data profile is characterized by an acceleration impulse near the beginning of the time window, a deceleration impulse near the end of the time window, and an audio signal impulse near the end of the time window;

determining that the specific gesture corresponds to a clapping gesture based on the analyzing of the kinematic sensor data profile and the audio sensor data profile; and

in response to determining that the specific gesture corresponds to a clapping gesture, generating an applause response;

wherein, the applause response comprises an audio output component; and

wherein, the audio output component of the applause response is based on the kinematic sensor data profile and the audio sensor data profile.

2. A computer implemented method as recited in claim 1, further comprising:

generating a gesture pattern for the operator according to multiple distinct inputs of gestures from the operator.

3. A computer implemented method as recited in claim 2, wherein the gesture pattern is dynamically constructed or expanded by analyzing the operator'"'"'s gesture pattern over time using predefined machine learning algorithms.

4. A computer implemented method as recited in claim 3, further comprising:

storing the operator'"'"'s gesture pattern on the portable device and/or a cloud computing service.

5. A computer implemented method as recited in claim 1, wherein the kinematic sensor is an accelerometer, and the audio sensor is a microphone.

6. A computer implemented method as recited in claim 1, wherein the portable device further comprises a pressure sensor.

7. A computer implemented method as recited in claim 6, the method further comprising:

obtaining a pressure sensor data profile associated with measurements made by the pressure sensor while the operator performs the specific gesture involving the portable device;

analyzing the pressure sensor data profile, wherein the analyzing comprises;

normalizing the pressure sensor data profile; and

comparing the normalized pressure sensor data profile against the database of known sensor data profiles;

wherein, the database of known sensor data profiles includes at least a second known sensor data profile known to correspond to a clapping gesture, the second known sensor data profile characterized by an acceleration impulse near the beginning of the time window, a deceleration impulse near the end of the time window, an audio signal impulse near the end of the time window, and a pressure impulse near the end of the time window; and

determining that the specific gesture corresponds to a clapping gesture based on the analyzing of the pressure sensor data profile, kinematic sensor data profile, and the audio sensor data profile.

8. A system for recognizing clapping gestures, the system comprising:

a portable device including at least a kinematic sensor and an audio sensor; and

a gesture recognition process executing on the system, the gesture recognition process operable to;

obtain a kinematic sensor data profile associated with measurements made by the kinematic sensor while an operator performs a specific gesture involving the portable device during a time window having a beginning and an end;

wherein, the specific gesture includes the operator holding the portable device in a palm of a first hand and clapping the portable device into a palm of a second hand;

obtain an audio sensor data profile associated with measurements made by the audio sensor while the operator performs the specific gesture involving the portable device;

wherein, the audio sensor data profile corresponds with a substantially identical time window as the kinematic sensor data profile;

analyze the kinematic sensor data profile and the audio sensor data profile, wherein the analysis comprises;

normalizing the kinematic sensor data profile and audio sensor data profile; and

comparing the normalized kinematic sensor data profile and normalized audio sensor data profile against a database of known sensor data profiles;

wherein, the database of known sensor data profiles includes at least a first known sensor data profile known to correspond to a clapping gesture, the first known sensor data profile characterized by an acceleration impulse near the beginning of the time window, a deceleration impulse near the end of the time window, and an audio signal impulse near the end of the time window;

determine that the specific gesture corresponds to a clapping gesture based on the analysis of the kinematic sensor data profile and the audio sensor data profile; and

in response to determining that the specific gesture corresponds to a clapping gesture, generate an applause response;

wherein the applause response comprises an audio output component;

wherein the audio output component of the applause response is based on the kinematic sensor data profile and the audio sensor data profile.

9. A system for recognizing clapping gestures as recited in claim 8, wherein the gesture recognition process executes on the portable device.

10. A system for recognizing clapping gestures as recited in claim 8, wherein the gesture recognition process is distributed across multiple computing devices and/or a cloud computing service.

11. A system for recognizing clapping gestures as recited in claim 8, wherein the gesture recognition process is further operable to generate a gesture pattern for the operator according to multiple distinct inputs of gestures from the operator.

12. A system for recognizing clapping gestures as recited in claim 11, wherein the gesture pattern is dynamically constructed or expanded by analyzing the operator'"'"'s gesture pattern over time using predefined machine learning algorithms.

13. A system for recognizing clapping gestures as recited in claim 12, wherein the gesture recognition process is further operable to store the operator'"'"'s gesture pattern on the portable device, multiple computing devices, and a cloud computing service.

14. A system for recognizing clapping gestures as recited in claim 8, wherein the kinematic sensor is an accelerometer, and the audio sensor is a microphone.

15. A system for recognizing clapping gestures as recited in claim 8, wherein the portable device further comprises a pressure sensor.

16. A system for recognizing clapping gestures as recited in claim 15, wherein the gesture recognition process is further operable to:

obtain a pressure sensor data profile associated with measurements made by the pressure sensor while the operator performs the specific gesture involving the portable device;

wherein, the pressure sensor profile corresponds with a substantially identical time window as the kinematic sensor data profile;

analyze the pressure sensor data profile, wherein the analysis comprises;

normalizing the pressure sensor data profile; and

comparing the normalized pressure sensor data profile against the database of known sensor data profiles;

wherein, the database of known sensor data profiles includes at least a second known sensor data profile known to correspond to a clapping gesture, the second known sensor data profile characterized by an acceleration impulse near the beginning of the time window, a deceleration impulse near the end of the time window, an audio signal impulse near the end of the time window, and a pressure impulse near the end of the time window; and

determine that the specific gesture corresponds to a clapping gesture based on the analysis of the pressure sensor data profile, kinematic sensor data profile, and the audio sensor data profile.

17. A system for recognizing clapping gestures as recited in claim 8, wherein the system is configured to provide the operator a set of predefined functions of gesture recognition sensitivity according the operator'"'"'s physical characteristics.