IDENTIFYING GESTURES USING MULTIPLE SENSORS

• US 20120280905A1
• Filed: 05/04/2012
• Published: 11/08/2012
• Est. Priority Date: 05/05/2011
• Status: Active Grant

• Current Assignee: Wickr Inc. (Amazon.com, Inc.)
• Original Assignee: Net Power & Light, Inc (Amazon.com, Inc.)
• Inventors: Vonog, Stanislav, Surin, Nikolay, Dubovik, Gleb
• Granted Patent: US 9,063,704 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry

Abstract

Systems and methods for recognizing human gestures are disclosed. In one embodiment, a method for recognizing a gesture made by an operator with a portable device, may comprise: obtaining a first sensor data profile associated with measurements made by the first sensor while the operator made a specific gesture involving the portable device; obtaining a second sensor data profile associated with measurements made by the second sensor while the operator made the specific gesture involving the portable device; and identifying the specific gesture by analyzing the first sensor data profile and the second sensor data profile.

Claims

1. A computer implemented method for recognizing a gesture made by an operator with a portable device, the portable device including at least a first sensor and a second sensor, wherein the first sensor and the second sensor are disparate, the computer implemented method comprising:

obtaining a first sensor data profile associated with measurements made by the first sensor while the operator made a specific gesture involving the portable device;

obtaining a second sensor data profile associated with measurements made by the second sensor while the operator made the specific gesture involving the portable device; and

identifying the specific gesture by analyzing the first sensor data profile and the second sensor data profile.

2. A computer implemented method as recited in claim 1, further comprising:

generating a gesture pattern for the operator according to multiple distinct inputs of gestures from the operator.

3. A computer implemented method as recited in claim 2, wherein the gesture pattern is dynamically constructed or expanded by analyzing the operator'"'"'s gesture pattern over time using predefined machine learning algorithms.

4. A computer implemented method as recited in claim 3, further comprising:

storing the operator'"'"'s gesture pattern on the portable device and/or a cloud computing service.

5. A computer implemented method as recited in claim 1, wherein the first sensor is an accelerometer, and the second sensor is an audio input device.

6. A computer implemented method as recited in claim 5, wherein the specific gesture corresponds to a clapping gesture made by the operator holding the portable device in a first hand and clapping both hands together.

7. A computer implemented method as recited in claim 5, wherein the specific gesture corresponds to a spell incantation involving the operator gesturing with the portable device while speaking an incantation.

8. A computer implemented method as recited in claim 7, wherein analyzing the second sensor data profile includes speech recognition.

9. A computer implemented method as recited in claim 1, wherein the first sensor is an accelerometer, and the second sensor is a pressure sensor, the pressure sensor configured to detect pressure from the operator holding the portable device.

10. A computer implemented method as recited in claim 9, wherein the portable device further includes an audio input device.

11. A system for recognizing human gestures, the system comprising:

a portable device including at least a first sensor and a second sensor; and

a gesture recognition process executing on the system, the gesture recognition process operable to;

obtain a first sensor data profile associated with measurements made by the first sensor while an operator made a specific gesture involving the portable device;

obtain a second sensor data profile associated with measurements made by the second sensor while the operator made the specific gesture involving the portable device; and

identify the specific gesture by analyzing the first sensor data profile and the second sensor data profile.

12. A system for recognizing human gestures as recited in claim 11, wherein the gesture recognition process executes on the portable device.

13. A system for recognizing human gestures as recited in claim 11, wherein the gesture recognition process is distributed across multiple computing devices and/or a cloud computing service.

14. A system for recognizing human gestures as recited in claim 11, wherein the gesture recognition process is further operable to generate a gesture pattern for the operator according to multiple distinct inputs of gestures from the operator.

15. A system for recognizing human gestures as recited in claim 14, wherein the gesture pattern is dynamically constructed or expanded by analyzing the operator'"'"'s gesture pattern over time using predefined machine learning algorithms.

16. A system for recognizing human gestures as recited in claim 15, wherein the gesture recognition process is further operable to store the operator'"'"'s gesture pattern on the portable device, multiple computing devices, and a cloud computing service.

17. A system for recognizing human gestures as recited in claim 11, wherein the first sensor is an accelerometer, and the second sensor is an audio input device.

18. A system for recognizing human gestures as recited in claim 17, wherein the specific gesture corresponds to a clapping gesture made by the operator holding the portable device in a first hand and clapping both hands together.

19. A system for recognizing human gestures as recited in claim 17, wherein the specific gesture corresponds to a spell incantation involving the operator gesturing with the portable device while speaking an incantation.

20. A system for recognizing human gestures as recited in claim 19, wherein analyzing the second sensor data profile includes speech recognition.

21. A system for recognizing human gestures as recited in claim 11, wherein the first sensor is an accelerometer, and the second sensor is a pressure sensor, the pressure sensor configured to detect pressure from the operator holding the portable device.

22. A system for recognizing human gestures as recited in claim 21, wherein the portable device further includes an audio input device.

23. A system for recognizing human gestures as recited in claim 11, wherein the system is configured to provide the operator a set of predefined functions of gesture recognition sensitivity according the operator'"'"'s physical characteristics.

24. A computer implemented method for interfacing a human with a computing device, comprising:

receiving sensor data from a plurality of sensors, wherein at least two of the plurality of sensors are disparate, and wherein the sensor data is associated with an event;

analyzing the sensor data to determine whether the event corresponds to an operator gesture; and

when the event corresponds to the operator gesture, performing or initiating a predefined action indicated by the operator gesture.

25. A computer implemented method for identifying a clapping gesture made by an operator of a portable device having a kinematic sensor and an audio input, the computer implemented method comprising:

analyzing a first profile based on data from the kinematic sensor, the first profile corresponding to a specific event;

analyzing a second profile based on data from the audio input, the second profile corresponding to the specific event; and

determining whether the first profile and the second profile indicate that the specific event is a clapping gesture.

26. A computer implemented method as recited in claim 25, wherein the kinematic sensor is an accelerometer.

27. A computer implemented method as recited in claim 25, wherein the portable device comprises a pressure sensor, the pressure sensor configured to detect a pressure the operator holding the portable device;

the method further comprising;

analyzing a third profile based on data from the pressure sensor, the third profile corresponding to the specific event; and

determining whether the first profile, the second profile, and the third profile indicate that the specific event is the clapping gesture.

28. A computer implemented method for identifying a spell cast by an operator of a portable device having a kinematic sensor and an audio input, the spell having a gesture component and an audio component, the computer implemented method comprising:

analyzing a first profile based on data from the kinematic sensor, the first profile corresponding to a specific event,

analyzing a second profile based on data from the audio input, the second profile corresponding to the specific event;

determining whether the first profile and the second profile indicate that the specific event is a spell; and

when the specific event is determined to be a spell, performing or initiating a predefined action indicated by the spell.

29. A system for recognizing an operator'"'"'s gestures, the system comprising:

a first sensor;

a second sensor; and

a gesture recognition process executing on the system, the gesture recognition process operable to;

obtain a first sensor data profile associated with measurements made by the first sensor corresponding to a specific event;

obtain a second sensor data profile associated with measurements made by the second sensor corresponding to the specific event; and

identifying whether the specific event corresponds to a specific gesture by analyzing the first sensor data profile and the second sensor data profile.

30. A system for recognizing human gestures as recited in claim 29, wherein the first sensor is an accelerometer, and the second sensor is a pressure sensor attached to an operator device, the pressure sensor configured to detect a pressure the operator holding the operator device.

31. A system for recognizing human gestures as recited in claim 30, wherein the system further comprising an audio input device.

32. A system for recognizing human gestures as recited in claim 29, wherein the gesture recognition process executes on an operator device.

33. A system for recognizing human gestures as recited in claim 29, wherein the gesture recognition process is distributed across multiple computing devices and/or a cloud computing service.

34. A system for recognizing human gestures as recited in claim 29, wherein the gesture recognition process is further operable to generate a gesture pattern for the operator according to multiple distinct inputs of gestures from the operator.

35. A system for recognizing human gestures as recited in claim 34, wherein the gesture pattern is dynamically constructed or expanded by analyzing the operator'"'"'s gesture pattern over time using predefined machine learning algorithms.

36. A system for recognizing human gestures as recited in claim 35, wherein the gesture recognition process is further operable to store the operator'"'"'s gesture pattern on an operator device, multiple computing devices, and a cloud computing service.

37. A system for recognizing human gestures as recited in claim 29, wherein the first sensor is an accelerometer, and the second sensor is an audio input device.

38. A system for recognizing human gestures as recited in claim 37, wherein the specific gesture corresponds to a clapping gesture made by the operator holding an operator device in a first hand and clapping both hands together.

39. A system for recognizing human gestures as recited in claim 37, wherein the specific gesture corresponds to a spell incantation involving the operator gesturing with an operator device while speaking an incantation.

40. A system for recognizing human gestures as recited in claim 39, wherein analyzing the second sensor data profile includes speech recognition.

41. A system for recognizing human gestures as recited in claim 29, wherein the system is configured to provide the operator a set of predefined functions of gesture recognition sensitivity according the operator'"'"'s physical characteristics.