Environmental gesture recognition

• US 8,704,767 B2
• Filed: 01/29/2009
• Issued: 04/22/2014
• Est. Priority Date: 01/29/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Dodge, Steven, Wang, Zheng
• Primary Examiner(s): HARRIS, DOROTHY H
• Application Number:
  US12/362,231
• Publication Number:
  US 20100188328A1
• Time in Patent Office: 1,909 Days
• Field of Search: 345/156, 715/863, 710 15- 16, 710/73
• US Class Current: 345/158
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

A data-holding subsystem. The data-holding subsystem includes instructions stored thereon that when executed by a logic subsystem in communication with the data-holding subsystem: receive one or more signals, determine a sensor type for each signal of the one or more signals, identify a sensor type specific pattern corresponding to a motion gesture in at least one of the signals, and generate a gesture message based on the motion gesture. The gesture message may be usable by an operating system of a computing device that includes the data-holding subsystem to provide a system-wide function usable by one or more application programs of the computing device to provide an application specific function.

Claims

1. A data-holding subsystem including instructions stored thereon that when executed by a logic subsystem in communication with the data-holding subsystem:

receive one or more signals from a set of sensors;

responsive to receiving the one or more signals, determine a sensor type for each sensor in the set of sensors, thereby recognizing which of a plurality of different possible sensors are included in the set of sensors;

identify a motion gesture corresponding to a sensor type specific pattern in at least one of the signals, the motion gesture also corresponding to one or more other sensor type specific patterns in signals receivable from different types of sensors; and

generate a gesture message based on the motion gesture, the gesture message being usable by an operating system of a computing device that includes the data-holding subsystem to provide a system-wide function usable by two or more application programs of the computing device to provide an application specific function, the operating system configured to provide the same system-wide function to application programs when the operating system is executed on a different computing device responsive to identifying the one or more other sensor type specific patterns in signals received from the different types of sensors.

2. The data-holding subsystem of claim 1, wherein the sensor type includes one or more of an accelerometer, a gyroscope, a camera, a light sensor, and a touch sensor.

3. The data-holding subsystem of claim 1, wherein the computing device is a mobile computing device and the one or more signals are generated by environmental sensors integrated into the mobile computing device.

4. The data-holding subsystem of claim 3, wherein the motion gesture includes a rotation of the mobile computing device along a relative X-axis of the mobile computing device and the application specific function includes a scrolling function.

5. The data-holding subsystem of claim 3, wherein the motion gesture includes a rotation of the mobile computing device along a relative Y-axis of the mobile computing device and the application specific function includes a change page function.

6. The data-holding subsystem of claim 3, wherein the motion gesture includes a rotation of the mobile computing device along a relative Z-axis of the mobile computing device and the application specific function includes a display rotation function.

7. The data-holding subsystem of claim 3, wherein the motion gesture includes a translation of the mobile computing device along a relative X-axis of the mobile computing device and the system-wide function includes an application switching function.

8. The data-holding subsystem of claim 3, wherein the motion gesture includes a translation of the mobile computing device along a relative Y-axis of the mobile computing device and the application specific function includes an editing function.

9. The data-holding subsystem of claim 3, wherein the motion gesture includes a translation of the mobile computing device along a relative Z-axis of the mobile computing device and the system-wide function includes a zoom function.

10. A computing device comprising:

an extensible gesture recognition engine configured to;

receive two or more different signals from two or more different types of sensors,

responsive to receiving the two or more signals, determine a sensor type for each sensor from which the two or more signals are received,

monitor each signal for sensor type specific patterns that are associated with the sensor type determined for that sensor, the sensor type specific patterns collectively corresponding to different possible composite gestures,

for each of the two or more different signals, in response to identification of a sensor type specific pattern, recognize a possible composite gesture corresponding to that sensor type specific pattern,

determine a confidence level for the possible composite gesture based on heuristics applied to the two or more different signals, and

generate a composite gesture message for the possible composite gesture responsive to the confidence level being above a confidence threshold, the composite gesture message being usable by one or more applications of the computing device to provide an application specific function.

11. The computing device of claim 10, wherein the extensible gesture recognition engine is configured to recognize sensor type specific patterns corresponding to customized gestures defined by a third party software application or a third party hardware device.

12. The computing device of claim 10, wherein the extensible gesture recognition engine is configured to check for degradation of a signal responsive to the confidence level being below the confidence threshold, and upon detection of degradation of the signal, determine a new confidence level based on heuristics applied to non-degraded signals.

13. The computing device of claim 10, wherein at least one of the two or more different signals is received from a sensor of a remote computing device separate from but in communication with the computing device.

14. The computing device of claim 10, wherein a plurality of signals is received over a duration, and the extensible gesture recognition engine is configured to determine usage patterns of the plurality of signals learned for the duration, and the confidence level is further based on the usage patterns.

15. The computing device of claim 10, wherein the composite gesture message is usable by an operating system of the computing device to generate a system-wide function.

16. The computing device of claim 10, wherein the application specific function is mapped to the composite gesture message via an application programming interface.

17. The computing device of claim 16, wherein the sensor type includes one or more of an accelerometer, a gyroscope, a camera, a light sensor, and a touch sensor.

18. A computing device comprising:

an extensible gesture recognition engine comprising;

a recognition module configured to;

receive one or more signals,

determine a sensor type for each signal of the one or more signals,

monitor each signal for sensor type specific patterns that are associated with the sensor type determined for that signal, the sensor type specific patterns corresponding to different gestures,

for each signal, in response to identification of a sensor type specific pattern, generate a gesture corresponding to that sensor type specific pattern,

determine a confidence level for the gesture of each signal based on heuristics applied to the one or more signals, and

generate a gesture message for the gesture responsive to the confidence level being above a confidence threshold, the gesture message being usable by one or more applications of the computing device to provide an application specific function; and

a customization module configured to;

add a new signal to a list of available sensor signals used for gesture recognition in response to receiving the new signal from a sensor in communication with the extensible gesture recognition engine, and

define one or more customized composite gestures based on two or more signals from the list of available sensor signals.

19. The computing device of claim 18, wherein the customization module adds and defines, responsive to input from a third party software application or a third party hardware device.