Recognizing a movement of a pointing device

• US 9,454,244 B2
• Filed: 05/07/2008
• Issued: 09/27/2016
• Est. Priority Date: 02/07/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wilson, Andrew
• Primary Examiner(s): HAILU, TADESSE
• Application Number:
  US12/116,813
• Publication Number:
  US 20080204411A1
• Time in Patent Office: 3,065 Days
• Field of Search: 715856-863, 715/864, 345/156, 345/158
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G08C 17/00 Arrangements for transmitting signals characterised by the use of a wireless electrical link
• CPC Class Codes: G08C 2201/31 Voice input
• CPC Class Codes: G08C 2201/32 Remote control based on movements, attitude of remote control device
• CPC Class Codes: G08C 2201/41 Remote control of gateways
• CPC Class Codes: G08C 2201/50 Receiving or transmitting feedback, e.g. replies, status updates, acknowledgements, from the controlled devices

Abstract

The present invention is directed toward a system and process that controls a group of networked electronic components using a multimodal integration scheme in which inputs from a speech recognition subsystem, gesture recognition subsystem employing a wireless pointing device and pointing analysis subsystem also employing the pointing device, are combined to determine what component a user wants to control and what control action is desired. In this multimodal integration scheme, the desired action concerning an electronic component is decomposed into a command and a referent pair. The referent can be identified using the pointing device to identify the component by pointing at the component or an object associated with it, by using speech recognition, or both. The command may be specified by pressing a button on the pointing device, by a gesture performed with the pointing device, by a speech recognition event, or by any combination of these inputs.

Claims

1. A movement recognition apparatus comprising:

receiving means for receiving a signal via a wireless medium, the signal originating from a pointing device;

demodulation means for demodulating the received signal;

control means for receiving the demodulated signal and providing a plurality of output data messages, the plurality of output data messages including at least one acceleration data value detected by an accelerometer during a motion of the pointing device;

storage means for storing the plurality of output data messages as well as a plurality of threshold definitions associated with accelerometer output of the pointing device, wherein each of the plurality of threshold definitions includes at least one threshold value utilized to indicate that the motion of the pointing device has occurred, each of the plurality of threshold definitions being associated with a specific respective gesture; and

processing means for processing the plurality of output data messages, wherein the processing includes accessing each of the plurality of threshold definitions, comparing each of the plurality of threshold definitions to the plurality of output data messages, identifying whether the at least one threshold value in any of the plurality of threshold definitions is exceeded by any of the plurality of output data messages, and selecting one of the plurality of threshold definitions, wherein the at least one threshold value in the selected one of the plurality of threshold definitions is exceeded by any of the plurality of output data messages.

2. The movement recognition apparatus of claim 1, wherein each of the plurality of threshold definitions is associated with a specific respective gesture having an associated specified control action.

3. The movement recognition apparatus of claim 1, wherein the plurality of output data messages include an orientation data value generated in response to the accelerometer detecting the motion of the pointing device.

4. The movement recognition apparatus of claim 1, wherein the plurality of output data messages include button activation data.

5. The movement recognition apparatus of claim 1, wherein the plurality of output data messages include button activation data and button deactivation data, wherein the controller is configured to store only the plurality of output data messages that occur after the button activation data but before the button deactivation data.

6. The movement recognition apparatus of claim 1, wherein the processing further includes performing a control action corresponding to the specific respective gesture associated with the selected one of the plurality of threshold definitions.

7. A movement recognition system comprising:

an antenna configured to receive a signal via a wireless medium, the signal originating from a pointing device;

a demodulator configured to demodulate the received signal;

a controller configured to receive the demodulated signal and to provide a plurality of output data messages, each of the plurality of output data messages including at least one acceleration data value generated as a result of an accelerometer detecting a motion of the pointing device;

a first storage media configured to store the plurality of output data messages;

a second storage media configured to store a plurality of threshold definitions associated with accelerometer output of the pointing device, wherein the plurality of threshold definitions each include a threshold value utilized to indicate that the motion of the pointing device has occurred, each of the plurality of threshold definitions being associated with a specific respective gesture; and

a processor configured to process the plurality of output data messages by performing processing including accessing the plurality of threshold definitions, comparing the plurality of threshold definitions to the plurality of output data messages, identifying whether the threshold value in any of the plurality of threshold definitions is exceeded by any of the at least one acceleration data value.

8. The system of claim 7, wherein the first storage media or the second storage media includes instructions for control actions, and wherein the processor further executes the instructions.

9. The system of claim 8, wherein the processor is further configured to manipulate an object in accordance with the one of the instructions for control actions.

10. The system of claim 7, wherein the plurality of output data messages include an orientation data value generated in response to the accelerometer detecting the motion of the pointing device.

11. The system of claim 7, wherein the plurality of output data messages include button activation data.

12. The system of claim 7, wherein the plurality of output data messages include button activation data and button deactivation data, wherein the controller is configured to store only the plurality of output data messages that occur after the button activation data but before the button deactivation data.

13. The system of claim 7, wherein each of the plurality of threshold definitions is associated with a specific respective gesture having an associated specified control action.

14. The system of claim 7, wherein the processor is further configured to select one of the plurality of threshold definitions, wherein the threshold value in the selected one of the plurality of threshold definitions is exceeded by any of the plurality of output data messages.

15. The system of claim 14, wherein the processor is further configured to perform a control action corresponding to the specific respective gesture associated with the selected one of the plurality of threshold definitions.

16. A method for recognizing a movement associated with a pointing device, the method comprising:

receiving a signal via a wireless medium, the signal originating from a pointing device;

demodulating the received signal;

receiving the demodulated signal and providing a plurality of output data messages, the plurality of output data messages including at least one acceleration data value detected by an accelerometer during a motion of the pointing device;

storing the plurality of output data messages as well as a plurality of threshold definitions associated with accelerometer output of the pointing device, wherein each of the plurality of threshold definitions includes at least one threshold value utilized to indicate that the motion of the pointing device has occurred, each of the plurality of threshold definitions being associated with a specific respective gesture; and

processing the plurality of output data messages, wherein the processing includes accessing each of the plurality of threshold definitions, comparing each of the plurality of threshold definitions to the plurality of output data messages, identifying whether the at least one threshold value in any of the plurality of threshold definitions is exceeded by any of the plurality of output data messages, and selecting one of the plurality of threshold definitions, wherein the at least one threshold value in the selected one of the plurality of threshold definitions is exceeded by any of the plurality of output data messages.

17. The method of claim 16, wherein each of the plurality of threshold definitions is associated with a specific respective gesture having an associated specified control action.

18. The method of claim 16, wherein the plurality of output data messages include button activation data.

19. The method of claim 16, wherein the plurality of output data messages include button activation data and button deactivation data, wherein only the plurality of output data messages that occur after the button activation data but before the button deactivation data are stored.

20. The method of claim 16, further comprising performing a control action corresponding to the specific respective gesture associated with the selected one of the plurality of threshold definitions.

21. One or more computer readable storage devices having stored thereon computer executable instructions that, upon execution by one or more processors of a computing system, wherein the one or more computer readable storage devices are not signals, cause the computing system to perform operations comprising:

receiving a signal via a wireless medium, the signal originating from a pointing device;

demodulating the received signal;

receiving the demodulated signal and providing a plurality of output data messages, the plurality of output data messages including at least one acceleration data value detected by an accelerometer during a motion of the pointing device;

storing the plurality of output data messages as well as a plurality of threshold definitions associated with accelerometer output of the pointing device, wherein each of the plurality of threshold definitions includes at least one threshold value utilized to indicate that the motion of the pointing device has occurred, each of the plurality of threshold definitions being associated with a specific respective gesture; and

processing the plurality of output data messages, wherein the processing includes accessing each of the plurality of threshold definitions, comparing each of the plurality of threshold definitions to the plurality of output data messages, identifying whether the at least one threshold value in any of the plurality of threshold definitions is exceeded by any of the plurality of output data messages, and selecting one of the plurality of threshold definitions, wherein the at least one threshold value in the selected one of the plurality of threshold definitions is exceeded by any of the plurality of output data messages.

22. The one or more computer readable storage devices of claim 21, wherein each of the plurality of threshold definitions is associated with a specific respective gesture having an associated specified control action.

23. The one or more computer readable storage devices of claim 21, wherein the plurality of output data messages include button activation data.

24. The one or more computer readable storage devices of claim 21, wherein the plurality of output data messages include button activation data and button deactivation data, wherein only the plurality of output data messages that occur after the button activation data but before the button deactivation data are stored.

25. The one or more computer readable storage devices of claim 21, wherein the operations further comprise performing a control action corresponding to the specific respective gesture associated with the selected one of the plurality of threshold definitions.