RECOGNIZING A MOTION OF A POINTING DEVICE

• US 20080204410A1
• Filed: 05/06/2008
• Published: 08/28/2008
• Est. Priority Date: 02/07/2002
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: WILSON, ANDREW
• Application Number:
  US12/116,049
• Publication Number:
  US 20080204410A1
• Time in Patent Office: Days
• US Class Current: 345/158
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G08C 17/00 Arrangements for transmitting signals characterised by the use of a wireless electrical link
• CPC Class Codes: G08C 2201/31 Voice input
• CPC Class Codes: G08C 2201/32 Remote control based on movements, attitude of remote control device
• CPC Class Codes: G08C 2201/41 Remote control of gateways
• CPC Class Codes: G08C 2201/50 Receiving or transmitting feedback, e.g. replies, status updates, acknowledgements, from the controlled devices

Abstract

The present invention is directed toward a system and process that controls a group of networked electronic components using a multimodal integration scheme in which inputs from a speech recognition subsystem, gesture recognition subsystem employing a wireless pointing device and pointing analysis subsystem also employing the pointing device, are combined to determine what component a user wants to control and what control action is desired. In this multimodal integration scheme, the desired action concerning an electronic component is decomposed into a command and a referent pair. The referent can be identified using the pointing device to identify the component by pointing at the component or an object associated with it, by using speech recognition, or both. The command may be specified by pressing a button on the pointing device, by a gesture performed with the pointing device, by a speech recognition event, or by any combination of these inputs.

Claims

1. A motion recognition apparatus comprising:

transceiver means for receiving a plurality of data messages from a pointing device, the data messages including data associated with a motion of the pointing device;

storage means for storing predetermined data in a first area, the predetermined data associated with a manipulation in a prescribed manner;

control means for saving the received plurality of data messages in a second area of the storage means; and

processing means for processing the plurality of data messages to generate an input sequence associated with the motion of the pointing device, comparing the input sequence with the predetermined data, and determining whether the comparison of the input sequence and the predetermined data is indicative of an event,

wherein an object is manipulated based on the existence of the event.

2. One or more computer-readable storage media in a computing device having computer-useable instructions embodied thereon for performing a method for recognizing a first motion of a pointing device, the method comprising:

receiving sensor data at the computing device, the sensor data being output from the pointing device, the sensor data indicating a first sequence of sensor values generated as a result of one or more sensors detecting the first motion of the pointing device over a period of time;

obtaining a second sequence, the second sequence including at least one value associated with a manipulation occurring in a particular manner;

comparing the sensor values of the first sequence to the at least one value of the second sequence;

identifying whether the comparison of the sensor values of the first sequence and the at least one value of the second sequence indicates that the first motion has occurred in the particular manner; and

manipulating an object based on the identification.

3. The one or more computer-readable storage media of claim 2, wherein identifying further comprises determining that at least one of the sensor values of the first sequence exceeds the at least one value of the second sequence, wherein the object is manipulated on a display screen in a first manner.

4. The one or more computer-readable storage media of claim 2, wherein obtaining further comprises obtaining a plurality of second sequences, at least one of the second sequences including at least one value associated with a manipulation occurring in a particular manner.

5. The one or more computer-readable storage media of claim 4, wherein identifying further comprises determining one of the values that is most closely associated with the first sequence of sensor values.

6. The one or more computer-readable storage media of claim 2, wherein identifying further comprises performing a modification to the at least one value.

7. The one or more computer-readable storage media of claim 6, wherein the modification includes scaling up or scaling down an amplitude of the at least one value.

8. The one or more computer-readable storage media of claim 6, wherein the modification includes time warping the at least one value.

9. The one or more computer-readable storage media of claim 2, wherein the step of obtaining further comprises:

receiving an indication of an activation of a button on the pointing device;

receiving training sensor data at the computing device, the training sensor data being output from the one or more sensors of the pointing device, the training sensor data indicating a training sequence of sensor values output by the one or more sensors associated with a second motion of the pointing device;

receiving an indication of a deactivation of the button on the pointing device;

recording the training sequence of sensor values; and

designating the training sequence of sensor values as the at least one value.

10. The one or more computer-readable storage media of claim 5, wherein determining further comprises computing a similarity indicator for each of the at least one values.

11. The one or more computer-readable storage media of claim 2, wherein the output sensor data comprises acceleration data.

12. The one or more computer-readable storage media of claim 2, further comprising normalizing the output sensor data output from the pointing device.

13. A method for recognizing a motion of a pointing device, the method comprising:

receiving sensor data at a computing device, the sensor data being output from the pointing device, the sensor data indicating a first sequence of values detected by one or more sensors of the pointing device over a period of time, corresponding to the motion of the pointing device;

obtaining stored data, the stored data including at least one value, the at least one value indicating a manipulation in a prescribed manner;

comparing the first sequence of values associated with the output sensor data to the at least one value;

determining from the first sequence of values and the at least one value whether an event has occurred; and

manipulating an object in accordance with a control action associated with the at least one value, if the event has occurred.

14. The method of claim 13, wherein the output sensor data comprises orientation data.

15. The method of claim 13, wherein the output sensor data comprises acceleration data.

16. The method of claim 13, wherein the step of determining further comprises determining an extent of similarity between the first sequence of values and the at least one second value.

17. The method of claim 16, further comprising determining if the extent of similarity indicates that a threshold is exceeded.

18. The method of claim 13, wherein the step of obtaining further comprises:

receiving an indication of an activation of a button on the pointing device;

receiving training sensor data at the computing device, the training sensor data being output from the pointing device, the training sensor data indicating a training sequence of sensor values recorded over a period of time;

receiving an indication of a deactivation of a button on the pointing device;

recording the training sequence of sensor values; and

associating the training sequence of sensor values with the at least one value.

19. The method of claim 13, wherein the step of determining further comprises:

performing a plurality of modifications to the at least one value;

comparing the first sequence of values to each of the plurality of modifications; and

matching the first sequence of values to one of the plurality of modifications.

20. The method of claim 19, wherein the modification includes scaling up or scaling down an amplitude of the at least one value.

21. The method of claim 19, wherein the modification includes time warping the at least one value.

22. A system for recognizing a motion of a pointing device, the system comprising:

a transceiver configured to receive a plurality of data messages from the pointing device, the data messages including data associated with the motion of the pointing device;

a storage system configured to store predetermined data indicating a prescribed manipulation;

a controller configured to save the received plurality of data messages in the storage system; and

a processor configured to process the plurality of data messages to generate an input sequence associated with the motion of the pointing device, compare the input sequence with the predetermined data, and determine whether the comparison of the input sequence and the predetermined data are indicative of an event,

wherein an object is manipulated on a display screen based on the existence of the event.

23. The system of claim 22, wherein the predetermined data is associated with a plurality of prescribed manipulations.

24. The system of claim 22, wherein the event comprises at least a portion of the input sequence exceeding at least one predetermined data value.

25. The system of claim 23, wherein each of the plurality of prescribed manipulations corresponds to a plurality of versions of the predetermined data.

26. The system of claim 25, wherein the processor identifies one of the plurality of versions to determine whether an event has occurred.

27. The system of claim 26, wherein the plurality of versions include scaling up or scaling down an amplitude of the predetermined data.

28. The system of claim 26, wherein the plurality of versions include time warping the predetermined data.