Manipulating an object utilizing a pointing device

• US 10,488,950 B2
• Filed: 04/16/2008
• Issued: 11/26/2019
• Est. Priority Date: 02/07/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wilson, Andrew
• Primary Examiner(s): Harris, Dorothy
• Application Number:
  US12/104,360
• Publication Number:
  US 20080259055A1
• Time in Patent Office: 4,241 Days
• Field of Search: 345179, 345156, 345163
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G08C 17/00 Arrangements for transmitting signals characterised by the use of a wireless electrical link
• CPC Class Codes: G08C 2201/31 Voice input
• CPC Class Codes: G08C 2201/32 Remote control based on movements, attitude of remote control device
• CPC Class Codes: G08C 2201/41 Remote control of gateways
• CPC Class Codes: G08C 2201/50 Receiving or transmitting feedback, e.g. replies, status updates, acknowledgements, from the controlled devices

Abstract

The present invention is directed toward a system and process that controls a group of networked electronic components using a multimodal integration scheme in which inputs from a speech recognition subsystem, gesture recognition subsystem employing a wireless pointing device and pointing analysis subsystem also employing the pointing device, are combined to determine what component a user wants to control and what control action is desired. In this multimodal integration scheme, the desired action concerning an electronic component is decomposed into a command and a referent pair. The referent can be identified using the pointing device to identify the component by pointing at the component or an object associated with it, by using speech recognition, or both. The command may be specified by pressing a button on the pointing device, by a gesture performed with the pointing device, by a speech recognition event, or by any combination of these inputs.

Claims

1. A computer-readable storage device having stored thereon computer-executable instructions comprising:

identifying a position and an orientation of a pointing device in three-dimensional space;

determining that the pointing device is directed to an object based on the position and the orientation of the pointing device in three-dimensional space;

determining that an input sequence of sensor values output by the pointing device matches a matching prototype sequence from a set of stored prototype sequences, wherein each stored prototype sequence represents a sequence of said sensor values that are generated if a user performs a unique gesture representing a different control action using the pointing device, wherein the matching prototype sequence is determined at least by comparing the matching prototype sequence to the input sequence of sensor values and by comparing the input sequence of sensor values to one or more versions of the matching prototype sequence that are scaled up and down in amplitude and warped in time; and

controlling a computer system based on a command and referent pair, wherein the referent is determined based on the position and the orientation of the pointing device in three-dimensional space, and wherein the command is determined based on a gesture associated with the matching prototype sequence.

2. The computer-readable storage device of claim 1, wherein the sensor values are provided by an output of an accelerometer.

3. The computer-readable storage device of claim 1, wherein identifying the position further comprises:

receiving data associated with one or more infrared lights detected by a camera.

4. The computer-readable storage device of claim 3, wherein the data associated with one or more infrared lights detected by the camera describes one or more images that are analyzed to identify the position.

5. The computer-readable storage device of claim 1, wherein the input sequence of sensor values includes sensor values that are provided by an output of a gyroscope.

6. A method for multimodal electronic component control comprising:

identifying a position and an orientation of a pointing device in three-dimensional space;

determining that the pointing device is directed to an object based on the position and the orientation of the pointing device in three-dimensional space;

determining that an input sequence of sensor values output by the pointing device matches a matching prototype sequence from a set of stored prototype sequences, wherein each stored prototype sequence represents a sequence of said sensor values that are generated if a user performs a unique gesture representing a different control action using the pointing device, wherein the matching prototype sequence is determined at least by comparing the matching prototype sequence to the input sequence of sensor values and by comparing the input sequence of sensor values to one or more versions of the matching prototype sequence that are scaled up and down in amplitude and warped in time; and

controlling a computer system based a command and referent pair, wherein the referent is determined based on the position and the orientation of the pointing device in three-dimensional space, and wherein the command is determined based on a gesture associated with the matching prototype sequence.

7. The method of claim 6, wherein identifying the position further comprises:

receiving data associated with one or more infrared lights detected by a camera.

8. The method of claim 7, wherein the data associated with one or more infrared lights detected by the camera describes one or more images that are analyzed to identify the position.

9. The method of claim 6, wherein the sensor values are provided by an output of an accelerometer.

10. The method of claim 6, further comprising:

computing a similarity between the input sequence of sensor values output by the pointing device and each stored prototype sequence;

determining if the computed similarity between the input sequence and any stored prototype sequence exceeds a prescribed similarity threshold; and

whenever it is determined that one of the computed similarities exceeds the similarity threshold, determining that the stored prototype sequence associated with the computed similarity matches the input sequence.

11. A system for multimodal electronic component control comprising:

one or more cameras that capture one or more images of a pointing device in a three-dimensional space;

a processing component configured to perform instructions comprising;

receiving data associated with the one or more images;

identifying a position and an orientation of the pointing device in the three-dimensional space based on the received data;

determining that the pointing device is directed to an object based on the position and the orientation of the pointing device in three-dimensional space;

determining that an input sequence of sensor values output by the pointing device matches a matching prototype sequence from a set of stored prototype sequences, wherein each stored prototype sequence represents a sequence of said sensor values that are generated if a user performs a unique gesture representing a different control action using the pointing device, wherein the matching prototype sequence is determined at least by comparing the matching prototype sequence to the input sequence of sensor values and by comparing the input sequence of sensor values to one or more versions of the matching prototype sequence that are scaled up and down in amplitude and warped in time; and

controlling the system based on a command and referent pair, wherein the referent is determined based on the position and the orientation of the pointing device in three-dimensional space, and wherein the command is determined based on a gesture associated with the matching prototype sequence.

12. The system of claim 11, wherein the sensor values are provided by an output of an accelerometer.

13. The system of claim 11, wherein the data associated with the one or more images comprises data associated with one or more infrared lights detected by the camera.

14. The system of claim 11, wherein the instructions further comprise:

computing a similarity between the input sequence of sensor values output by the pointing device and each stored prototype sequence;

determining if the computed similarity between the input sequence and any stored prototype sequence exceeds a prescribed similarity threshold; and

whenever it is determined that one of the computed similarities exceeds the similarity threshold, determining that the stored prototype sequence associated with the computed similarity matches the input sequence.