System and process for controlling electronic components in a ubiquitous computing environment using multimodal integration

• US 20050257174A1
• Filed: 07/19/2005
• Published: 11/17/2005
• Est. Priority Date: 02/07/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wilson, Andrew
• Granted Patent: US 7,552,403 B2
• Time in Patent Office: Days
• US Class Current: 715/863
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G08C 17/00 Arrangements for transmitting signals characterised by the use of a wireless electrical link
• CPC Class Codes: G08C 2201/31 Voice input
• CPC Class Codes: G08C 2201/32 Remote control based on movements, attitude of remote control device
• CPC Class Codes: G08C 2201/41 Remote control of gateways
• CPC Class Codes: G08C 2201/50 Receiving or transmitting feedback, e.g. replies, status updates, acknowledgements, from the controlled devices

Abstract

The present invention is directed toward a system and process that controls a group of networked electronic components using a multimodal integration scheme in which inputs from a speech recognition subsystem, gesture recognition subsystem employing a wireless pointing device and pointing analysis subsystem also employing the pointing device, are combined to determine what component a user wants to control and what control action is desired. In this multimodal integration scheme, the desired action concerning an electronic component is decomposed into a command and a referent pair. The referent can be identified using the pointing device to identify the component by pointing at the component or an object associated with it, by using speech recognition, or both. The command may be specified by pressing a button on the pointing device, by a gesture performed with the pointing device, by a speech recognition event, or by any combination of these inputs.

Claims

1-57. -57. (canceled)

58. A computer-implemented process for controlling a user-selected electronic component within an environment using a pointing device, comprising:

a computing step for computing the similarity between an input sequence of sensor values output by the pointing device and recorded over a prescribed period of time and at least one stored prototype sequence, wherein each prototype sequence represents the sequence of said sensor values that are generated if the user performs a unique gesture representing a different control action for the selected electronic component using the pointing device;

a determining step for determining if the computed similarity between the input sequence and any prototype sequence exceeds a prescribed similarity threshold; and

an implementing step for implementing a command represented by the gesture which is performed whenever it is determined that one of the computed similarities exceeds the similarity threshold.

59. A computer-implemented process for controlling a user-selected electronic component within an environment using a pointing device, comprising:

an inputting step for inputting orientation messages transmitted by the pointing device, said orientation messages comprising orientation sensor readings generated by orientation sensors of the pointing device;

a recording step for recording a prescribed one or ones of the pointing device sensor outputs taken from an orientation message inputted whenever a user indicates the performance of a gesture representing a control action for the selected electronic component is taking place;

a determining step for determining whether a threshold if just one, or all the thresholds if more than one, of a gesture threshold definition under consideration are exceeded by the recorded sensor output associated with the same sensor output as the threshold for each gesture threshold definition assigned to the selected electronic component;

a designating step for designating that the user has performed the gesture associated with a gesture threshold definition whenever it is determined that the threshold if just one, or all the thresholds if more than one, of one of the gesture threshold definitions are exceeded by the recorded sensor output associated with the same sensor output; and

an implementing step for implementing the command represented by the gesture that the user was designated to have performed.

60. The process of claim 59, wherein the recording step comprises:

(a) a waiting step for waiting for an orientation message to be received;

(b) a switch state determining step for determining whether a switch state indicates that a pointing device switch has been activated by the user to indicate that a control gesture is being performed, said switch state determining step being executed whenever an orientation message is received;

(c) a repeating step for repeating steps (a) and (b) whenever it is determined that the switch state does not indicate that the pointing devices switch is activated; and

(d) an output recording step for recording the prescribed one or ones of the pointing device sensor outputs taken from the last received orientation message whenever it is determined that the switch state does indicate that the pointing devices switch is activated.

61. The process of claim 59, wherein prior to performing the determining step, performing an establishing step for establishing, for each gesture representing a control action for the selected electronic component, a gesture threshold definition, each of said gesture threshold definitions comprising either (i) a prescribed threshold applicable to a particular single sensor output or (ii) a set of thresholds applicable to a particular group of sensor outputs, which are indicative of the pointing device being moved in a particular direction from a starting point wherein the pointing device is pointed at the object in the environment corresponding to or associated with the selected electronic component.