System and process for controlling electronic components in a ubiquitous computing environment using multimodal integration

• US 7,596,767 B2
• Filed: 06/17/2005
• Issued: 09/29/2009
• Est. Priority Date: 02/07/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wilson, Andrew
• Primary Examiner(s): HAILU, TADESSE
• Application Number:
  US11/156,873
• Publication Number:
  US 20050257173A1
• Time in Patent Office: 1,565 Days
• Field of Search: 715/863, 715/866, 715/771, 704/275, 704/270.1, 709/207
• US Class Current: 715/863
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G08C 17/00 Arrangements for transmitting signals characterised by the use of a wireless electrical link
• CPC Class Codes: G08C 2201/31 Voice input
• CPC Class Codes: G08C 2201/32 Remote control based on movements, attitude of remote control device
• CPC Class Codes: G08C 2201/41 Remote control of gateways
• CPC Class Codes: G08C 2201/50 Receiving or transmitting feedback, e.g. replies, status updates, acknowledgements, from the controlled devices

Abstract

The present invention is directed toward a system and process that controls a group of networked electronic components using a multimodal integration scheme in which inputs from a speech recognition subsystem, gesture recognition subsystem employing a wireless pointing device and pointing analysis subsystem also employing the pointing device, are combined to determine what component a user wants to control and what control action is desired. In this multimodal integration scheme, the desired action concerning an electronic component is decomposed into a command and a referent pair. The referent can be identified using the pointing device to identify the component by pointing at the component or an object associated with it, by using speech recognition, or both. The command may be specified by pressing a button on the pointing device, by a gesture performed with the pointing device, by a speech recognition event, or by any combination of these inputs.

Claims

1. A multimodal system for controlling electronic components, comprising:

a general purpose computing system which is in communication with said electronic components via a computer network, said electronic components being separate from the computing system;

a computer program comprising program modules executable by the computing system, said program modules comprising;

an object selection module that identifies an object selected by a user via a pointing device associated with at least one camera and at least one light-emitting diode (LED),

a gesture recognition module that recognizes one or more motions of the pointing device in three-dimensional space, the pointing device associated with at least one accelerometer, and

a speech control module that identifies a component selected by a user,

each of the object selection module, the gesture recognition module, and the speech control module providing inputs to an integration module that integrates said inputs to arrive at a unified interpretation of what object the user wants to control and what control action is desired.

2. The system of claim 1, wherein the integration module comprises a dynamic Bayes network which determines from the individual inputs of the object selection module, gesture recognition module, and speech control module, the identity of a component the user wants to control (i.e., the referent), a command that the user wishes to implement (i.e., the command), and the appropriate control action to be taken to affect the identified referent in view of the command.

3. A computer-implemented multimodal electronic component control process comprising:

a pointer-based object selection process module,

a gesture recognition process module that recognizes one or more motions of a pointing device in three-dimensional space, the pointing device being associated with one or more accelerometers, and

a speech control process module that identifies a command a user desires to implement,

each of the pointer-based object selection process module, the gesture recognition process module, and the speech control process module providing inputs to an integration process module that integrates the inputs to arrive at a unified interpretation of what component a user wants to control and what control action is desired, wherein the electronic component being controlled is separate from a computer system implementing the multimodal electronic component control process and is in communication with the computer system via a computer network.

4. The process of claim 3, wherein integration process module is a dynamic Bayes network.

5. The process of claim 4, wherein the dynamic Bayes network determines from the individual inputs of the object selection module, gesture recognition module, and speech control module, the identity of a component the user wants to control (i.e., the referent), a command that the user wishes to implement (i.e., the command), and the appropriate control action to be taken to affect the identified referent in view of the command.

6. A computer-readable storage medium having computer-executable instructions for causing a computer system to control electronic components using multimodal integration, said computer-executable instructions comprising:

accepting inputs from;

a pointer-based object selection process module that identifies an electronic component selected by a user via a pointing device in association with at least one camera and at least one light-emitting diode that are used for identifying the electronic component selected by the user,

a gesture recognition process module that recognizes one or more motions performed by the user in three-dimensional space via the pointing device in association with at least one accelerometer that is used for detecting motion of the pointing device, and

a speech control process module that identifies the electronic component the user desires to manipulate and a command the user desires to implement;

integrating said inputs from the pointer-based object selection process module and the speech control process module to arrive at a unified interpretation of what electronic component the user wants to control; and

integrating said inputs from the gesture recognition process module and the speech control process module to arrive at a unified interpretation of what control action is desired, wherein the electronic components being controlled are separate from the computer system but are in communication with the computer system via a computer network.

7. The computer-readable storage medium of claim 6, wherein the instruction for accepting inputs, comprises a sub-instruction for inputting from the object selection process module an indication as to what electronic component a user wants to affect via a network connection to a host computer based on what object a pointer is being pointed at, wherein the object corresponds to, or is associated with, the indicated electronic component.

8. The computer-readable storage medium of claim 6, wherein the instruction for accepting inputs, comprises a sub-instruction for inputting from the object selection process module an indication as to whether a user-activated switch on the pointer has been activated.

9. The computer-readable storage medium of claim 6, wherein the instruction for accepting inputs, comprises a sub-instruction for inputting from the gesture recognition process module an indication of what command the user wants implemented based on a gesture performed by the user.

10. The computer-readable storage medium of claim 6, wherein the instruction for accepting inputs, comprises a sub-instruction for inputting from the speech control process module an indication as to what electronic component a user wants to affect via a network connection to a host computer based on a word or phase spoken by the user.

11. The computer-readable storage medium of claim 6, wherein the instruction for accepting inputs, comprises a sub-instruction for inputting from the speech control process module an indication as to what command the user wants implemented based on a word or phase spoken by the user.

12. A computer-readable storage medium having computer-executable instructions for causing a computer system to control an object associated with an electronic component using multimodal integration, said computer-executable instructions comprising:

accepting inputs from;

a pointer-based object selection module that identifies the object selected by a user via a pointing device in three-dimensional space,

a gesture recognition module that recognizes one or more motions of the pointing device in three-dimensional space, and

a speech control module that recognizes speech indicating the object a user desires to manipulate and a command the user desires to implement,

wherein the pointing device is associated with one or more cameras, one or more light-emitting diodes (LEDs), and one or more accelerometers;

integrating said inputs from the pointer-based object selection module and the speech control module to arrive at a unified interpretation of what object the user wants to control; and

integrating said inputs from the gesture recognition module and the speech control module to arrive at a unified interpretation of what control action is desired by the user.