System and process for controlling electronic components in a ubiquitous computing environment using multimodal integration
First Claim
1. A multimodal system for controlling electronic components, comprising:
- a general purpose computing system which is in communication with said electronic components via a computer network, said electronic components being separate from the computing system;
a computer program comprising program modules executable by the computing system, said program modules comprising;
an object selection module that identifies an object selected by a user via a pointing device associated with at least one camera and at least one light-emitting diode (LED),a gesture recognition module that recognizes one or more motions of the pointing device in three-dimensional space, the pointing device associated with at least one accelerometer, anda speech control module that identifies a component selected by a user,each of the object selection module, the gesture recognition module, and the speech control module providing inputs to an integration module that integrates said inputs to arrive at a unified interpretation of what object the user wants to control and what control action is desired.
1 Assignment
0 Petitions
Accused Products
Abstract
The present invention is directed toward a system and process that controls a group of networked electronic components using a multimodal integration scheme in which inputs from a speech recognition subsystem, gesture recognition subsystem employing a wireless pointing device and pointing analysis subsystem also employing the pointing device, are combined to determine what component a user wants to control and what control action is desired. In this multimodal integration scheme, the desired action concerning an electronic component is decomposed into a command and a referent pair. The referent can be identified using the pointing device to identify the component by pointing at the component or an object associated with it, by using speech recognition, or both. The command may be specified by pressing a button on the pointing device, by a gesture performed with the pointing device, by a speech recognition event, or by any combination of these inputs.
93 Citations
12 Claims
-
1. A multimodal system for controlling electronic components, comprising:
-
a general purpose computing system which is in communication with said electronic components via a computer network, said electronic components being separate from the computing system; a computer program comprising program modules executable by the computing system, said program modules comprising; an object selection module that identifies an object selected by a user via a pointing device associated with at least one camera and at least one light-emitting diode (LED), a gesture recognition module that recognizes one or more motions of the pointing device in three-dimensional space, the pointing device associated with at least one accelerometer, and a speech control module that identifies a component selected by a user, each of the object selection module, the gesture recognition module, and the speech control module providing inputs to an integration module that integrates said inputs to arrive at a unified interpretation of what object the user wants to control and what control action is desired. - View Dependent Claims (2)
-
-
3. A computer-implemented multimodal electronic component control process comprising:
-
a pointer-based object selection process module, a gesture recognition process module that recognizes one or more motions of a pointing device in three-dimensional space, the pointing device being associated with one or more accelerometers, and a speech control process module that identifies a command a user desires to implement, each of the pointer-based object selection process module, the gesture recognition process module, and the speech control process module providing inputs to an integration process module that integrates the inputs to arrive at a unified interpretation of what component a user wants to control and what control action is desired, wherein the electronic component being controlled is separate from a computer system implementing the multimodal electronic component control process and is in communication with the computer system via a computer network. - View Dependent Claims (4, 5)
-
-
6. A computer-readable storage medium having computer-executable instructions for causing a computer system to control electronic components using multimodal integration, said computer-executable instructions comprising:
-
accepting inputs from; a pointer-based object selection process module that identifies an electronic component selected by a user via a pointing device in association with at least one camera and at least one light-emitting diode that are used for identifying the electronic component selected by the user, a gesture recognition process module that recognizes one or more motions performed by the user in three-dimensional space via the pointing device in association with at least one accelerometer that is used for detecting motion of the pointing device, and a speech control process module that identifies the electronic component the user desires to manipulate and a command the user desires to implement; integrating said inputs from the pointer-based object selection process module and the speech control process module to arrive at a unified interpretation of what electronic component the user wants to control; and integrating said inputs from the gesture recognition process module and the speech control process module to arrive at a unified interpretation of what control action is desired, wherein the electronic components being controlled are separate from the computer system but are in communication with the computer system via a computer network. - View Dependent Claims (7, 8, 9, 10, 11)
-
-
12. A computer-readable storage medium having computer-executable instructions for causing a computer system to control an object associated with an electronic component using multimodal integration, said computer-executable instructions comprising:
-
accepting inputs from; a pointer-based object selection module that identifies the object selected by a user via a pointing device in three-dimensional space, a gesture recognition module that recognizes one or more motions of the pointing device in three-dimensional space, and a speech control module that recognizes speech indicating the object a user desires to manipulate and a command the user desires to implement, wherein the pointing device is associated with one or more cameras, one or more light-emitting diodes (LEDs), and one or more accelerometers; integrating said inputs from the pointer-based object selection module and the speech control module to arrive at a unified interpretation of what object the user wants to control; and integrating said inputs from the gesture recognition module and the speech control module to arrive at a unified interpretation of what control action is desired by the user.
-
Specification