MANAGING APPLICATION INTERACTIONS USING DISTRIBUTED MODALITY COMPONENTS

• US 20080244059A1
• Filed: 06/09/2008
• Published: 10/02/2008
• Est. Priority Date: 12/19/2003
• Status: Active Grant

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: McCobb, Gerald M., Bou-Ghannam, Akram A.
• Granted Patent: US 8,095,939 B2
• Time in Patent Office: Days
• US Class Current: 709/223
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces

Abstract

A method for managing multimodal interactions can include the step of registering a multitude of modality components with a modality component server, wherein each modality component handles an interface modality for an application. The modality component can be connected to a device. A user interaction can be conveyed from the device to the modality component for processing. Results from the user interaction can be placed on a shared memory are of the modality component server.

Claims

1. A modality component server for managing distributed multimodal interactions comprising:

a modality activator configured to dynamically activate at least one modality component responsive to an occurrence of an application event initiated by a multimodal application from a client device, connecting said activated modality component to said client device and for disconnecting the activated modality component from the client device and deactivating the modality component upon completion of an interaction response; and

a multimodal engine including an inference engine, and a shared memory area for storing a set of activation conditions registered by each modality component, wherein the sent of activation conditions defines how the modality component is activated, and how input and output between the modality component and the client device is started and stopped, wherein the modality component is activated whenever one of the set of activation conditions is detected, and wherein activation conditions are added or removed by an application module, said multimodal engine being configured to detect an interaction and to responsively initiate an interaction response by comparing activation conditions submitted by the multimodal application with the stored activation conditions in the shared memory area, wherein said interaction and said interaction response have been specified by a previously registered modality component.

2. The server of claim 1, wherein a plurality of modality components are simultaneously utilized, wherein said plurality of modality components specify a plurality of interactions and associated interaction responses, and wherein said multimodal engine is configured to detect any of said plurality of interactions and to responsively initiate an programmatic action.

3. The server of claim 1, wherein said client lacks sufficient resources to locally execute one or more functions of said multimodal application.

4. The server of claim 1, wherein said multimodal engine is further configured to manage multimodal interactions involving multiple modality components.

5. A machine-readable storage having stored thereon, a computer program having a plurality of code sections, said code sections executable by a machine for causing the machine to perform the steps of:

registering a plurality of distributed modality components with a modality component server, wherein each modality component handles an interface modality for an application, wherein each modality component places a set of activation conditions in a shared memory area of the modality component server, wherein the set of activation conditions defines how the modality component is activated, and how input and output between the modality component and a client device are started and stopped, wherein the modality component is activated whenever one of the set of activation conditions is detected, and wherein activation conditions are added or removed by an application module;

initiating a multimodal application from a client device, the multimodal application submitting activation conditions for modality components it supports to a multimodal engine of the modality component server;

matching the activation conditions submitted by the multimodal application with activation conditions stored in the shared memory area by an inference engine of the modality component server;

activating a modality component by a modality activator of the modality component server when one of the set of activation conditions for said modality component is satisfied;

connecting said activated modality component to said client device;

conveying a user interaction from the client device to the activated modality component for processing; and

disconnecting the activated modality component from the client device and deactivating the modality component by the modality activator upon completion of an interaction response.

6. The machine-readable storage of claim 5, said method further comprising the step of:

placing results from said user interaction onto a shared memory area of said modality component server.

7. The machine-readable storage of claim 5, wherein at least one operation of the modality component is fired when one of said activation conditions is detected.

8. The machine-readable storage of claim 5, wherein at least one of said plurality of modality components is remotely located from said client device.

9. The machine-readable storage of claim 8, wherein said client device lacks available resources to locally execute at least one function that is handled by the remotely located modality component.

10. The machine-readable storage of claim 8, wherein at least one of said plurality of modality components is disposed within said client device.

11. A system for managing distributed multimodal interactions comprising:

a processor means for registering a plurality of modality components, wherein each modality component handles an interface modality for an application, wherein each modality component places a set of activation conditions in a shared memory area of the modality component server, wherein the set of activation conditions defines how the modality component is activated, and how input and output between the modality component and a client device are started and stopped, wherein the modality component is activated whenever one of the set of activation conditions is detected, and wherein activation conditions are added or removed by an application module;

means for receiving activation conditions for modality components supported by a client device, the activation conditions being submitted by the client device;

means for comparing the activation conditions submitted by the client device with the activation conditions stored in the shared memory area of the modality component server;

means for activating a modality component when activation conditions for said modality component are satisfied;

means for connecting said activated modality component to said client device;

means for conveying a user interaction from the client device to the activated modality component for processing; and

means for disconnecting the activated modality component from the client device and deactivating the modality component upon completion of an interaction response.