Managing application interactions using distributed modality components

• US 8,352,962 B2
• Filed: 12/22/2011
• Issued: 01/08/2013
• Est. Priority Date: 12/19/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Bou-Ghannam, Akram A., McCobb, Gerald M.
• Primary Examiner(s): CAO, DIEM K
• Application Number:
  US13/334,308
• Publication Number:
  US 20120096108A1
• Time in Patent Office: 383 Days
• Field of Search: 719/318, 719/320, 704/270.1, 704/231
• US Class Current: 719/318
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces

Abstract

A method for managing multimodal interactions can include the step of registering a multitude of modality components with a modality component server, wherein each modality component handles an interface modality for an application. The modality component can be connected to a device. A user interaction can be conveyed from the device to the modality component for processing. Results from the user interaction can be placed on a shared memory are of the modality component server.

Claims

1. A method for managing distributed multimodal interactions comprising:

registering a plurality of distributed modality components with a modality component server, wherein each modality component handles an interface modality for one or more applications, wherein each modality component places a set of activation conditions in a shared memory area of the modality component server, wherein the set of activation conditions defines how the modality component is activated and how input and output between the modality component and a client device is started and stopped, wherein activation conditions are added and/or removed by an application module;

receiving, from a first multimodal application, activation conditions for at least one of the modality components that the first multimodal application supports;

matching the activation conditions submitted by the first multimodal application with activation conditions stored in the shared memory area by an inference engine of the modality component server;

activating a modality component, by a modality activator of the modality component server, when the set of activation conditions for said modality component is satisfied by the activation conditions submitted by the first multimodal application;

connecting said activated modality component to a client device on which the first multimodal application is executing; and

disconnecting the activated modality component from the client device and deactivating the modality component by the modality activator upon completion of an interaction response.

2. The method of claim 1, wherein at least one of the plurality of distributed modality components provides a speech based modality.

3. The method of claim 1, wherein the modality component server comprises an interface that converts data conveyances from a format compatible with the activated modality component to a format compatible with the first multimodal application.

4. The method of claim 1, wherein at least one of the plurality of distributed modality components provides a speech input modality.

5. The method of claim 1, wherein at least one of the plurality of distributed modality components provides a synthetic speech generation modality.

6. The method of claim 1, wherein the client device comprises a cellular telephone, and wherein the modality component server comprises an interface that is compatible with the cellular telephone.

7. At least one non-transitory computer readable medium encoded with instructions that, when executed by a computer system, perform a method for managing distributed multimodal interactions, the method comprising:

registering a plurality of distributed modality components with a modality component server, wherein each modality component handles an interface modality for one or more applications, wherein each modality component places a set of activation conditions in a shared memory area of the modality component server, wherein the set of activation conditions defines how the modality component is activated and how input and output between the modality component and a client device is started and stopped, wherein activation conditions are added and/or removed by an application module;

receiving, from a first multimodal application, activation conditions for at least one of the modality components that the first multimodal application supports;

matching the activation conditions submitted by the first multimodal application with activation conditions stored in the shared memory area by an inference engine of the modality component server;

activating a modality component, by a modality activator of the modality component server, when the set of activation conditions for said modality component is satisfied by the activation conditions submitted by the first multimodal application;

connecting said activated modality component to a client device on which the first multimodal application is executing; and

disconnecting the activated modality component from the client device and deactivating the modality component by the modality activator upon completion of an interaction response.

8. The at least one non-transitory computer readable medium of claim 7, wherein at least one of the plurality of distributed modality components provides a speech based modality.

9. The at least one non-transitory computer readable medium of claim 7, wherein at least one of the plurality of distributed modality components provides a speech input modality.

10. The at one non-transitory computer readable medium of claim 7, wherein the client device comprises a cellular telephone, and wherein the modality component server comprises an interface that is compatible with the cellular telephone.

11. The at least one non-transitory computer readable medium of claim 7, wherein the modality component server comprises an interface that converts data conveyances from a format compatible with the activated modality component to a format compatible with the first multimodal application.

18. The apparatus of claim 7, wherein at least one of the plurality of distributed modality components provides a synthetic speech generation modality.

12. An apparatus for managing distributed multimodal interactions, the apparatus comprising:

at least one communication interface;

at least one memory to store processor-executable instructions; and

at least one processor coupled to the at least one communication interface and the at least one memory, wherein upon execution of the processor-executable instructions by the at least one processor, the at least one processor;

A) registers a plurality of distributed modality components with a modality component server, wherein each modality component handles an interface modality for one or more applications, wherein each modality component places a set of activation conditions in a shared memory area of the modality component server, wherein the set of activation conditions defines how the modality component is activated and how input and output between the modality component and a client device is started and stopped, wherein activation conditions are added and/or removed by an application module;

B) controls the at least one communication interface to receive, from a first multimodal application, activation conditions for at least one of the modality components that the first multimodal application supports;

C) matches the activation conditions submitted by the first multimodal application with activation conditions stored in the shared memory area by an inference engine of the modality component server;

D) activates a modality component, by a modality activator of the modality component server, when the set of activation conditions for said modality component is satisfied by the activation conditions submitted by the first multimodal application;

E) controls the at least one communication interface to connect said activated modality component to a client device on which the first multimodal application is executing;

F) controls the at least one communication interface to disconnect the activated modality component from the client device; and

G) deactivates the modality component by the modality activator upon completion of an interaction response.

13. The apparatus of claim 12, wherein at least one of the plurality of distributed modality components provides a speech based modality.

14. The apparatus of claim 12, wherein at least one of the plurality of distributed modality components provides a speech input modality.

15. The apparatus of claim 12, wherein at least one of the plurality of distributed modality components provides a synthetic speech generation modality.

16. The apparatus of claim 12, wherein the client device comprises a cellular telephone, and wherein the modality component server comprises an interface that is compatible with the cellular telephone.

17. The apparatus of claim 12, wherein the modality component server comprises an interface that converts data conveyances from a format compatible with the activated modality component to a format compatible with the first multimodal application.