Application module for managing interactions of distributed modality components

• US 9,201,714 B2
• Filed: 12/23/2010
• Issued: 12/01/2015
• Est. Priority Date: 12/19/2003
• Status: Active Grant

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Bou-Ghannam, Akram A., McCobb, Gerald M.
• Primary Examiner(s): CAO, DIEM K
• Application Number:
  US12/977,671
• Publication Number:
  US 20110093868A1
• Time in Patent Office: 1,804 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 2209/544 Remote
• CPC Class Codes: G06F 2209/545 Gui
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications

Abstract

A method for managing application modalities using dialogue states can include the step of asserting a set of activation conditions associated with a dialogue state of an application. Each of the activation conditions can be linked to at least one programmatic action, wherein different programmatic actions can be executed by different modality components. The application conditions can be monitored. An application event can be detected resulting in an associated application condition being run. At least one programmatic action linked to the application condition can be responsively initiated.

Claims

1. A method of managing a multimodal application executing at least in part on a client device, the client device communicating with at least one server device, the client device being configured to accept input from a user via a plurality of modalities, the method comprising:

detecting, with at least one control module, that input from a user via one or more modalities has been interpreted by at least one first software module;

in response to interpretation of the input, activating, with the at least one control module, at least one second software module to process further input from the user, the at least one second software module being configured to, when executed by at least one processor of the at least one server device, cause the at least one processor to perform a task for processing the further input from the user, the task resulting in a change of a dialog state of the multimodal application from a first dialog state to a second dialog state; and

receiving a notification at the at least one control module, output by the at least one second software module following performance of the task, that the dialog state of the multimodal application is to change from the first dialog state to the second dialog state, the notification received by the control module identifying the second dialog state to which the dialog state of the multimodal application is to change.

2. The method of claim 1, further comprising:

registering the at least one second software module with the at least one control module, the registering comprising identifying at least one condition on which the at least one second software module is to be activated; and

in response to detecting that the input from the user has been interpreted, determining, with the at least one control module, whether the interpretation of the input by the at least one first software module meets the at least one condition on which the at least one second software module is to be activated,

wherein the activating the at least one second software module comprises activating the at least one second software module in response to determining that the interpretation of the input meets the at least one condition.

3. The method of claim 1, further comprising:

receiving the further input from the user via one or more of the plurality of modalities; and

with the at least one control module, triggering execution of the at least one second software module to cause the at least one processor to perform the task for processing the further input, wherein triggering the execution comprises providing the further input to the at least one second software module.

4. The method of claim 3, further comprising:

monitoring, with the at least one control module, for a notification from the at least one second software module that the further input from the user has been interpreted by the at least one second software module; and

in response to the dialog state of the multimodal application changing to the second dialog state as a result of the performance of the task, activating, with the at least one control module, at least one third software module to process third input from the user, the at least one third software module being configured to, when executed by the at least one processor, cause the at least one processor to perform a second task for processing the third input from the user, the second task resulting in a change of a dialog state of the multimodal application from the second dialog state to a third dialog state.

5. The method of claim 4, further comprising:

in response to the dialog state of the multimodal application changing to the third dialog state, deactivating, with the at least one control module, the at least one second software module.

6. The method of claim 4, further comprising:

monitoring, with the at least one control module, for a notification of occurrence of at least one of a set of events, each event in the set of the events being associated with a programmatic action to be performed on occurrence of the event; and

wherein the activating the at least one second software module comprises adding at least one event to the set of events, the at least one event comprising an interpretation of the further input from the user and a change of a dialog state of the multimodal application from the first dialog state to a second dialog state, the programmatic action associated with the interpretation of the input being activation of the at least one third software module.

7. The method of claim 1, wherein the activating the at least one second software module comprises instantiating the at least one second software module.

8. The method of claim 1, wherein the activating the at least one software module comprises activating a server-side module on the at least one server device and activating a client-side module on the client device, the server-side module being configured to communicate with the client-side module to receive the further input from the user and to perform the task.

9. At least one non-transitory computer-readable storage medium having encoded thereon computer-executable instructions that, when executed by at least one processor, cause the at least one processor to carry out a method of managing a multimodal application executing at least in part on a client device, the client device communicating with at least one server device, the client device being configured to accept input from a user via a plurality of modalities, the method comprising:

detecting, with at least one control module, that input from a user via one or more modalities has been interpreted by at least one first software module;

in response to interpretation of the input, activating, with the at least one control module, at least one second software module to process further input from the user, the at least one second software module being configured to, when executed by at least one processor of the at least one server device, cause the at least one processor to perform a task for processing the further input from the user, the task resulting in a change of a dialog state of the multimodal application from a first dialog state to a second dialog state; and

receiving a notification at the at least one control module, output by the at least one second software module following performance of the task, that the dialog state of the multimodal application is to change from the first dialog state to the second dialog state, the notification received by the control module identifying the second dialog state to which the dialog state is to change.

10. The at least one computer-readable storage medium of claim 9, wherein the method further comprises:

registering the at least one second software module, the registering comprising identifying at least one condition on which the at least one second software module is to be activated; and

in response to detecting that the input from the user has been interpreted, determining whether the interpretation of the input by the at least one first software module meets the at least one condition on which the at least one second software module is to be activated,

wherein the activating the at least one second software module comprises activating the at least one second software module in response to determining that the interpretation of the input meets the at least one condition.

11. The at least one computer-readable storage medium of claim 9, wherein the method further comprises:

receiving the further input from the user via one or more of the plurality of modalities; and

triggering execution of the at least one second software module to cause the at least one processor to perform the task for processing the further input, wherein triggering the execution comprises providing the further input to the at least one second software module.

12. The at least one computer-readable storage medium of claim 9, wherein the method further comprises:

monitoring for a notification from the at least one second software module that the further input from the user has been interpreted by the at least one second software module; and

in response to the dialog state of the multimodal application changing to the second dialog state as a result of the performance of the task, activating at least one third software module to process third input from the user, the at least one third software module being configured to, when executed by the at least one processor, cause the at least one processor to perform a second task for processing the third input from the user, the second task resulting in a change of a dialog state of the multimodal application from the second dialog state to a third dialog state.

13. The at least one computer-readable storage medium of claim 12, wherein the method further comprises:

in response to the dialog state of the multimodal application changing to the third dialog state, deactivating the at least one second software module.

14. An apparatus comprising:

at least one processor; and

at least one storage medium having encoded thereon executable instructions that, when executed by the at least one processor, cause the at least one processor to carry out a method of managing a multimodal application executing at least in part on the client device, the client device communicating with at least one server device, the client device being configured to accept input from a user via a plurality of modalities, the method comprising;

detecting, with at least one control module, that input from a user via one or more modalities has been interpreted by at least one first software module;

in response to interpretation of the input, activating, with the at least one control module, at least one second software module to process further input from the user, the at least one second software module being configured to, when executed by at least one processor of the at least one server device, cause the at least one processor to perform a task for processing the further input from the user, the task resulting in a change of a dialog state of the multimodal application from a first dialog state to a second dialog state; and

receiving a notification at the at least one control module, output by the at least one second software module following performance of the task, that the dialog state of the multimodal application is to change from the first dialog state to the second dialog state, the notification received by the control module identifying the second dialog state to which the dialog state is to change.

15. The apparatus of claim 14, wherein the method further comprises:

registering the at least one second software module, the registering comprising identifying at least one condition on which the at least one second software module is to be activated; and

in response to detecting that the input from the user has been interpreted, determining whether the interpretation of the input by the at least one first software module meets the at least one condition on which the at least one second software module is to be activated,

wherein the activating the at least one second software module comprises activating the at least one second software module in response to determining that the interpretation of the input meets the at least one condition.

16. The apparatus of claim 14, wherein the method further comprises:

receiving the further input from the user via one or more of the plurality of modalities; and

triggering execution of the at least one second software module to cause the at least one processor to perform the task for processing the further input, wherein triggering the execution comprises providing the further input to the at least one second software module.

17. The apparatus of claim 14, wherein the method further comprises:

monitoring for a notification from the at least one second software module that the further input from the user has been interpreted by the at least one second software module; and

in response to the dialog state of the multimodal application changing to the second dialog state as a result of the performance of the task, activating at least one third software module to process third input from the user, the at least one third software module being configured to, when executed by the at least one processor, cause the at least one processor to perform a second task for processing the third input from the user, the second task resulting in a change of a dialog state of the multimodal application from the second dialog state to a third dialog state.

18. The apparatus of claim 17, wherein the method further comprises:

in response to the dialog state of the multimodal application changing to the third dialog state, deactivating the at least one second software module.

19. The apparatus of claim 14, wherein the apparatus is the at least one server device and the at least one processor is the at least one processor of the server device.