System and method for providing dialog management and arbitration in a multi-modal environment

• US 20030005174A1
• Filed: 06/29/2001
• Published: 01/02/2003
• Est. Priority Date: 06/29/2001
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Kleindienst, Jan, Hosn, Rafah A., Maes, Stephane H., Coffman, Daniel M., Raman, Thiruvilwamalai V.
• Granted Patent: US 6,839,896 B2
• Time in Patent Office: Days
• US Class Current: 719/318
• CPC Class Codes: G06F 2209/543 Local
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications

Abstract

A system and method for providing conversational computing via a protocol for automatic dialog management and arbitration between a plurality of conversational applications, and a framework for supporting such protocol, in a multi-modal and/or multi-channel environment. A DMAF (dialog manager and arbitrator facade) interfaces with one or more applications, and a hierarchical DMA architecture enables arbitration across the applications and within the same application between various sub-dialogs.

Claims

1. A method for managing dialog of one or more applications, comprising the steps of:

instantiating a DMA (dialog manager and arbitrator) interface comprising a hierarchical tree structure comprising a root DMA and one or more application DMAs;

sending, by the root DMA, notification of a user input event to an application DMA;

obtaining, by the application DMA, a symbolic representation of the user input event;

calling, by the application DMA, an application method to perform context resolution of the symbolic representation;

receiving, by the application DMA, a query from the application, wherein the query comprises the results of the context resolution;

determining, by the DMA interface, if the application DMA is currently active based on the query received by the application DMA; and

launching, by the application DMA, a callback function associated with the query, if the application DMA is determined to be currently active.

2. The method of claim 1, wherein the step of instantiating a DMA interface comprises registering, by an application, with the root DMA to obtain an application DMA handle.

3. The method of claim 2, wherein the step of registering comprises registering an algorithm string comprising a set and order of engines required by the application for processing user input.

4. The method of claim 1, wherein the step of instantiating a DMA interface comprises instantiating a main DMA to manage the main dialog of an application and instantiating a plurality of child instances of the main DMA to manage sub-dialogs of the application.

5. The method of claim 1, wherein the step of determining if the application DMA is currently active comprises the step of using a bottom-up arbitration protocol to determine if the query is a top-scoring query among all queries received by other application DMAs.

6. The method of claim 5, wherein arbitration is performed between friend applications based on security settings registered by the applications.

7. The method of claim 5, wherein the step of using a bottom-up arbitration protocol comprises:

arbitrating, by each parent application DMA in the hierarchical tree, between queries received from application DMAs that are children of the parent; and

arbitrating, by the root DMA, between queries received from application DMAs that are children of the root DMA to determine if there is an overall winning query.

8. The method of claim 7, further comprising the steps of:

disregarding, by a parent DAM, a child DMA that does not provide a query within a predetermined time period; and

extending the predetermined time period upon a request from a child DMA.

9. The method of claim 7, further comprising the steps of:

generating, by the root DMA, an arbitration result comprising the overall winning query and the associated application DMA; and

sending, by each parent application DMA, notification of the arbitration result to each child application DMA.

10. The method of claim 1, wherein the step of launching the callback function comprises using a command registry to determine the callback function associated with the query.

11. The method of claim 1, wherein the symbolic representation comprises one of a speech recognition result, a natural language understanding result and a combination thereof.

12. The method of claim 1, further comprising the step of maintaining, by the DMA interface, a short term history comprising events associated with a user input event.

13. The method of claim 1, further comprising the step of maintaining, by the DMA interface, a long term history of events associated with an executed task.

14. The method of claim 1, further comprising the step of maintaining, by the DMA interface, a focus history that tracks active application DMAs during a dialog session.

15. The method of claim 1, further comprising the step of collaborating between an application and an application DMA to disambiguate an associated query.

16. The method of claim 15, further comprising the step of disambiguating a query using one of a short term history, long term history, focus history, and a combination thereof.

17. The method of claim 1, further comprising the step of generating a message in an appropriate modality, if a message must be presented to a user.

18. The method of claim 17, wherein the step of generating a message in an appropriate modality comprises converting, by a modality-specific I/O agent, a modality-independent output event to a modality-specific output event.

19. The method of claim 1, wherein if the callback function comprises a dictation mode, notifying the root DMA to send all input events to the application DMA.

20. The method of claim 19, further comprising the step of termination the dictation mode in response to a user input command.

21. A program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform method steps for managing dialog of one or more applications, the method steps comprising:

instantiating a DMA (dialog manager and arbitrator) interface comprising a hierarchical tree structure comprising a root DMA and one or more application DMAs;

sending, by the root DMA, notification of a user input event to an application DMA;

obtaining, by the application DMA, a symbolic representation of the user input event;

calling, by the application DMA, an application method to perform context resolution of the symbolic representation;

receiving, by the application DMA, a query from the application, wherein the query comprises the results of the context resolution;

determining, by the DMA interface, if the application DMA is currently active based on the query received by the application DMA; and

launching, by the application DMA, a callback function associated with the query, if the application DMA is determined to be currently active.

22. The program storage device of claim 21, wherein the instructions for instantiating a DMA interface comprise instructions for registering, by an application, with the root DMA to obtain an application DMA handle.

23. The program storage device of claim 21, wherein the instructions for instantiating a DMA interface comprise instructions for instantiating a main DMA to manage the main dialog of an application and instantiating a plurality of child instances of the main DMA to manage sub-dialogs of the application.

24. The program storage device of claim 21, wherein the instructions for determining if the application DMA is currently active comprise instructions for utilizing a bottom-up arbitration protocol to determine if the query is a top-scoring query among all queries received by other application DMAs.

25. The program storage device of claim 24, wherein the instructions for using a bottom-up arbitration protocol comprise instructions for:

arbitrating, by each parent application DMA in the hierarchical tree, between queries received from application DMAs that are children of the parent;

arbitrating, by the root DMA, between queries received from application DMAs that are children of the root DMA to determine if there is an overall winning query.

26. The program storage device of claim 25, further comprising instruction for:

generating, by the root DMA, an arbitration result comprising the overall winning query and the associated application DMA; and

sending, by each parent application DMA, notification of the arbitration result to each child application DMA.

27. The program storage device of claim 21, wherein the instructions for launching the callback function comprise instructions for using a command registry to determine the callback function associated with the query.

28. The program storage device of claim 21, wherein the symbolic representation comprises one of a speech recognition result, a natural language understanding result and a combination thereof.

29. The program storage device of claim 21, further comprising instructions for maintaining, by the DMA interface, a short term history comprising events associated with a user input event during a state of the dialog.

30. The program storage device of claim 21, further comprising instructions for maintaining, by the DMA interface, a long term history of events associated with an executed task.

31. The program storage device of claim 21, further comprising instruction for maintaining, by the DMA interface, a focus history that tracks active application DMAs during a dialog session.

32. The program storage device of claim 21, further comprising instructions for collaborating between an application and an application DMA to disambiguate an associated query.

33. The program storage device of claim 32, further comprising instructions for disambiguating a query using one of a short term history, a long term history, a focus history, and a combination thereof.

35. The DMA interface of claim 34, wherein the DMA interface is implemented in a multi-modal platform.

36. The DMA interface of claim 34, wherein the DMA interface stores and manages application information.

37. The DMA interface of claim 34, wherein each of the root DMA and application DMAs maintain a list of events that are generated for a given user input.

38. The DMA interface of claim 34, wherein each of the root DMA and application DMAs maintain a list of tasks that have been executed for a given dialog session.

39. The DMA interface of claim 34, further comprising a mechanism for collaborating with an application to disambiguate a user input event.

40. The DMA interface of claim 34, wherein the root DMA comprises a registration table to maintain a list of registered application DMAs, and wherein the root DMA tracks which of the registered application DMAs is currently active.

41. The DMA interface of claim 34, wherein the root DMA communicates with an I/O manager, wherein the I/O manager abstracts user input events into modality-independent events for processing by the DMA interface, and wherein the I/O manager converts abstract output events into one or more modality-specific output events.

42. The DMA interface of claim 41, wherein the I/O manager comprises a central I/O manager and a modality agent for each modality supported by an underlying platform.

43. The DMA interface of claim 42, wherein the central I/O manager communicates with each modality agent using a common XML-based messaging protocol.

44. The DMA interface of claim 34, wherein the application DMAs communicate with a task manager for utilizing one or more conversational engines, wherein the task manager comprises a thread pool manager for tracking threads associated with the application DMAs, and an engine manager for communicating with conversational engine APIs.

45. The DMA interface of claim 44, wherein the engine manager communicates with a resource manager, wherein the resource manager designates a resource requested by the task manager.

46. The DMA interface of claim 34, further comprising a security mechanism for controlling information exchanges between application DMAs based on security settings registered by one or more applications.

47. The DMA interface of claim 46, wherein the security mechanism provides arbitration only among friend applications.

48. The DMA interface of claim 47, further comprising a mechanism for switching the arbitration function between groups of friend applications.

49. The DMA interface of claim 46, wherein the security mechanism comprises digital certificate certification.

50. The DMA interface of claim 46, wherein the security mechanism comprises encrypted information exchanges.

51. The DMA interface of claim 34, comprising a pluggable arbitration protocol.

52. The DMA interface of claim 51, wherein the arbitration protocol comprises one of a heuristics, deterministic and statistical algorithm.

53. The DMA interface of claim 34, wherein the DMA interface is employed in a voice portal.

54. The DMA interface of claim 53, wherein each application comprises an application browser that communicates with a main application DMA.

34. A DMA (dialog manager and arbitrator) interface, comprising:

a root DMA for arbitrating among a plurality of applications to determine an active application for a given user input event; and

a plurality of application DMAs, wherein at least one application DMA is associated with each application, for arbitrating among a plurality of sub-dialogs within an application to determine a target application DMA managing the sub-dialog associated with the user input;

wherein the DMA interface comprises a hierarchical tree structure and wherein arbitration is performed by the DMA interface using a bottom-up approach.