Systems and methods for synchronizing multi-modal interactions

• US 7,216,351 B1
• Filed: 02/18/2000
• Issued: 05/08/2007
• Est. Priority Date: 04/07/1999
• Status: Expired due to Term

• Current Assignee: Uniloc 2017 LLC (SoftBank Group Corp.)
• Original Assignee: International Business Machines Corporation
• Inventors: Maes, Stephane H.
• Primary Examiner(s): Thomson; William
• Assistant Examiner(s): Ho; Andy
• Application Number:
  US09/507,526
• Time in Patent Office: 2,636 Days
• Field of Search: 709/310, 718/1, 718/100, 718/101, 719/310, 719/311, 719/312, 719/313, 719/328
• US Class Current: 719/328
• CPC Class Codes: G06F 9/545 where tasks reside in different layers, e.g. user- and kernel-space

Abstract

System and methods for synchronizing interactions between mono-mode applications, different modes of a multi-modal application, and devices having different UI modalities. In one aspect, a multi-modal shell coordinates multiple mode processes (i.e. modalities) of the same application or multiple applications through API calls, whereby each mode process registers its active commands and the corresponding actions in each of the registered modalities. The multi-modal shell comprises a registry that is implemented with a command-to-action table. With the execution of a registered command, each of the corresponding actions are triggered to update each mode process accordingly, and possible update the registry to support new commands based on the change in state of the dialog or application. In another aspect, separate applications (with UI of different modalities) are coordinated via threads (e.g., applets) connected by socket connections (or virtual socket connections implemented differently). Any command in one mode triggers the corresponding thread to communicate the action to the thread of the other application. This second thread modifies accordingly the state of the second process mode. The threads are updated or replaced by new threads.

Claims

1. A system for synchronizing multi-modal interactions, comprising:

a program storage device that stores a multi-modal application, the multi-modal application comprising at least a first mode process that enables user interaction with the application in a first modality and a second mode process that enables user interaction with the application in a second modality;

a program execution system that executes the multi-modal application and synchronizes the first and second mode processes while a user interacts with the multi-modal application, wherein the program execution system comprises;

a multi-modal shell that manages information exchanges between the first and second mode processes of the multi-modal application to enable a synchronized multi-modal interaction with the multi-modal application wherein user interaction in one modality results in execution of corresponding commands in both the first and second mode processes; and

an API (application program interface) that allows the first and second mode processes to register their respective active commands and corresponding actions with the multi-modal shell.

2. The system of claim 1, wherein the program execution system comprises a registry having a registration table, managed by the multi-modal shell, wherein the registration table comprises a list of each of the registered commands and corresponding synchronized actions that results in both the first and second mode processes upon execution of a registered command by one of the first and second mode processes.

3. The system of claim 1, wherein the multi-modal application comprises at least a first mono-mode application for the first mode process and a second mono-mode application for the second mode process, wherein the multi-modal shell manages and synchronizes information exchanges between the first and second mono-mode applications.

4. The system of claim 1, further comprising at least a first device having a first user interface modality and a second device having a second user interface modality, wherein the multi-modal shell manages and synchronizes information exchanges between the first and second devices.

5. The system of claim 4, wherein the first device, second device and multi-modal shell are distributed over a network, and wherein the API is implemented using distributed APIs or protocols.

6. The system of claim 2, wherein the API comprises a mechanism for converting a mono-mode application to a multi-modal application.

7. The system of claim 6, wherein the mono-mode application is a GUI application, and wherein the mechanism provides speech enablement of the GUI application by registering the active commands of the GUI application and building a grammar for the registered commands to support the commands in a speech modality.

8. The system of claim 2, wherein the API comprises a mechanism for building a multi-modal application.

9. The system of claim 8, wherein the mechanism is used for directly programming the registry by building a registration table having user-defined commands and corresponding actions for each of the modalities of the multi-modal application.

10. The system of claim 1, further comprising an operating system, wherein the multi-modal shell executes on top of the operating system.

11. The system of claim 1, wherein the system is distributed over a network.

12. The system of claim 1, wherein the multi-modal application is a multi-modal browser comprising a first browser application and a second browser application.

13. The system of claim 12, wherein the first browser is a GUI browser and the second browser is a speech browser.

14. The system of claim 12, wherein the multi-modal shell processes the multi-modal application to send modality-specific presentation information to the respective browsers.

15. The system of claim 14, wherein the multi-modal application is authored using a modality-independent description and wherein the multi-modal shell generates the modality-specific presentation information from the modality-independent description.

16. The system of claim 14, wherein the multi-modal application comprises a combination of declarative markup languages to describe the first and second mode processes.

17. The system of claim 16, wherein the multi-modal application combines the declarative markup languages and synchronization elements in single pages to provide tight synchronization between the first and second mode processes.

18. The system of claim 16, wherein the multi-modal application comprises separate files of the declarative markup languages for the first and second mode processes and wherein the separate files are loosely synchronized at predefined points.

19. A method for synchronizing multi-modal interactions, comprising the steps of:

activating a multi-modal application comprising at least a first mode process that enables user interaction with the application in a first modality and a second mode process that enables user interaction with the application in a second modality

receiving a command or event in the first modality;

triggering an action by the first mode process based on the received command or event and triggering a corresponding action by the second mode process; and

updating application states or device states associated with the first and second mode processes.

20. The method of claim 19, further comprising the steps of:

registering active commands associated with the first mode process and active commands associated with the second mode process;

associating, with each registered command of the first mode process, an action on the first mode process and a corresponding action on the second mode process; and

associating, with each registered command of the second mode process, an action on the second mode process and a corresponding action on the first mode process.

21. The method of claim 20, further comprising the step of building a command-to-action registration table based on the registered commands and actions.

22. The method of claim 21, wherein the registration table is built by a multi-modal shell via API calls from the applications or devices associated with the first and second mode processes.

23. The method of claim 20, wherein the step of triggering comprises the steps of:

looking up the received command in the registration table; and

executing the actions associated with the received command by the first and second mode processes.

24. The method of claim 20, further comprising the steps of:

registering a callback handle for each of the registered commands to notify the first and second mode processes of completion of the actions corresponding to the registered commands.

25. The method of claim 24, wherein the step of updating the application states or the device states comprises the steps of executing the callback handle associated with the received command to trigger a callback action on the first mode process and a callback action on the second mode process.

26. The method of claim 19, wherein the step of triggering comprises the steps of:

executing first thread associated with the received command; and

triggering a corresponding second thread to initiate the corresponding action by the second mode process.

27. The method of claim 26, wherein the threads are applets.

28. The method of claim 26, wherein the threads communicate via socket connections.

29. A program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform method steps for synchronizing multi-modal interactions, the method comprising the steps of:

activating a multi-modal application comprising at least a first mode process that enables user interaction with the application in a first modality and a second mode process that enables user interaction with the application in a second modality

receiving a command or event in the first modality;

triggering an action by the first mode process based on the received command or event and triggering a corresponding action by the second mode process; and

updating application states or device states associated with the first and second mode processes.

30. The program storage device of claim 29, further comprising instructions for performing the steps of:

registering active commands associated with the first mode process and active commands associated with the second mode process;

associating, with each registered command of the first mode process, an action on the first mode process and a corresponding action on the second mode process; and

associating, with each registered command of the second mode process, an action on the second mode process and a corresponding action on the first mode process.

31. The program storage device of claim 30, further comprising instructions for performing the step of building a command-to-action registration table based on the registered commands and actions.

32. The program storage device of claim 30, wherein the instructions for performing the step of triggering comprise instructions for performing the steps of:

looking up the received command in the registration table; and

executing the actions associated with the received command by the first and second mode processes.

33. The program storage device of claim 30, further comprising instructions for performing the steps of:

registering a callback handle for each of the registered commands to notify the first and second mode processes of completion of the actions corresponding to the registered commands.

34. The program storage device of claim 33, wherein the instructions for performing the step of updating the application states or the device states comprise instructions for performing the step of executing the callback handle associated with the received command to trigger a callback action on the first mode process and a callback action on the second mode process.

35. The program storage device of claim 29, wherein the instructions for performing the step of triggering comprise instructions for performing the steps of:

executing first thread associated with the received command; and

triggering a corresponding second thread to initiate the corresponding action by the second mode process.

36. The program storage device of claim 35, wherein the threads are applets.

37. The program storage device of claim 35, wherein the threads communicate via socket connections.