Multi modal interface

• US 20040117804A1
• Filed: 09/17/2003
• Published: 06/17/2004
• Est. Priority Date: 03/30/2001
• Status: Abandoned Application

• Current Assignee: British Telecommunications PLC (BT Group PLC)
• Original Assignee: British Telecommunications PLC (BT Group PLC)
• Inventors: Ringland, Simon P A, Scahill, Francis J
• Application Number:
  US10/472,046
• Publication Number:
  US 20040117804A1
• Time in Patent Office: Days
• US Class Current: 719/320
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 2209/544 Remote
• CPC Class Codes: G06F 2209/545 Gui
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications
• CPC Class Codes: H04L 41/0681 Configuration of triggering conditions
• CPC Class Codes: H04L 67/75 Indicating network or usage conditions on the user display
• CPC Class Codes: H04L 69/329 in the application layer [OSI layer 7]

Abstract

A system for synchronizing application programs which together provide a multi-modal user interface, which comprises multiple application programs which provide the various interface of the multi-modal interface and which are in communication with a synchronization manager. Means are provided to detect status changes in the application programs and to communicate such status changes, in the form of data updates to the synchronisation manager. The synchronization manager is operative to communicate such a data update to the application program in which the data update did not originate so that the application programs are synchronised.

Claims

1. A system for synchronizing application programs which together provide a multi-modal user interface, the system comprising:

i) a plurality of application programs, a first of which provides a first user interface of the multi-modal interface, and a second of which provides an second user interface of the multi-modal interface;

ii) a synchronization manager;

iii) communications links between the synchronization manager and each of the application programs and by means of which the synchronization manager can communicate with the application programs;

iv) communications links between the synchronization manager and each of the application programs over which the application programs can transfer data to the synchronization manager;

wherein the synchronization manager comprises a client component for each of the first and second application programs and a server component, the client components being operative to detect user interface related actions in the application programs and application generated events and to transmit such detected actions, in the form of data updates, to the server component, the server component being operative to communicate such data updates to the application programs, the arrangement being such that user interface related actions in respect of one application program are detected by a client component, and the relevant data from the detected actions are communicated by the server component to the other application programs so that the application programs are synchronised.

3. A system as claimed in claim 1 or claim 2, wherein the first user interface is a visual user interface.

4. A system as claimed in any one of claims 1 to 3, wherein there are multiple visual user interfaces provided by multiple application programs.

5. A system as claimed in any one of claims 1 to 4, wherein the second user interface is an audio user interface.

6. A system as claimed in any one of the preceding claims, wherein there are multiple audio user interface provided by multiple application programs.

7. A system as claimed in any one of the preceding claims, wherein at least one application program provides a haptic user interface.

8. A system as claimed in claim 7, wherein the first application program provides a Braille interface and the second application program provides a voice interface.

9. A system as claimed in any one of the preceding claims, wherein the synchronisation manager has translation means to convert data updates from one application program into a form suitable for use within another of the application programs.

10. A system as claimed in any one of the preceding claims, wherein means are provided to select from the data updates received by the synchronisation manager only those data updates which are relevant to the other application programs, only the selected data updates then being passed on to the other application programs.

11. A system as claimed in claim 2 or any one of claims 3 to 10 as dependent on claim 2, wherein some of the status changes relate to user interaction.

12. A system as claimed in claim 2 or any one of claims 3 to 11 as dependent on claim 2, where some of the status changes relate to application internal events (e.g. DocLoaded, error conditions, or application specific events e.g. random number generated in roulette application).

2. A system for synchronizing application programs which together provide a multi-modal user interface, the system comprising

i) first and second application programs, the first of which provides a first user interface of the multi-modal interface, and the second of which provides a second user interface of the multi-modal interface;

ii) a synchronization manager;

iii) communications links between the synchronization manager and each of the application programs by means of which the synchronization manager can communicate with the application programs;

iv) communications links between the synchronization manager and each of the application programs over which the application programs can transfer data to the synchronization manager;

wherein means are provided to detect status changes in the first and second application programs, means being provided to communicate such status changes, in the form of data updates to the synchronisation manager, the synchronization manager being operative to communicate such a data update to the application program in which the data update did not originate so that the first and second application programs are synchronised.

13. A method for synchronizing application programs which together provide a multi-modal user interface, the multi-modal interface comprising a plurality of application programs, a first of which provides a first user interface of the multi-modal interface, and a second of which provides a second user interface of the multi-modal interface, and a synchronization manager which can communicate with the application programs, the synchronization manager comprising a client component for each of the first and second application programs and a server component, the client components being operative to detect user interface related actions in the application programs and changes in the state of the application programs and to transmit such detected actions and changes of state, in the form of data updates, to the server component, the server component being operative to communicate such data updates to the application programs;

the method comprising;

(i) detecting user interface related actions in the application programs;

(ii) transmitting such detected actions, in the form of data updates, to the synchronisation manager;

(iii) converting, as necessary, under the control of the synchronisation manager, the data updates into forms suitable for each of the other application programs,

(iv) communicating the converted data updates from the synchronisation manager to the application programs;

so that user interface related actions in respect of one application program are detected by the client component, and the relevant data from the detected actions are communicated by the server component to the other application programs to synchronise the application programs.

15. A method as claimed in claim 13 or claim 14, wherein the first user interface is a visual user interface.

16. A method as claimed in any one of claims 13 to 15, wherein there are multiple visual user interfaces provided by multiple application programs.

17. A method as claimed in any one of claims 13 to 16, wherein the second user interface is an audio user interface.

18. A method as claimed in any one of claims 13 to 17, wherein there are multiple audio user interface provided by multiple application programs.

19. A method as claimed in any one of claims 13 to 18, wherein at least one application program provides a haptic user interface.

20. A method as claimed in claim 19, wherein the first application program provides a Braille interface and the second application program provides a voice interface.

22. A method as claimed in any one of claims 13 to 21, wherein a selection process is carried out to select from the data updates received by the synchronisation manager only those data updates which are relevant to the other application programs, only the selected data updates then being passed on to the other application programs.

14. A method for synchronizing application programs which together provide a multi-modal user interface, the multi-modal interface comprising first and second application programs, the first of which provides a first user interface of the multi-modal interface, and the second of which provides a second user interface of the multi-modal interface, a synchronization manager able to communicate with the application programs, the method comprising the steps of

(i) detecting status changes in the first and second application;

(ii) communicating such status changes, in the form of data updates to the synchronisation manager; and

(iii) transmitting from the synchronization manager such a data update to the application program in which the data update did not originate so that the first and second application programs are synchronised.

21. A method as claimed in claim 14 or any one of claims 15 to 20 as dependent on claim 14, wherein the synchronisation manager causes the translation of data updates received from one application program into a form suitable for use within another of the application programs.

23. A system for the provision of a multi-modal user interface which has a first user interface part and a second user interface part, at least the first user interface part operating according to stored dialogues;

and

control means arranged to control the operation of the multi-modal interface and operatively connected to the first and second parts;

wherein the first part has, for at least some of the possible dialogues which it supports, multiple alternative versions of the dialogues, the system being configured to switch between dialogues and between the alternative versions of the dialogues in dependence upon conditions in the multi-modal user interface.

24. A system as claimed in claim 23, wherein the second user interface part operates according to stored dialogues;

wherein the second user interface part has, for at least some of the possible dialogues which it supports, multiple alternative versions of the dialogues, the system being configured to switch between dialogues and between the alternative versions of the dialogues in dependence upon conditions in the multi-modal user interface.

27. A system as claimed in any one of claims 23 to 26, wherein the first user interface part provides a visual user interface and wherein the second user interface part is an audio interface.

25. A system for the provision of a multi-modal user interface which has a first user interface part and a second user interface part, at least the first user interface part including first means to provide cues to a user of the system according to stored dialogues and second means to receive input from the user;

and

control means arranged to control the operation of the multi-modal interface and operatively connected to the first and second means;

wherein the first means has, for at least some of the possible dialogues which it supports, multiple alternative versions of the dialogues, the system being configured to switch between dialogues and between the alternative versions of the dialogues in dependence upon conditions in the multi-modal user interface.

26. A system as claimed in claim 25, wherein the second user interface part includes third means to provide prompts to a user of the system according to stored dialogues and fourth means to receive input from the user;

wherein the third means has, for at least some of the possible dialogues which it supports, multiple alternative versions of the dialogues, the system being configured to switch between dialogues and between the alternative versions of the dialogues in dependence upon conditions in the multi-modal user interface.

33. A system as claimed in any one of claims 30 to 32, wherein the conditions in the multi-modal user interface to which can cause switching between dialogues and/or tracks in a dialogue include:

user input;

user preferences;

the presence or absence of additional modes of the multi-modal user interface; and

system state.