Server for handling multimodal information

• US 6,859,451 B1
• Filed: 04/21/1998
• Issued: 02/22/2005
• Est. Priority Date: 04/21/1998
• Status: Expired due to Term

• Current Assignee: RPX Clearinghouse LLC (RPX Corporation)
• Original Assignee: Nortel Networks Limited (Nortel Networks Corporation)
• Inventors: Ferrero, Daniel, Fumai, Nicola, Pasternack, Henry Adam
• Primary Examiner(s): Yao, Kwang Bin
• Application Number:
  US09/062,969
• Time in Patent Office: 2,499 Days
• Field of Search: 370/352, 370/353, 370/354, 370/355, 370/356, 370/400, 370/401, 370/402, 370/403, 370/404, 370/405, 370/410, 370/447, 370/461, 370/462, 370/463, 370/465, 370/466, 379/88.01, 379/88.02, 379/88.06, 379/88.17, 379/88.04, 379/93.09, 704/231, 704/246, 704/270, 704/275, 709/217, 709/218, 709/219
• US Class Current: 370/352
• CPC Class Codes: H04L 65/1026 at the edge
• CPC Class Codes: H04L 65/1036 at the edge
• CPC Class Codes: H04L 65/1043 Gateway controllers, e.g. media gateway control protocol [MGCP] controllers
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 69/18 Multiprotocol handlers, e.g. single devices capable of handling multiple protocols
• CPC Class Codes: H04L 9/40 Network security protocols
• CPC Class Codes: H04M 7/003 Click to dial services

Abstract

To integrate telephony with Web-based information services, and develop multi-modal telephony applications that combine graphical input and output with traditional speech-based user interfaces, a Web server for handling information which is in these different modal forms, has an internet interface for supporting connections on the internet, and a terminal interface for supporting connections from the server to user terminals, and for passing information in at least one of the modal forms. A service controller controls input or output of the information on the interfaces, and processes the information received from or sent to either interface, according to its modal form.

Claims

1. A server for handling information which is in different modal forms suitable for more than one mode of user interface, the server comprising:

an internet interface for supporting one or more connections on the internet;

a terminal interface for supporting one or more connections from the server to user terminals providing multi-modal user interfaces capable of supporting input and output in one or more of audio, visual and tactile modal forms, and for passing information in audio, visual and tactile modal forms; and

a service controller being operable to interact with the user terminals for controlling input or output of the information on the terminal interface and the internet interface, and for processing the information received from or sent to either interface, according to the modal form of the input and output

the service controller being operable to receive input in different modal forms simultaneously from a user interaction at the multimodal interface, and to resolve conflicts.

2. The server of claim 1, the service controller being operable with a multi-modal user terminal to select which modal form or forms of input and output to use.

3. The server of claim 2, the service controller being arranged to select which modal form or forms to use according to the content of the information, and the context of the user interaction.

4. The server of claim 1, the service controller being operable to determine an intention of the user based on the inputs.

5. The server of claim 1, the terminal interface being arranged to recognize speech as an input.

6. The server of claim 1, the terminal interface being arranged to generate audio as an output mode.

7. The server of claim 1, the service controller being arranged to conduct a dialogue with the user in the form of a sequence of interactions.

8. The server of claim 1, further comprising means for initiating a connection to the user'"'"'s terminal.

9. The server of claim 1, further comprising a link to a telephone network, and a call processor for making and receiving telephone calls on the telephone network.

10. The server of claim 1, comprising a set of servlets arranged to communicate with each other by event driven messages.

11. A server for interfacing between telephone calls and the internet, and comprising:

a terminal interface for supporting one or more connections from the server to user terminals-providing multi-modal user interfaces capable of supporting input and output in one or more of audio, visual and tactile modal forms, the terminal interface providing a telephony interlace for receiving or making a telephone call, and arranged to interact with a user on the call by recognizing speech or generating audio signals;

an internet interface for receiving or outputting information from or to other parts of the internet; and

a service controller for controlling interaction between the terminal interface, telephony interface and the internet interface,

the service controller being operable to interact with a user having a terminal capable of handling telephony or internet communications, to select which modal form or forms to use for controlling input or output of the information on the terminal interface and the internet interface, according to the modal form of the input and output,

and the service controller being operable to receive input in different modal forms simultaneously from a user interaction at the multimodal interface, and to resolve conflicts.

12. The server of claim 11, wherein the controller is operable to select which of telephony or internet communications to use when sending information to the terminal, and to convert the information to suit the selected type of communication.

13. A method of using a server to handle information in different modal forms suitable for more than one mode of user interface, and comprising the steps of:

supporting one or more connections on the internet;

supporting one or more connections from the server to user terminals having a multi-modal user interface capable of supporting user input and output in one or more of audio, visual and tactile modal forms; and

for each user input,

passing information in different modal forms between the user and the server;

processing the information received from or sent to either interface, according to the modal form of the input and output

and when receiving inputs in different modal forms simultaneously from the same user input and, resolving conflicts.

14. A method of using a multi-modal service provided by a server on the internet, the server having an internet interface for supporting one or more connections on the internet, a terminal interface for supporting a connection to a user of the service, and for passing information in one or more of audio, visual and tactile modal forms;

and a service controller for controlling input or output of information on the terminal interface and the internet interface, and for processing the information received from or sent to either interface, according to the modal form, the method comprising the step of;

providing input to the terminal interface of the server;

engaging in a dialogue with the server to cause the server to process the information according to the modal form of the information; and

receiving a response from the terminal interface of the server, according to the result of the information processing

and when receiving inputs in different modal forms simultaneously from the terminal interface, resolving conflicts, and determining an intention of the user based on the inputs.

15. Software on a computer readable medium for carrying out a method of using a server to handle information in different modal forms suitable for more than one mode of user interface, the method comprising the steps of:

supporting one or more connections on the internet;

supporting one or more connections from the server to the user terminals, having multi-modal user interfaces capable of supporting user input and output in one or more of audio, visual and tactile modal forms;

passing information in different modal forms between a user terminal and the server;

controlling input or output of the information on the terminal and internet connections; and

processing the information received from or sent to the terminal and internet interface, according to its modal form,

resolving conflicts when receiving inputs modal forms simultaneously from the same user.

16. A server for handling information which is in different modal forms suitable for more than one mode of user interface, the server comprising:

an internet interface for supporting one or more connections on the internet;

a terminal interface for supporting one or more connections from the server to user terminals having a multi-modal user interface capable of supporting user input and output in one or more of audio, visual and tactile modal forms and for passing information in at least one of the modal forms; and

a service controller being operable to interact with a multi-modal terminal, to select which modal form or forms to use, and for controlling information from user input or output on the terminal interface and the internet interface according to the modal form of the user input and output; and

the service controller being operable to receive inputs in different modal forms simultaneously from the same user, to resolve any conflicts, and determine an intention of the user based on the inputs.