Multi-modal synchronization

• US 7,337,405 B2
• Filed: 02/06/2003
• Issued: 02/26/2008
• Est. Priority Date: 02/07/2002
• Status: Active Grant

• Current Assignee: SAP SE
• Original Assignee: SAP AG (SAP SE)
• Inventors: Guddad, Vinod, Weng, Jie, Raiyani, Samir, Swan, Richard J., Vogler, Hartmut K., Gong, Li
• Primary Examiner(s): Hailu; Tadesse
• Assistant Examiner(s): CHAUDHURI, ANITA
• Application Number:
  US10/358,665
• Publication Number:
  US 20030174155A1
• Time in Patent Office: 1,846 Days
• Field of Search: 715/727, 715/762, 715/760, 715/748, 715/500.1, 709/217, 709/246
• US Class Current: 715/748
• CPC Class Codes: H04M 2201/40 using speech recognition
• CPC Class Codes: H04M 3/4931 Directory assistance systems
• CPC Class Codes: H04M 3/4936 Speech interaction details speech recognition per se G10L15/00
• CPC Class Codes: H04M 3/4938 comprising a voice browser which renders and interprets, e.g. VoiceXML

Abstract

A system for synchronizing multiple modalities is described. A user may use multiple modalities, such as voice and browser, to interact with data on a network, such as the World Wide Web. All of the modalities may be synchronized so that all are updated when the user enters information in just one. A method of communicating between devices includes receiving a request for first-modality data that includes first content, and sending a message in response to receiving the request, the message including information allowing the request of second-modality data that includes second content overlapping the first content. Another method includes requesting first data for a first modality, the first data including first content, and automatically requesting second data for a second modality, wherein the second data includes second content that overlaps the first content.

Claims

1. A computer-implemented method of performing multi-modal synchronization, the method comprising:

enabling, at a given point in time, a first-modality gateway and a second-modality gateway to output the same specific information from the same Uniform Resource Locator (URL) as both first-modality content and second-modality content, and to receive input to the output same specific information using either of the first-modality gateway and the second-modality gateway, further comprising;

opening a first-modality session via a first connection from a first-modality program on a client device to the first-modality gateway;

opening a second-modality session via a second connection from a second-modality program on the client device to the second-modality gateway on a server;

acquiring a unique identifier of the client device by the first-modality gateway and the second-modality gateway;

transmitting the unique identifier from the first-modality gateway to a synchronization handler;

creating a unique messaging channel on a third connection, based upon the unique identifier at the synchronization handler; and

subscribing the first-modality gateway and the second-modality program to the unique messaging channel at a synchronization handler based on the unique identifier; and

automatically determining a first-modality version of a resource if the input is received using the second-modality gateway, and a second-modality version of the resource if the input is received using the first-modality gateway.

2. The method of claim 1 further including associating the first-modality session and the second-modality session with the unique messaging channel, at the server.

3. The method of claim 1 further including publishing an indication of the second-modality content to the first-modality gateway on the unique messaging channel.

4. The method of claim 3 wherein the indication is published on the unique messaging channel by the server.

5. The method of claim 3, further comprising requesting first-modality content corresponding to the second-modality content at the server from the first-modality gateway.

6. The method of claim 3, wherein the published indication includes the Uniform Resource Locator (URL).

7. The method of claim 6, wherein the URL points to a dummy page.

8. The method of claim 1, wherein the first-modality gateway or the second-modality gateway are a voice gateway or a web server.

9. The method of claim 1, wherein the first-modality program or a second-modality program are a voice client or a web browser.

10. The method of claim 1, wherein the unique identifier is a network address of the client device.

11. The method of claim 1, wherein the unique identifier is a user ID of the user operating the client device.

12. The method of claim 1, wherein the unique identifier is a hardware identifier of the client device.

13. The method of claim 1, wherein the unique identifier is a combination of a firewall network address and a network address of the client device.

14. A computer-implemented method of performing multi-modal synchronization, the method comprising:

enabling, at a given point in time, a first-modality gateway and a second-modality gateway to output the same specific information from the same Uniform Resource Locator (URL) as both first-modality content and second-modality content, and to receive input to the output same specific information using either of the first-modality gateway and the second-modality gateway, further comprising;

opening a first-modality session via a first connection from a first-modality program on a client device to the first-modality gateway;

opening a second-modality session via a second connection from a second-modality program on the client device to the second-modality gateway on a server;

acquiring a unique identifier of the client device by the first-modality gateway and the second-modality gateway;

transmitting the unique identifier from the first-modality gateway to a synchronization handler;

creating a unique messaging channel on a third connection, based upon the unique identifier at the synchronization handler;

subscribing the first-modality gateway and the second-modality program to the unique messaging channel at the synchronization handler based on the unique identifier;

associating the unique messaging channel with the second-modality session, at the server;

transmitting the second-modality content from the sewer to the second-modality program via the second connection;

publishing an indication of the second-modality content to the first-modality gateway on the unique messaging channel; and

requesting the first-modality content corresponding to the second-modality content at the sewer from the first-modality gateway; and

automatically determining a first-modality version of a resource if the input is received using the second-modality gateway, and a second-modality version of the resource if the input is received using the first-modality gateway.

15. The method of claim 14, wherein the first-modality is a voice modality.

16. The method of claim 14, wherein the first-modality connection is established using the International Telecommunications Union-T Recommendation H.323 (H.323) standard protocol or a Session Initiation Protocol (SIP).

17. The method of claim 14, wherein the unique identifier is an Internet Protocol (IP) address.

18. The method of claim 14, wherein acquiring the unique identifier further comprises pulling the unique identifier out of a connection header.

19. The method of claim 14, wherein acquiring the unique identifier further comprises querying the client device for the unique identifier.

20. The method of claim 14, wherein the second-modality content is a dummy page that does not include content.

21. The method of claim 14 wherein the second-modality content includes a web page.

22. The method of claim 14, wherein the server is a web server.

23. The method of claim 14, wherein the second-modality connection is established in response to a user entering the Uniform Resource Locator (URL) into the client device.

24. The method of claim 14, further comprising determining the first-modality content corresponding to the second-modality content, at the server.

25. The method of claim 14, wherein the client device is a mobile device.

26. The method of claim 14, further comprising:

requesting a response from the server, at the first-modality gateway; and

transmitting the response from the server to the first-modality gateway.

27. A system for performing multi-modal synchronization, the system comprising:

a client device including a first-modality program and a second-modality program;

a first-modality gateway configured to;

open a first-modality session from the first-modality program via a first connection,

acquire a unique identifier of the client device, and

transmit the unique identifier;

a server including a second-modality gateway configured to;

open a second-modality session from the second-modality program to a second-modality gateway via a second connection;

acquire the unique identifier of the client device at the second-modality gateway; and

automatically determine a first-modality version of a resource if input is received using the second-modality gateway, and a second-modality version of the resource if the input is received using the first-modality gateway; and

a synchronization handler configured to enable, at a given point in time, a first-modality gateway and a second-modality gateway to output the same specific information from the same Uniform Resource Locator as both first-modality content and second-modality content, and to receive the input to the output same specific information using either of the first-modality gateway and the second-modality gateway, by;

receiving the unique identifier from the first-modality gateway,

creating a unique messaging channel on a third connection, based upon the unique identifier handler, and

subscribing the first-modality gateway and the second-modality program to the unique messaging channel based on the unique identifier.

28. A computer program product tangibly embodied in a machine-readable storage medium, the computer program product for performing multi-modal synchronization, where the computer program product comprises instructions that, when read by a machine, operate to cause a data processing apparatus to:

enable, at a given point in time, a first-modality gateway and a second-modality gateway to output the same specific information from the same Uniform Resource Locator (URL) as both first-modality content and second-modality content, and to receive input to the output same specific information using either of the first-modality gateway and the second-modality gateway, by;

opening a first-modality session via a first connection from a first-modality program on a client device to the first-modality gateway;

opening a second-modality session via a second connection from a second-modality program on the client device to the second-modality gateway on a server;

acquiring a unique identifier of the client device by the first-modality gateway and the second-modality gateway;

transmitting the unique identifier from the first-modality gateway to a synchronization handler;

creating a unique messaging channel on a third connection, based upon the unique identifier at the synchronization handler; and

subscribing the first-modality gateway and the second-modality program to the unique messaging channel at a synchronization handler based on the unique identifier; and

automatically determine a first-modality version of a resource if the input is received using the second-modality gateway, and a second-modality version of the resource if the input is received using the first-modality gateway.