Multi-modal synchronization

• US 20030174155A1
• Filed: 02/06/2003
• Published: 09/18/2003
• Est. Priority Date: 02/07/2002
• Status: Active Grant

• Current Assignee: SAP SE
• Original Assignee: SAP AG (SAP SE)
• Inventors: Weng, Jie, Raiyani, Samir, Swan, Richard J., Vogler, Hartmut K., Gong, Li, Guddad, Vinod
• Granted Patent: US 7,337,405 B2
• Time in Patent Office: Days
• US Class Current: 715/700
• CPC Class Codes: H04M 2201/40 using speech recognition
• CPC Class Codes: H04M 3/4931 Directory assistance systems
• CPC Class Codes: H04M 3/4936 Speech interaction details speech recognition per se G10L15/00
• CPC Class Codes: H04M 3/4938 comprising a voice browser which renders and interprets, e.g. VoiceXML

Abstract

A system for synchronizing multiple modalities is described. A user may use multiple modalities, such as voice and browser, to interact with data on a network, such as the World Wide Web. All of the modalities may be synchronized so that all are updated when the user enters information in just one. A method of communicating between devices includes receiving a request for first-modality data that includes first content, and sending a message in response to receiving the request, the message including information allowing the request of second-modality data that includes second content overlapping the first content. Another method includes requesting first data for a first modality, the first data including first content, and automatically requesting second data for a second modality, wherein the second data includes second content that overlaps the first content.

Claims

1. A method of communicating between devices, the method comprising:

receiving a request for first-modality data that includes first content, the first-modality data being configured to be presented using a first modality, and the request coming from a requestor and being received at a first device; and

sending a message from the first device for receipt by a second device, the message being sent in response to receiving the request and including information allowing the second device to request second-modality data that includes second content that overlaps the first content, the second-modality data being configured to be presented using a second modality,

wherein the first-modality data and the second-modality data may be synchronized by presenting the first-modality data and the second-modality data to a user such that the user may respond to the overlapping content using either the first modality or the second modality.

2. The method of claim 1 further comprising determining the information.

3. The method of claim 1 wherein the information includes a pointer to the second-modality data.

4. The method of claim 1 wherein the information includes a first pointer that points to a second pointer that points to the second-modality data.

5. The method of claim 1 wherein the information includes a pointer to the first-modality data.

6. The method of claim 1 further comprising presenting the first-modality data and the second-modality data.

7. The method of claim 6 wherein the first-modality data and the second-modality data are presented during an overlapping time period.

8. The method of claim 7 wherein:

the first-modality data is presented to a first user at a first mobile device,

the second-modality data is presented to a second user at a second mobile device, and

either the first user or the second user may respond to the overlapping content.

9. The method of claim 1 wherein sending the message comprises publishing the message on a channel in a publish/subscribe system to which the second device has already been subscribed.

10. The method of claim 9 wherein the channel is identified by an Internet Protocol (“

IP”

) address of a user device.

11. The method of claim 9 wherein the channel is identified by an identifier of a user device.

12. The method of claim 9 wherein the channel is identified by an identifier of a user that is using the first modality and the second modality.

13. The method of claim 9 wherein:

the first device comprises a browser,

the first-modality data comprises a hypertext markup language (“

HTML”

) page,

the requestor comprises a user,

the browser sends the message using a browser adaptor,

the second device comprises a voice gateway including a voice gateway adaptor, and

the second-modality data comprises an indicator of a voice extensible markup language (“

VXML”

) page that corresponds to the HTML page.

14. The method of claim 13 wherein the indicator of the VXML page comprises a uniform resource locator (“

URL”

) of the VXML page.

15. The method of claim 13 wherein the indicator of the VXML page comprises a URL of the HTML page.

16. The method of claim 9 wherein:

the first device comprises a voice gateway,

the first-modality data comprises a VXML page,

the requestor comprises a voice-over-IP (“

VoIP”

) client on a user device,

the voice gateway sends the message using a voice gateway adaptor,

the second device comprises a browser including a browser adaptor, and

the second-modality data comprises an indicator of a HTML page that corresponds to the VXML page.

17. The method of claim 9 wherein the requester comprises a feature extraction unit on a user device.

18. The method of claim 9 wherein the channel is identified by an identifier of a user.

19. The method of claim 9 wherein:

the first device comprises a server,

the first-modality data comprises a HTML page,

the requestor comprises a browser including a browser adaptor,

the server receives the request from the browser,

the second device comprises a voice gateway including a voice gateway adaptor, and

the second-modality data comprises an indicator of a VXML page that corresponds to the HTML page.

20. The method of claim 19 wherein the server receives the request from the browser through an intermediary selected from a group consisting of a firewall, a proxy, and a gateway.

21. The method of claim 1 wherein the first device and the second device are part of a network that provides access to data from a remote location.

22. The method of claim 1 wherein the first device comprises a device selected from a group consisting of (i) a gateway, (ii) an adaptor, and (iii) a server.

23. The method of claim 1 wherein the requestor comprises an entity selected from a group consisting of (i) a user, (ii) a personal communications device, (iii) a gateway, and (iv) an adaptor.

24. The method of claim 1 wherein the requestor comprises the second device.

25. The method of claim 1 further comprising:

receiving another request at the first device, the other request coming from a second requestor and requesting second first-modality data that includes third content, the second first-modality data being configured to be presented using the first modality; and

sending another message from the first device, the other message being sent in response to receiving the other request, being sent for receipt by a third device, and including second information allowing the third device to request second second-modality data that includes fourth content that overlaps the third content, the second second-modality data being configured to be presented using the second modality.

26. The method of claim 25 wherein the second first-modality data and the second second-modality data may be synchronized by presenting the second first-modality data and the second second-modality data to a second user such that the second user may respond to the overlapping content using either the first modality or the second modality.

27. The method of claim 25 wherein:

the first device comprises a server,

the requester comprises a first browser,

the second device comprises a first voice gateway including a first voice gateway adaptor,

the second requester comprises a second voice gateway including a second voice gateway adaptor, and

the third device comprises a second browser.

28. The method of claim 1 further comprising sending a second message from the first device, the second message being sent in response to receiving the request, being sent for receipt by a third device, and including second information allowing the third device to request third-modality data that includes third content that overlaps both the first content and the second content, the third-modality data being configured to be presented using a third modality.

29. The method of claim 28 wherein the first-modality data, the second-modality data, and the third-modality data may be synchronized by presenting each to the user such that the user may respond to the overlapping content using either the first modality, the second modality, or the third modality.

30. The method of claim 28 wherein:

sending the message and sending the second message are performed by sending a single message for receipt by both the second device and the third device, and

the information comprises the second information.

31. The method of claim 1 further comprising:

sending the first-modality data to the requestor;

receiving a request for the second-modality data from the second device; and

sending the second-modality data to the second device.

32. The method of claim 31 wherein:

the requestor comprises a gateway,

the first device comprises a server, and

the second device comprises another gateway.

33. An apparatus comprising a computer readable medium having instructions stored thereon that when executed result in at least the following:

receiving a request for first-modality data that includes first content, the first-modality data being configured to be presented using a first modality, and the request coming from a requestor and being received at a first device; and

sending a message from the first device for receipt by a second device, the message being sent in response to receiving the request and including information allowing the second device to request second-modality data that includes second content that overlaps the first content, the second-modality data being configured to be presented using a second modality,

wherein the first-modality data and the second-modality data may be synchronized by presenting the first-modality data and the second-modality data to a user such that the user may respond to the overlapping content using either the first modality or the second modality.

34. The apparatus of claim 33 wherein sending the message comprises publishing the message on a channel in a publish/subscribe system to which the second device has already been subscribed.

35. The apparatus of claim 33 further comprising a processing device coupled to the computer readable medium for executing instructions stored thereon.

36. A communication device comprising:

means for receiving a request for first-modality data that includes first content, the first-modality data being configured to be presented using a first modality, and the request coming from a requester and being received at a first device; and

means for sending a message from the first device for receipt by a second device, the message being sent in response to receiving the request and including information allowing the second device to request second-modality data that includes second content that overlaps the first content, the second-modality data being configured to be presented using a second modality,

wherein the first-modality data and the second-modality data may be synchronized by presenting the first-modality data and the second-modality data to a user such that the user may respond to the overlapping content using either the first modality or the second modality.

37. The device of claim 36 wherein:

the means for receiving the request comprises a gateway including a gateway adaptor, and

the means for sending the message comprises the gateway adaptor.

38. The device of claim 36 wherein:

the means for receiving the request comprises a server including a server adaptor, and

the means for sending the message comprises the server adaptor.

39. A method comprising:

requesting first data for a first modality, the first data including first content; and

requesting, automatically after requesting the first data, second data for a second modality, wherein the second data includes second content that overlaps the first content,

wherein the first modality may be synchronized with the second modality by presenting the first content and the second content to a user such that the user may respond to the overlapping content using either the first modality or the second modality.

40. The method of claim 39 further comprising:

presenting the first content to a user using the first modality; and

presenting the second content to the user using the second modality, such that the first content and the second content are presented to the user in an overlapping time period in which the user may respond to the overlapping content using either the first modality or the second modality.

41. An apparatus comprising a computer readable medium having instructions stored thereon that when executed result in at least the following:

requesting first data for a first modality, the first data including first content; and

requesting, automatically after requesting the first data, second data for a second modality, wherein the second data includes second content that overlaps the first content,

wherein the first modality may be synchronized with the second modality by presenting the first content and the second content to a user such that the user may respond to the overlapping content using either the first modality or the second modality.

42. The apparatus of claim 41 wherein execution of the instructions further results in at least the following:

presenting the first content to a user using the first modality; and

presenting the second content to the user using the second modality, such that the first content and the second content are presented to the user in an overlapping time period in which the user may respond to the overlapping content using either the first modality or the second modality.

43. A communication device comprising:

a first-modality user interface coupled to a first gateway, the first-modality user interface configured to request first data including first content for a first modality, and to present to a user the first content using the first modality;

a second gateway configured to automatically request second data for a second modality after the first-modality user interface requests the first data, the second data including second content that overlaps the first content; and

a second-modality user interface being configured to present to the user the second content using the second modality.

44. The device of claim 43 wherein the device is configured to present to the user the first content and the second content such that the user may respond to the overlapping content using either the first modality or the second modality.

45. The device of claim 43 further comprising a synchronization module coupled to the first-modality user interface and the second gateway, the synchronization module being configured to determine the second data to be requested based on the first data.

46. A communication device comprising:

means for requesting first data for a first modality, the first data including first content; and

means for requesting, automatically after requesting the first data, second data for a second modality, wherein the second data includes second content that overlaps the first content,

wherein the first modality may be synchronized with the second modality by presenting the first content and the second content to a user such that the user may respond to the overlapping content using either the first modality or the second modality.

47. The device of claim 46 further comprising:

means for presenting the first content to a user using the first modality; and

means for presenting the second content to the user using the second modality, such that the first content and the second content are presented to the user in an overlapping time period in which the user may respond to the overlapping content using either the first modality or the second modality.

48. A method of presenting updated data in different modalities, the method comprising:

presenting content using a first modality;

presenting the content using a second modality;

receiving input in response to presenting the content, the input being received from the first modality;

presenting automatically new content using the first modality in response to receiving the input, the new content being determined based on the received input; and

presenting automatically the new content using the second modality in response to receiving the input from the first modality.

49. The method of claim 48 wherein presenting the new content using the second modality comprises determining, based on the received input, a second-modality data item that includes the new content.

50. The method of claim 48 wherein the content is presented using the first modality in a first time period, and the content is presented using the second modality in a second time period that overlaps the first time period.

51. An apparatus comprising a storage medium having instructions stored thereon that when executed result in at least the following:

presenting content using a first modality;

presenting the content using a second modality;

receiving input in response to presenting the content, the input being received from the first modality;

presenting automatically new content using the first modality in response to receiving the input, the new content being determined based on the received input; and

presenting automatically the new content using the second modality in response to receiving the input from the first modality.