System and method for concurrent multimodal communication

• US 20030167172A1
• Filed: 02/27/2002
• Published: 09/04/2003
• Est. Priority Date: 02/27/2002
• Status: Active Grant

• Current Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Original Assignee: Motorola, Inc. (Motorola Solutions, Inc.)
• Inventors: Balasuriya, Senaka, Johnson, Greg, Ferrans, James, Galagedara, Dilani, Jahnke, Jerome, Pierce, Rainu, Cuka, David
• Granted Patent: US 6,807,529 B2
• Time in Patent Office: Days
• US Class Current: 704/270.1
• CPC Class Codes: G06F 16/957 Browsing optimisation, e.g. caching or content distillation
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: H04M 1/72445 for supporting Internet browser applications
• CPC Class Codes: H04M 3/4938 comprising a voice browser which renders and interprets, e.g. VoiceXML

Abstract

A multimodal network element facilitates concurrent multimodal communication sessions through differing user agent programs on one or more devices. For example, a user agent program communicating in a voice mode, such as a voice browser in a voice gateway that includes a speech engine and call/session termination, is synchronized with another user agent program operating in a different modality, such as a graphical browser on a mobile device. The plurality of user agent programs are operatively coupled with a content server during a session to enable concurrent multimodal interaction.

Claims

1. A method for multimodal communication comprising:

obtaining modality specific instructions for a plurality of user agent programs that operate in different modalities with respect to each other; and

during a session, synchronizing output from the plurality of user agent programs based on the modality specific instructions.

2. The method of claim 1 wherein the step of synchronizing output by the plurality of user agent programs includes sending, for use by the plurality of user agent programs, at least portions of markup language representing different modalities such that each user agent program outputs information in different modalities based on the at least portions of markup language.

3. The method of claim 2 wherein one of the plurality of user agent programs includes a graphical browser and wherein another of the plurality of user agent programs includes a voice browser and wherein the output from the plurality of user agent programs includes user input entered concurrently in different modalities through the plurality of user agent programs.

4. The method of claim 2 wherein the step of sending, for use by the plurality of user agent programs, at least portions of markup language representing different modalities includes sending markup language forms associated with different modalities to different devices wherein each of the devices operates one of the plurality of user agent programs.

5. The method of claim 2 wherein the step of sending, for use by the plurality of user agent programs, at least portions of markup language representing different modalities includes sending markup language forms associated with different modalities to a same device wherein the device operates the plurality of user agent programs in different modalities.

6. The method of claim 1 including the step of determining, during a session, a proxy for communication with each of the plurality of user agent programs.

7. The method of claim 2 wherein the step of obtaining modality specific instructions for the plurality of user agent programs includes communicating with an application that provides differing markup language formats for each of the plurality of user agent programs wherein the differing markup language formats represent different modalities.

8. The method of claim 1 wherein obtaining modality specific instructions for the plurality of user agent programs that operate in different modalities includes fetching a markup language form written in a base markup language representing modality specific instructions for at least one the user agent programs, and wherein the markup language form contains data identifying modality specific instructions for another user agent program operating in a different modality.

9. The method of claim 1 wherein synchronizing output from the plurality of user agent programs based on the modality specific instructions includes analyzing fetched modality specific instructions from a multimodal application to detect a concurrent multimodal tag (CMMT) and if detected, obtaining modality specific instructions for at least one user agent program based on the CMMT.

10. A multimodal network element comprising:

an information fetcher operative to obtain modality specific instructions for a plurality of user agent programs that operate in different modalities with respect to each other during a same session; and

a concurrent multimodal synchronization coordinator, operatively coupled to the information fetcher and operative to, during the session, synchronize output from the plurality of user agent programs based on the modality specific instructions.

11. The multimodal network element of claim 10 wherein the concurrent multimodal synchronization coordinator determines, during a session, a proxy for communication with each of the plurality of user agent programs.

12. A method for multimodal communication comprising:

sending a request for concurrent multimodal input information corresponding to multiple input modalities associated with a plurality of user agent programs operating during a same session; and

fusing received concurrent multimodal input information sent from the plurality of user agent programs sent in response to the request for concurrent different multimodal information.

13. The method of claim 12 including the step of:

prior to sending the request for concurrent multimodal input information, determining a proxy for each different modality associated with each application associated with a different modality for a given session; and

synchronizing the request for concurrent different multimodal input information to the plurality of user agent programs using the proxies determined for each different modality.

14. The method of claim 12 including waiting a period of time to determine whether the concurrent multimodal input information was suitably received for fusion.

15. The method of claim 14 including waiting a different period of time depending on a modality of each user agent program.

16. The method of claim 12 wherein sending the request for concurrent multimodal input information corresponding to multiple input modalities includes sending modality specific instructions for a plurality of user agent programs that operate in different modalities with respect to each other to request concurrent entry of information in different modalities.

17. The method of claim 12 wherein the step of sending the request for concurrent multimodal input information corresponding to multiple input modalities includes sending a first modality based markup language form to a device and sending a second modality markup language based form to one or more devices to request concurrent entry of information in different modalities.

18. The method of claim 12 including the step of combining received multimodal input information sent from one of the plurality of user agent programs sent in response to the request for concurrent different multimodal information based on a time received and based on confidence values of individual results received.

19. The method of claim 12 including transcoding modality specific instructions to include data identifying modality specific instructions for another user agent program operating in a different modality.

20. A multimodal network element comprising:

a plurality of proxies that each send a request for concurrent multimodal input information corresponding to multiple input modalities associated with a plurality user agent programs operating during a same session; and

a multimodal fusion engine, operatively responsive to received concurrent multimodal input information sent from the plurality of user agent programs sent in response to the request for concurrent different multimodal information and operative to fuse the different multimodal input information sent from the plurality of user agent programs to provide concurrent multimodal communication from differing user agent programs during a same session.

21. The multimodal network element of claim 20 wherein the request for concurrent different multimodal information includes modality specific instructions for the plurality of user agent programs that operate in different modalities with respect to each other to request concurrent entry of information in different modalities, and wherein the multimodal network element includes:

an information fetcher operative to obtain modality specific instructions for the plurality user agent programs that operate in different modalities with respect to each other during a same session; and

a concurrent multimodal synchronization coordinator, operatively coupled to the information fetcher and the plurality of proxies, operative to synchronize the received concurrent multimodal input information output from the plurality of user agent programs, during a session.

22. The multimodal network element of claim 21 including:

a concurrent multimodal session persistence controller operatively coupled to the plurality user agent programs, to maintain, during non-session conditions and on a per user basis, concurrent multimodal session status information of the plurality user agent programs configured for different concurrent modality communication during a session and to re-establish a concurrent multimodal session in response to accessing the concurrent multimodal session status information; and

memory, operatively coupled to the concurrent multimodal session persistence controller, containing the concurrent multimodal session status information.

23. The multimodal network element of claim 20 including a markup language interpreter and a transcoder, operatively coupled to the markup language interpreter, wherein the transcoder transcodes modality specific instructions to include data identifying modality specific instructions for another user agent program operating in a different modality.