Reusable mulitmodal application

• US 8,670,754 B2
• Filed: 06/07/2012
• Issued: 03/11/2014
• Est. Priority Date: 05/05/2006
• Status: Active Grant

• Current Assignee: Gula Consulting Limited Liability Company (Intellectual Ventures LLC)
• Original Assignee: Waloomba Tech Limited LLC (Intellectual Ventures LLC)
• Inventors: Anderl, Ewald C.
• Primary Examiner(s): Batista, Marcos
• Application Number:
  US13/491,424
• Publication Number:
  US 20120245946A1
• Time in Patent Office: 642 Days
• Field of Search: 455/414.4, 370/338
• US Class Current: 455/414.4
• CPC Class Codes: G06F 16/178 Techniques for file synchronisation in file systems
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04M 2201/42 Graphical user interfaces
• CPC Class Codes: H04M 2207/18 wireless networks
• CPC Class Codes: H04M 3/42204 Arrangements at the exchange for service or number selection by voice at the terminal H04M1/27
• CPC Class Codes: H04M 3/493 Interactive information services, e.g. directory enquiries ; Arrangements therefor, e.g. interactive voice response [IVR] systems or voice portals
• CPC Class Codes: H04M 3/4938 comprising a voice browser which renders and interprets, e.g. VoiceXML

Abstract

A method and system are disclosed herein for accepting multimodal inputs and deriving synchronized and processed information. A reusable multimodal application is provided on the mobile device. A user transmits a multimodal command to the multimodal platform via the mobile network. The one or more modes of communication that are inputted are transmitted to the multimodal platform(s) via the mobile network(s) and thereafter synchronized and processed at the multimodal platform. The synchronized and processed information is transmitted to the multimodal application. If required, the user verifies and appropriately modifies the synchronized and processed information. The verified and modified information are transferred from the multimodal application to the visual application. The final result(s) are derived by inputting the verified and modified results into the visual application.

Claims

1. A method comprising:

processing, by a mobile device, input information for an application executing on the mobile device, the input information received in a first mode of a plurality of modes of communication;

transmitting, via a communications network, data indicative of the input information to a multimodal platform hosted by a service provider;

receiving, from the multimodal platform, synchronized information generated by synchronizing, by the multimodal platform, the input information using a second mode of the plurality of modes of communication, wherein the mobile device is configured to switch between the first and second modes of communication based on a selection of the first and second modes and to continue execution of the application when switching between the first and second modes; and

rendering the synchronized information for verification and modification in the second mode while continuing execution of the application.

2. The method of claim 1, wherein the plurality of modes of communication comprise audio, visual and haptic modes.

3. The method of claim 1, further comprising:

determining that the input information comprises a search request;

transmitting the search request and associated search grammar via the communications network to the multimodal platform;

receiving search words determined by synchronizing the input information using the second mode;

identifying grammar elements associated with the search request and the associated search grammar; and

determining search words based on the identified grammar elements.

4. The method of claim 1, wherein the rendering is performed by a user interface application.

5. The method of claim 1, wherein the processing is performed by a multimodal application.

6. The method of claim 5, wherein the multimodal application is operable to render a search interface.

7. The method of claim 5, wherein the multimodal application is operable to render an input interface.

8. The method of claim 1, wherein the input information is freeform text.

9. The method of claim 1, wherein the input information is a uniform resource locator address.

10. The method of claim 5, wherein the multimodal application allows users to switch between visual and voice modes.

11. The method of claim 5, wherein the multimodal application allows users to simultaneously use voice and visual modes.

12. The method of claim 5, wherein the multimodal application further comprises global grammar elements that can be invoked by predefined multimodal invocations.

13. A mobile device comprising a processor and memory, the device configured to:

execute a multimodal application operable to receive multimodal commands and send information representative of the multimodal commands to a multimodal platform communicatively coupled to a mobile communications network, the multimodal platform hosted by a service provider and configured to receive and synchronize the information and process the synchronized information to generate synchronized and processed information;

receive the synchronized and processed information via the mobile communications network; and

render the synchronized and processed information for verification and modification of the synchronized and processed information in one or more input modes, the mobile device being further configured to switch between the one or more input modes based on a selection of the one or more input modes and to continue execution of the multimodal application when switching between the modes and during generation of the synchronized and processed information.

14. The mobile device of claim 13, wherein the one or more input modes comprise audio and visual modes.

15. A method for processing, by a multimodal platform hosted by a service provider, multimodal information from a plurality of mobile devices connected to a mobile communications network, the method comprising:

receiving, via the mobile network, voice input information from a mobile device, the voice input information having been processed by a multimodal application operable to execute on the mobile device and receive inputs in voice and visual modes of communication, wherein the multimodal application is operative to allow a user to communicate both in the voice and visual modes of communication and switch between the voice and visual modes of communication based on a selection of one of the voice and visual modes;

synchronizing the voice input information to generate synchronized voice input information, the multimodal application being operative to use the synchronized voice input information for continued execution of the multimodal application using the voice and visual modes of communication; and

processing the synchronized voice input information and transmitting the synchronized and processed information to the mobile device for rendering, verification, and modification of the synchronized and processed information in the selected one of the voice and visual input modes during continued execution of the multimodal application.

16. A method of inputting data on a mobile device, comprising:

in response to an indication that a voice input mode is to be used for receiving inputs to an application executing on the mobile device, executing a voice input function on the mobile device;

transmitting data indicative of voice input information to a multimodal platform via a communications network, the multimodal platform executing on at least one computing device;

receiving, via the communications network, processed information operable for inputting to the executing application, the processed information generated using the data indicative of the voice input information;

rendering the received processed information for verification and modification; and

inputting the verified and modified processed information to the executing application, wherein the mobile device is configured to switch between the voice input mode and another mode of communication based on a user selection of the voice input mode and the another mode of communication and to continue execution of the executing application when switching between modes and during said inputting.

17. The method of claim 16, wherein the rendering comprises displaying the received processed information on a display device.

18. The method of claim 16, wherein the verification and modification is performed by receiving inputs received via a user input mechanism on the mobile device.

19. The method of claim 16, wherein the processed information comprises one or more voice-to-text conversions of the voice input information.

20. A non-transitory computer readable storage medium having computer executable instructions stored thereon, the instructions comprising:

instructions for activating an application on a mobile device;

instructions for, in response to an indication that a voice input mode is to be used for receiving inputs to the activated application, causing execution of a voice input application;

instructions for switching between the voice input mode and another mode of communication based on a selection of one of the voice input mode or the another mode of communication;

instructions for continuing execution of the application when switching between modes;

instructions for receiving voice input information by the voice input application;

instructions for sending data indicative of the voice input information to a multimodal application via a mobile communications network, the multimodal application executing on another device;

instructions for receiving information processed by the multimodal application using the data indicative of the voice input information;

instructions for rendering the received processed information for verification and modification by the voice input application; and

instructions for inputting the verified and modified processed information to the activated application while continuing execution of the activated application.