CONVERSATIONAL COMPUTING VIA CONVERSATIONAL VIRTUAL MACHINE

• US 20070043574A1
• Filed: 10/23/2006
• Published: 02/22/2007
• Est. Priority Date: 10/02/1998
• Status: Active Grant

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: Nahamoo, David, Gopalakrishnan, Ponani, DeGennaro, Steven, Coffman, Daniel, Comerford, Liam, Maes, Stephan, Epstein, Edward
• Granted Patent: US 7,729,916 B2
• Time in Patent Office: Days
• US Class Current: 704/275
• CPC Class Codes: G06F 16/957 Browsing optimisation, e.g. caching or content distillation
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/285 Memory allocation or algorithm optimisation to reduce hardware requirements
• CPC Class Codes: G10L 2015/228 of application context
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04M 1/72445 for supporting Internet browser applications
• CPC Class Codes: H04M 2201/40 using speech recognition
• CPC Class Codes: H04M 2201/60 Medium conversion
• CPC Class Codes: H04M 2203/355 Interactive dialogue design tools, features or methods
• CPC Class Codes: H04M 2250/74 with voice recognition means
• CPC Class Codes: H04M 3/42204 Arrangements at the exchange for service or number selection by voice at the terminal H04M1/27
• CPC Class Codes: H04M 3/44 Additional connecting arrangements for providing access to frequently-wanted subscribers, e.g. abbreviated dialling at the subscriber's set H04M1/27; automatic redialling H04M3/424
• CPC Class Codes: H04M 3/493 Interactive information services, e.g. directory enquiries ; Arrangements therefor, e.g. interactive voice response [IVR] systems or voice portals
• CPC Class Codes: H04M 3/4931 Directory assistance systems
• CPC Class Codes: H04M 3/4936 Speech interaction details speech recognition per se G10L15/00
• CPC Class Codes: H04M 3/4938 comprising a voice browser which renders and interprets, e.g. VoiceXML
• CPC Class Codes: H04M 3/50 Centralised arrangements for answering calls; Centralised arrangements for recording messages for absent or busy subscribers H04M3/487 takes precedence; centralised dictation systems H04M11/10; Centralised arrangements for recording messages
• CPC Class Codes: H04M 7/00 Arrangements for interconnection between switching centres

Abstract

A conversational computing system that provides a universal coordinated multi-modal conversational user interface (CUI) 10 across a plurality of conversationally aware applications (11) (i.e., applications that “speak” conversational protocols) and conventional applications (12). The conversationally aware applications (11) communicate with a conversational kernel (14) via conversational application APIs (13). The conversational kernel 14 controls the dialog across applications and devices (local and networked) on the basis of their registered conversational capabilities and requirements and provides a unified conversational user interface and conversational services and behaviors. The conversational computing system may be built on top of a conventional operating system and APIs (15) and conventional device hardware (16). The conversational kernel (14) handles all I/O processing and controls conversational engines (18). The conversational kernel (14) converts voice requests into queries and converts outputs and results into spoken messages using conversational engines (18) and conversational arguments (17). The conversational application API (13) conveys all the information for the conversational kernel (14) to transform queries into application calls and conversely convert output into speech, appropriately sorted before being provided to the user.

Claims

1. A method for providing conversational computing between a user and one of a single application and a plurality of applications, the method comprising the steps of:

engaging in dialog with the user; and

processing the dialog to one of complete a query, disambiguate a query, summarize a query, correct a query, correct a result of an executed task, communicate the result of such execution, determine a target application of an input/output event, and a combination thereof, based on one of past dialog history, context, user preferences, meta information, and a combination thereof.

2. The method of claim 1, wherein the step of engaging in dialog comprises engaging in multi-modal dialog.

3. The method of claim 1, wherein the step of engaging in dialog comprises engaging in mixed-initiative dialog.

4. The method of claim 1, wherein the step of engaging in dialog comprises engaging in natural language dialog.

5. The method of claim 1, wherein the step of processing the dialog comprises deriving a user-specified action from the dialog and dispatching the user-specified action.

6. The method of claim 1, further comprising the step of adapting the rendering of an application to one of a registered dialog modality, device capability and channel capability.

7. The method of claim 1, further comprising the step of maintaining the context of each active task.

8. The method of claim 7, wherein the context comprises one of short term history, long term history, and both short-term and long-term history.

9. The method of claim 7, wherein the step of maintaining the context comprises maintaining one of query arguments, I/O events, application state, conversational arguments, application events, and a combination thereof, associated with the active task.

10. The method of claim 1, further comprising the step of presenting a unified and coordinated user interface across the plurality of applications.

11. The method of claim 10, wherein the step of presenting a unified and coordinated user interface across the plurality of applications comprises coordinating a plurality of tasks spawned one of locally and remotely.

12. The method of claim 10, wherein the step of presenting a unified and coordinated user interface across the plurality of applications comprises dynamically negotiating a topology among the applications based on their registered information.

13. The method of claim 12, wherein the step of dynamically negotiating a topology comprises the step of repeatedly renegotiating the topology among the applications during a dialog session based on currently registered application states and resource requirements and capabilities.

14. The method of claim 10, wherein the step of presenting a unified and coordinated user interface across a plurality of applications comprises the steps of:

registering, by each application, information comprising application state, application modes, arguments, context, modalities, engine resources, and a combination thereof; and

managing the dialog across the plurality of applications based on their registered information.

15. The method of claim 1, further comprising the steps of arbitrating a response to an input query from each application to determine the target application.

16. The method of claim 15, further comprising the steps of one of prompting the user for additional input, checking for user confirmation, clarifying an interpretation of user intent, dispatching a user action that is derived from user input, and a combination thereof, after determining the target application.

17. The method of claim 14, wherein the step of managing the dialog comprises the step of coordinating the dialog through one of a master/slave, client/server, and peer-to-peer topology among the applications.

18. The method of claim 10, wherein the step of presenting a unified and coordinated user interface across the plurality of applications comprises managing the dialog through one of a centralized and distributed context.

19. The method of claim 10, wherein the step of presenting a unified and coordinated user interface across the plurality of applications comprises the step of discovering conversationally aware applications and systems.

20. The method of claim 1, further comprising the step of abstracting one of a user query, context, manipulated information, and a combination thereof, based on one of contexts, applications, history, user preferences, biometrics, and a combination thereof.

21. The method of claim 1, farther comprising the steps of:

associating abstract meta-information with elements, wherein the elements comprises one of objects, files, applications, data stream handles, networks, peripherals, hardware, a local file system, a remote file system, dispatched tasks, links, macros, shortcuts, bookmarks, user preferences, user habits and behavior, biometrics, a-priori knowledge acquired, acquired user knowledge, learned user knowledge, and a combination thereof; and

maintaining the abstract meta information of the elements.

22. The method of claim 21, wherein the step of associating abstract meta information comprises one of registering an abstract meta information category of an elements creating a new abstract meta information category of an element, and both.

23. The method of claim 21, further comprising the step of searching for a target element using the abstract meta information.

24. The method of claim 23, wherein the step of searching comprises searching the categories through multi-modal conversational dialog that comprises filling sets of attribute values out of set of possible types of queries and refining or modifying the result of current search queries based on similarities of the associated categories.

25. The method of claim 21, further comprising the step of selecting a desired element using the abstract meta information.

26. The method of claim 25, wherein the step of selecting comprises selecting the categories through multi-modal conversational dialog that comprises filling sets of attribute values out of set of possible types of queries and refining or modifying the result of current selection queries based on similarities of the associated categories.

27. The method of claim 21, wherein the meta-information comprises security meta-information, further comprising the step of automatically authenticating a user when a query to a restricted element is made, based on the security meta-information associated with the element.

28. The method of claim 1, further comprising the steps of coordinating and prioritizing output generation based on one of an active dialog, history, and a combination thereof.

29. The method of claim 1, further comprising the step of:

memorizing an event, wherein the event comprises one of a user command, a user preference, an I/O event, and results of an executed task; and

subsequently activating the memorized event at a desired time during the dialog.

30. The method of claim 29, wherein the step of activating comprises activating the memorized event for presentation to the user.

31. The method of claim 29, wherein the step of activating comprises activating the memorized event to resume a memorized dialog commencing from a state in which it was memorized.

32. The method of claim 29, wherein the memorized event is used to undo the effect and context impact of the memorized dialog.

33. The method of claim 29, wherein the memorized event is activated prior to dispatching an agent or executing a task.

34. The method of claim 29, wherein the step of memorizing comprises storing a current context, at the time of memorization of the event, as meta-information.

35. The method of claim 29, further comprising the steps of:

summarizing the memorized event; and

activating the event summarization for presentation of the event summarization to the user.

36. The method of claim 35, wherein the step of activating the event summarization is performed by one of a user query and a triggering event specified by an application.

37. The method of claim 1, further comprising the step of customizing the dialog based on one of user identity, usage history of the user, user preferences, active applications, context, and a combination

38. A program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform method steps for providing conversational computing between a user and one of a single application and a plurality of applications, the method steps comprising:

engaging in dialog with the user; and

processing the dialog to one of complete a query, disambiguate a query, summarize a query, correct a query, correct a result of an executed task, communicate the result of such execution, determine a target application of an input/output event, and a combination thereof, based on one of past dialog history, context, user preferences, meta information, and a combination thereof.

39. A method for performing a transaction or accessing or modifying information through conversational dialog, comprising interacting between a user and an application or multiple applications through multi-modal dialog, wherein the multi-modal dialog comprises one of GUI (graphic user interface) I/O (input/output), text I/O, speech I/O, DTMF (dual tone multi frequency) input and a combination thereof, and processing the multi-modal dialog based on one of past dialog history, context, user preferences, user meta-information, application meta-information, device meta-information, domain meta-information, and a combination thereof, to determine a target of an I/O event, and to perform one of prompting a user for additional input, checking for user confirmation, clarifying an interpretation of user intent, dispatching a user specified action that is derived from a user input, and a combination thereof.