Conversational computing via conversational virtual machine

• US 7,137,126 B1
• Filed: 10/01/1999
• Issued: 11/14/2006
• Est. Priority Date: 10/02/1998
• Status: Expired due to Term

• Current Assignee: Uniloc 2017 LLC (FIG LLC (d/b/a Fortress Investment Group LLC))
• Original Assignee: International Business Machines Corporation
• Inventors: Coffman, Daniel, Nahamoo, David, Maes, Stephane H., Gopalakrishnan, Ponani, Epstein, Edward A., DeGennaro, Steven, Comerford, Liam D.
• Primary Examiner(s): BULLOCK JR, LEWIS ALEXANDER
• Application Number:
  US09/806,565
• Time in Patent Office: 2,601 Days
• Field of Search: 719/328, 719/310, 709200-203, 709217-226, 704/200, 704/231, 718/1
• US Class Current: 719/328
• CPC Class Codes: G06F 16/957 Browsing optimisation, e.g. caching or content distillation
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/285 Memory allocation or algorithm optimisation to reduce hardware requirements
• CPC Class Codes: G10L 2015/228 of application context
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04M 1/72445 for supporting Internet browser applications
• CPC Class Codes: H04M 2201/40 using speech recognition
• CPC Class Codes: H04M 2201/60 Medium conversion
• CPC Class Codes: H04M 2203/355 Interactive dialogue design tools, features or methods
• CPC Class Codes: H04M 2250/74 with voice recognition means
• CPC Class Codes: H04M 3/42204 Arrangements at the exchange for service or number selection by voice at the terminal H04M1/27
• CPC Class Codes: H04M 3/44 Additional connecting arrangements for providing access to frequently-wanted subscribers, e.g. abbreviated dialling at the subscriber's set H04M1/27; automatic redialling H04M3/424
• CPC Class Codes: H04M 3/493 Interactive information services, e.g. directory enquiries ; Arrangements therefor, e.g. interactive voice response [IVR] systems or voice portals
• CPC Class Codes: H04M 3/4931 Directory assistance systems
• CPC Class Codes: H04M 3/4936 Speech interaction details speech recognition per se G10L15/00
• CPC Class Codes: H04M 3/4938 comprising a voice browser which renders and interprets, e.g. VoiceXML
• CPC Class Codes: H04M 3/50 Centralised arrangements for answering calls; Centralised arrangements for recording messages for absent or busy subscribers H04M3/487 takes precedence; centralised dictation systems H04M11/10; Centralised arrangements for recording messages
• CPC Class Codes: H04M 7/00 Arrangements for interconnection between switching centres

Abstract

A conversational computing system that provides a universal coordinated multi-modal conversational user interface (CUI) (10) across a plurality of conversationally aware applications (11) (i.e., applications that “speak” conversational protocols) and conventional applications (12). The conversationally aware maps, applications (11) communicate with a conversational kernel (14) via conversational application APIs (13). The conversational kernel (14) controls the dialog across applications and devices (local and networked) on the basis of their registered conversational capabilities and requirements and provides a unified conversational user interface and conversational services and behaviors. The conversational computing system may be built on top of a conventional operating system and APIs (15) and conventional device hardware (16). The conversational kernel (14) handles all I/O processing and controls conversational engines (18). The conversational kernel (14) converts voice requests into queries and converts outputs and results into spoken messages using conversational engines (18) and conversational arguments (17). The conversational application API (13) conveys all the information for the conversational kernel (14) to transform queries into application calls and conversely convert output into speech, appropriately sorted before being provided to the user.

Claims

1. A conversational computing system, comprising:

a multi-modal CUI (conversational user interface) manager, operatively connected to a plurality of I/O (input/output) renderers, which can receive input queries and input events across different user interface modalities of different active applications and generate output messages and output events in connection with the active applications in one or more of the different user interface modalities;

a conversational kernel for generating multi-modal dialogs in response to the input queries and input events, and for managing context associated with the active applications;

a conversational API (application program interface) for providing an interface between the active applications and the conversational kernel;

a plurality of I/O resources; and

an I/O API for interfacing with the plurality of I/O resources and for registering the plurality of I/O resources with the conversational kernel.

2. The system of claim 1, further comprising:

a conversational engine API; and

a plurality of conversational engines, wherein the conversational kernel controls and accesses the conversational engines through the conversational engine API to process the input queries and input events and to generate the multi-modal dialog and output events.

3. The system of claim 2, wherein the conversational engines comprise NLU (natural language understanding) and NLG (natural language generation) engines, and wherein the multi-modal dialogs comprise NLU and NLG dialogs.

4. The system of claim 3, wherein the conversational kernel provides support at the dialog management level for NL (natural language) and use of context for disambiguation and mixed initiative.

5. The system of claim 1, wherein the conversational kernel provides conversational services and behaviors that are accessible by an application through the conversational API.

6. The system of claim 5, wherein the conversational services and behaviors comprise one of conversational security, conversational customization, conversational search, conversational selection, conversational help, conversational memorization, conversational summarization, conversational agents, conversational formatting, conversational prioritization, conversational re-direction, conversational categorization, conversational abstraction and object management, conversational navigation, conversational undo, and a combination thereof.

7. The system of claim 1, wherein the conversational API comprises a library comprising library functions of conversational foundation classes (CFCs) and fundamental dialog components, wherein the CFCs and fundamental dialog components are accessible through the library for constructing conversational objects, wherein the conversational objects are employed for one of performing conversational procedures and building conversational applications.

8. The system of claim 7, wherein the CFCs can share context and be activated in parallel within an application or across a plurality of applications.

9. The system of claim 8, wherein the CFCs and fundamental dialog components comprise one of CUI building blocks, conversational platform libraries, dialog modules, dialog scripts, beans, conversational gestures, and a combination thereof.

10. The system of claim 9, further comprising a conversational browser, wherein the CFCs, dialog components, CUI building blocks, conversational platform libraries, dialog modules, dialog scripts, beans, conversational gestures, and conversational API are loaded into the system through CML (conversational markup language) pages that are processed via the conversational browser.

11. The system of claim 7, wherein the conversational objects are implemented one of declaratively, imperatively and a combination thereof.

12. The system of claim 1, wherein the conversational kernel is a platform that executes on top of one of an operating system, a real-time operating system, a virtual machine and a browser.

13. The system of claim 1, wherein the conversational kernel executes as an operating system service layer.

14. The system of claim 1, wherein the system executes on one of a personal computer platform, a server platform, an embedded client device platform and a distributed combination of these platforms.

15. The system of claim 1, wherein the conversational kernel comprises a conversational transcoder for providing adaptation of the behavior, CUI and dialog presented to a user based on capabilities of the I/O resources and conversational engines.

16. The system of claim 1, wherein the I/O API layer comprises one of I/O abstractions, user interface abstractions, device abstractions and a combination thereof.

17. The system of claim 1, wherein the conversational kernel comprises:

a dialog manager for managing dialog across active applications and for selecting an active dialog, context and application based on input queries;

a resource manager for managing and allocating conversational engines that are used during execution of conversational tasks;

a conversational task dispatcher for coordinating and dispatching conversational tasks; and

a context stack for accumulating a context of an active discourse of a conversational task, the context comprising query arguments, a list of attribute value—

uples and conversational state.

18. The system of claim 17, wherein the context stack comprises a global history of context.

19. The system of claim 17, wherein the context stack comprises a meta-information repository.

20. The system of claim 19, wherein the meta-information repository comprises a-priori known information.

21. The system of claim 19, further comprising a meta information manager for managing the meta-information repository, wherein the meta-information comprises a plurality of abstract categories associated with elements comprising one of files, directories, objects, data stream handles, networks, peripherals, hardware, applications, networked file systems and a combination thereof.

22. The system of claim 21, wherein the meta-information is used to provide shortcuts to the elements.

23. The system of claim 21, wherein a user can navigate the categories of the meta-information through multi-modal conversational dialogs that comprise filling sets of attribute values out of a set of possible query types, and wherein the user can refine or modify the result of current navigation queries based on similarities of the corresponding categories.

24. The system of claim 21, wherein the meta-information provides a-priori knowledge information for the categories of meta-information.

25. The system of claim 19, wherein the meta-information comprises user meta-information representing one of user preferences, security, habits, biometrics, behavior and a combination thereof.

26. The system of claim 25, wherein the user meta-information is one of provided directly by the user, learned from past usage of the system by the user, and a combination thereof.

27. The system of claim 17, wherein the conversational kernel further comprises a back-end abstraction layer for accessing back-end logic via the dialog manager.

28. A conversational computing system, comprising:

a multi-modal CUI (conversational user interface) manager, operatively connected to a plurality of I/O (input/output) renderers, which can receive input queries and input events across different user interface modalities of different active applications and generate output messages and output events in connection with the active applications in one or more of the different user interface modalities;

a conversational kernel for generating multi-modal dialogs in response to the input queries and input events, and for managing context associated with the active applications;

a conversational API (application program interface) for providing an interface between the active applications and the conversational kernel; and

a communication stack that implements conversational protocols for exchanging information with a conversationally aware system, wherein the conversationally aware system comprises one of a remote application, a remote device, a remote conversational computing system, and a combination thereof.

29. The system of claim 28, wherein the conversational protocols comprise distributed conversational protocols for exchanging information comprising one of conversational state, conversational arguments, context, conversational engine API calls, results, and a combination thereof.

30. The system of claim 28, wherein information exchanged between applications through the conversational protocols comprises a description of a conversational state of the applications, wherein the information representing a conversational state comprises one of a grammar, a vocabulary, acoustic models, language models, prompts, synthesis details, parsing details, tagging details, information to manage the dialog, and a combination thereof.

31. The system of claim 28, wherein information exchanged through the conversational protocols comprises a modality-independent description of an interface of a remote control unit, wherein the interface allows the user to remotely control a conversationally aware appliance though the remote control unit.

32. The system of claim 28, wherein the conversational protocols comprise conversational discovery protocols for automatically discovering the conversationally aware systems, and wherein the information exchanged through the conversational discovery protocols comprises broadcast requests for handshake, exchange of identifiers, exchange of handles for first registration and exchange of handles for first negotiation.

33. The system of claim 28, wherein the conversational protocols comprise conversational negotiation protocols for exchanging information to negotiate network topology between the system and a conversationally aware system.

34. The system of claim 28, wherein the conversational protocols comprise conversational registration protocols for exchanging registering information, wherein the registering information comprises one of conversational capabilities, conversational state, context, and a combination thereof.

35. A computer readable medium having stored thereon a set of instructions that are executable by a computer to implement a conversational virtual machine, the conversational virtual machine comprising:

a kernel adapted to manage dialog and context, conversational engines and resources and communication across different platforms, applications, devices or combinations thereof, each having one or more different user interface modalities, to provide a coordinated, universal conversational user interface (CUI) across the different user interface modalities;

an API (application program interface) comprising abstractions adapted to access conversational services from the kernel on behalf of said platforms, applications, and devices or combinations thereof;

a dialog manager adapted to manage conversational dialog across registered applications; and

a context stack for maintaining the context of an active application or task under the control of the dialog manager.

36. The computer readable medium of claim 35, wherein the conversational virtual machine is a shell that executes on top of one of an operating system, a real-time operating system, a virtual machine, and a conversational browser.

37. The computer readable medium of claim 35, wherein the conversational virtual machine executes as an operating system service layer.

38. The computer readable medium of claim 35, further comprising an engine API comprising abstractions adapted to access a conversational engine.

39. The computer readable medium of claim 35, wherein the API comprises a plurality of conversational foundation classes that are accessible through library functions for building one of conversationally aware applications, dialog components for performing specific reusable dialog tasks, and both.

40. The computer readable medium of claim 39, wherein the conversational foundation classes comprise conversational gestures that characterize a modality-independent dialog.

41. The computer readable medium of claim 35, wherein the conversationally aware applications and dialog components are implemented one of declaratively, imperatively, and a combination thereof.

42. The computer readable medium of claim 35, wherein the kernel comprises a task manager adapted to drive a conversational engine and a task, process or thread running on the conversational virtual machine.

43. The computer readable medium of claim 35, wherein the kernel comprises:

a resource manager adapted to manage one of local resources, distributed resources, and both; and

an input/output (I/O) manager adapted to manage multi-modal I/O events.

44. The computer readable medium of claim 35, wherein the kernel further comprises an arbitrator for arbitrating a target application of an I/O event between the registered applications.

45. The computer readable medium of claim 35, wherein the API comprises conversational protocols adapted to provide distribution of the conversational virtual machine.

46. The computer readable medium of claim 45, wherein the distribution of the conversational virtual machine comprises distribution of functions and components of the conversational virtual machine across multiple devices or resources that collectively contribute to providing a complete functionality of the conversational virtual machine.

47. The computer readable medium of claim 45, wherein the distribution of the conversational virtual machine comprises distribution of functions of the conversational virtual machine among a plurality of conversational virtual machines.

48. The computer readable medium of claim 47, wherein the role of each of the plurality of conversational virtual machines is dynamically negotiated between the conversational virtual machines.

49. The computer readable medium of claim 45, wherein the distribution of the conversational virtual machine comprises distribution of the dialog manager and context stack across the registered applications.

50. The computer readable medium of claim 45, wherein the conversational protocols comprise conversational distributed protocols adapted to allow coordination of a dialog among a plurality of applications and devices.

51. The computer readable medium of claim 50, wherein the plurality of applications and devices comprise silent partners.

52. The computer readable medium of claim 45, wherein the conversational protocols comprise negotiation protocols adapted to allow the kernel to dynamically negotiate one of a master/slave, a client/server, and a peer-to-peer topology with at least one kernel of a remote conversational virtual machine.

53. The computer readable medium of claim 35, wherein the conversational virtual machine transcribes input events into ASCII input streams and processes the ASCII input streams as objects.

54. The computer readable medium of claim 53, wherein the ASCII input streams comprise a list of attribute-value n-uples.

55. The computer readable medium of claim 35, wherein the conversational virtual machine is implemented as an interface for a UCRC (universal conversational remote control), wherein the UCRC is used to control home appliances that are conversationally aware.

56. The computer readable medium of claim 55, wherein the UCRC comprises a speech-enabled PDA (personal digital assistant) device.

57. A computer readable medium having stored thereon a set of instructions that are executable by a computer to implement a conversational virtual machine, the conversational virtual machine comprising:

a kernel adapted to manage dialog and context conversational engines and resources and communication across different platforms, applications, devices or combinations thereof, each having one or more different user interface modalities, to provide a coordinated, universal conversational user interface (CUI) across the different user interface modalities;

an API (application program interface) comprising abstractions adapted to access conversational services from the kernel on behalf of said platforms, applications, and devices or combinations thereof,

wherein the conversational virtual machine compares dialogs with conversational logic.

58. The computer readable medium of claim 57, wherein the conversational logic comprises logic statements including one of true, false, incomplete, ambiguous, different/equivalent for ASCII comparison, different/equivalent for NLU-converted query comparison, different/equivalent for active query field comparison, unknown, incompatible, incomparable, and a combination thereof.

59. A conversational computing system, comprising:

a multi-modal CUI (conversational user interface) manager, operatively connected to a plurality of I/O (input/output) renderers, which can receive input queries and input events across different user interface modalities of different active applications and generate output messages and output events in connection with the active applications in one or more of the different user interface modalities;

a conversational kernel for generating multi-modal dialogs in response to the input queries and input events, and for managing context associated with the active applications;

a conversational API (application program interface) for providing an interface between the active applications and the conversational kernel;

wherein the conversational kernel comprises;

a dialog manager for managing dialog across active applications and for selecting an active dialog, context and application based on input queries;

a resource manager for managing and allocating conversational engines that are used during execution of conversational tasks;

a conversational task dispatcher for coordinating and dispatching conversational tasks; and

a context stack for accumulating a context of an active discourse of a conversational task, the context comprising query arguments, a list of attribute value—

uples and conversational state.

60. The system of claim 59, wherein the context stack comprises a global history of context.

61. The system of claim 59, wherein the context stack comprises a meta-information repository.

62. The system of claim 61, wherein the meta-information repository comprises a-priori known information.

63. The system of claim 61, further comprising a meta information manager for managing the meta-information repository, wherein the meta-information comprises a plurality of abstract categories associated with elements comprising one of files, directories, objects, data stream handles, networks, peripherals, hardware, applications, networked file systems and a combination thereof.

64. The system of claim 63, wherein the meta-information is used to provide shortcuts to the elements.

65. The system of claim 63, wherein a user can navigate the categories of the meta-information through multi-modal conversational dialogs that comprise filling sets of attribute values out of a set of possible query types, and wherein the user can refine or modify the result of current navigation queries based on similarities of the corresponding categories.

66. The system of claim 63, wherein the meta-information provides a-priori knowledge information for the categories of meta-information.

67. The system of claim 61, wherein the meta-information comprises user meta-information representing one of user preferences, security, habits, biometrics, behavior and a combination thereof.

68. The system of claim 67, wherein the user meta-information is one of provided directly by the user, learned from past usage of the system by the user, and a combination thereof.

69. The system of claim 59, wherein the conversational kernel further comprises a back-end abstraction layer for accessing back-end logic via the dialog manager.