Methods and systems for authoring of mixed-initiative multi-modal interactions and related browsing mechanisms

• US 20030225825A1
• Filed: 05/28/2002
• Published: 12/04/2003
• Est. Priority Date: 05/28/2002
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Hosn, Rafah A., Raman, Thiruvilvama Lai V., Tiwari, Alpana, Maes, Stephane Herman, Healey, Jennifer A.
• Granted Patent: US 7,546,382 B2
• Time in Patent Office: Days
• US Class Current: 709/203
• CPC Class Codes: G06F 8/00 Arrangements for software engineering testing or debugging G06F11/36; administrative, planning or organisation aspects of software project management G06Q10/06

Abstract

Application authoring techniques, and information browsing mechanisms associated therewith, which employ programming in association with mixed-initiative multi-modal interactions and natural language understanding for use in dialog systems. Also, a conversational browsing architecture is provided for use with these and other authoring techniques.

Claims

1. A method of generating an application accessible by a user in accordance with a dialog system, the method comprising the steps of:

representing interactions that the user may have with the dialog system as a data model and one or more user interaction elements that populate an application state of the data model and that are bound thereto, such that a dialog that the user has with the dialog system may be a mixed-initiative dialog;

wherein at least a portion of the one or more user interaction elements can be transformed or associated to one or more modality-specific renderings of the application which are presentable to the user and are one of selected and generated by a dialog manager algorithm.

2. The method of claim 1, wherein the representing step is performed declaratively.

3. The method of claim 1, wherein the transforming operation comprises transcoding.

4. The method of claim 1, wherein the associating operation comprises selecting a precompiled presentation snippet.

5. The method of claim 1, wherein the associating operation comprises use of an authored presentation.

6. The method of claim 1, wherein the application state is represented as at least one form having one or more slots that are populatable by the one or more user interaction elements in accordance with the user'"'"'s interaction with the dialog system.

7. The method of claim 6, wherein at least one of the one or more slots of the application state is populatable in response to semantic meaning extracted from a user utterance in accordance with a natural language understanding engine associated with the dialog system.

8. The method of claim 7, wherein population of the one or more slots serves to interpret the intention of the user such that the mixed-initiative dialog may occur between the user and the dialog system.

9. The method of claim 1, wherein the one or more modality-specific renderings comprise a speech-based representation of portions of the application.

10. The method of claim 9, wherein the speech-based representation is based on Voice Extensible Markup Language (VoiceXML).

11. The method of claim 10, wherein VoiceXML is extended to support natural language understanding of a user input and generation of resulting attribute value pairs.

12. The method of claim 11, wherein the extension of VoiceXML comprises support for a language model.

13. The method of claim 10, wherein VoiceXML is extended to specify natural language (NL) parsing via introduction of a NL grammar as a speech data file.

14. The method of claim 13, wherein the extension of VoiceXML comprises support for a natural language grammar tag.

15. The method of claim 1, wherein the one or more modality-specific renderings comprise a visual-based representation of portions of the application.

16. The method of claim 15, wherein the visual-based representation comprises a menu in a frame or a portion of a graphical user interface and an active region under focus.

17. The method of claim 15, wherein the visual-based representation is based on at least one of HyperText Markup Language (HTML), Extensible HTML (XHTML), and XHTML-Mobile Profile.

18. The method of claim 1, wherein the one or more modality-specific renderings comprise a representation of the application based on at least one of Extensible HyperText Markup Language-Mobile Profile (XHTML-MP) and Wireless Markup Language (WML).

19. The method of claim 1, wherein the one or more user interaction elements represent conversational gestures.

20. The method of claim 1, wherein the declarative representation is based on Extensible Markup Language (XML).

21. The method of claim 1, wherein the transcoding is performed in accordance with XSL transformation (XSLT) rules.

22. The method of claim 1, wherein the representation supports presentation of one or more snippets of the application to the user, the one or more snippets comprising at least a prompt and an input field, wherein the prompt is presented to the user and the input field specifies user input receivable in response to the prompt.

23. The method of claim 22, wherein the one or more snippets further comprise a submit instruction.

24. The method of claim 22, wherein the one or more snippets are one or more Voice Extensible Markup Language (VoiceXML) snippets.

25. The method of claim 23, wherein the one or more VoiceXML snippets circumvent dialog management imposed by the VoiceXML form interpretation algorithm.

26. The method of claim 1, wherein the representation permits modeling of the application state along with associated type and cross-field constraints.

27. The method of claim 1, wherein the representation permits continuous maintaining and updating of the application state.

28. The method of claim 1, wherein the representation permits passing a user utterance to a natural language understanding (NLU) engine to extract semantic meaning.

29. The method of claim 28, wherein the representation permits using contextual knowledge present in the application state along with the semantic meaning returned by the NLU engine to permit contextual interpretation of the user utterance.

30. The method of claim 29, wherein the representation permits using the contextual interpretation to decide which portion of a dialog gets traversed next and updating of the application state.

31. The method of claim 1, wherein the representation permits defined events and attaching appropriate event handlers to the events.

32. The method of claim 31, wherein the representation specification of navigation inside the application thereby causing a focus shift based on one of the application state and a result of a generated event.

33. The method of claim 31, wherein the defined events comprise at least one of a parser event, a request for at least one of canonicalization and disambiguation, results from at least one of canonicalization and disambiguation, an update event and submission of context.

34. The method of claim 1, wherein the representation permits reuse of dialogs to allow the user to provide information about a new request inherited from a similar previous request.

35. The method of claim 1, further wherein the one or more user interaction elements are independent of content and presentation associated with the application.

36. The method of claim 1, further wherein the representation permits extraction of attribute value pairs from user input.

37. A dialog system for providing access to an application by a user, the dialog system comprising:

one or more processors operable to;

(i) obtain the application from an application server, the application representing interactions that the user may have with the dialog system as a data model and one or more user interaction elements that populate an application state of the data model and that are bound thereto, such that a dialog that the user has with the dialog system may be a mixed-initiative dialog; and

(ii) one of transforming and associating at least a portion of the one or more user interaction elements to one or more modality-specific renderings which are one of generated and selected by a dialog manager algorithm; and

memory, operatively coupled to the one or more processors, for storing at least a portion of results associated with at least one of the obtaining, transforming and associating operations.

38. A conversational browser apparatus for providing access to an application by a user, the apparatus comprising:

an interaction manager;

at least one modality based browser operatively coupled to the interaction manager, the modality based browser being a voice-based browser; and

a dialog manager operatively coupled to the interaction manager;

wherein interaction between the user, the interaction manager, the voice-based browser and the the dialog manager comprises;

(i) the interaction manager obtains the application from a server in response to a user request, and passes a representation of the application to the dialog manger;

(ii) the dialog manager determines at least one of a focus of the application and an intent of the user in accordance with the representation of the application, and informs the interaction manager of at least one of the focus of the application and the intent of the user; and

(iii) the interaction manager then designates a particular portion of the application for presentation to the user in accordance with the voice-based browser based on the determination of at least one of the focus of the application and the intent of the user, the portion comprising at least a prompt and an input field, wherein the prompt is presented to the user and the input field specifies user input receivable in response to the prompt for use in redetermining at least one of the focus of the application and the intent of the user.

39. The apparatus of claim 38, wherein the interaction manager parses the application into a data model and one or more user interaction elements bound to the data model, generates a document object model (DOM) representation of at least the data model, and passes the DOM representation of the data model to the dialog manager.

40. The apparatus of claim 38, wherein the conversational browsing mechanism further comprises a natural language attribute value (AV) pair extractor operatively coupled between the voice-based browser and the interaction manager for extracting AV pairs associated with one or more user utterances for use by the dialog manager in determining at least one of the focus of the application and the intent of the user.

41. The apparatus of claim 38, wherein the designated portion further comprises a submit instruction to submit a result of the user input to the interaction manager.

42. The apparatus of claim 38, wherein the designated portion specifies a type of processing to be performed in accordance with at least one of the prompt and the user input.

43. The apparatus of claim 42, wherein the processing performed comprises at least one of speech recognition, speech synthesis, natural language parsing and attribute value pair extraction.

44. The apparatus of claim 42, wherein the voice-based browser serves as an interface to the processing to be performed in accordance with at least one of the prompt and the user input.

45. The apparatus of claim 44, wherein the dialog manager abstractly operates separate from the processing interface provided by the voice-based browser.

46. The apparatus of claim 38, wherein the dialog manager operates independent of modalities.

47. The apparatus of claim 38, wherein the application server attempts to disambiguate an original user input provided thereto and, upon failing to do so, generates a sub-dialog to attempt to clarify the input such that the application server can reattempt to disambiguate the original user input based on additional input received from the user in response to the sub-dialog.

48. The apparatus of claim 47, wherein the disambiguated user input is provided to the dialog manager.

49. The apparatus of claim 38, wherein the designated portion is a Voice Extensible Markup Language (VoiceXML) snippet.

50. The apparatus of claim 38, wherein a menu is presented in an Extensible HyperText Markup Language (XHTML) frame or as part of a page of the application.

51. The apparatus of claim 38, wherein the voice-based browser supports Voice Extensible Markup Language (VoiceXML).

52. The apparatus of claim 51, wherein VoiceXML is extended to support natural language understanding of a user input and generation of resulting attribute value pairs.

53. The apparatus of claim 52, wherein the extension of VoiceXML comprises support for a language model.

54. The apparatus of claim 51, wherein VoiceXML is extended to specify natural language (NL) parsing via introduction of a NL grammar as a speech data file.

55. The apparatus of claim 51, wherein the extension of VoiceXML comprises support for a natural language grammar tag.

56. The apparatus of claim 38, wherein the conversational browsing mechanism further comprises a graphical user interface (GUI) based browser operatively coupled to the interaction manager for providing the user with the ability to interact with the application via at least a second modality.

57. The apparatus of claim 38, wherein communication between the interaction manager, the dialog manager and the application server is event-based.

58. The apparatus of claim 57, wherein an event comprises a focus event wherein the dialog manager informs the interaction manager of a next step in the interaction.

59. The apparatus of claim 57, wherein an event comprises a slot mutation event wherein the dialog manager informs the interaction manager of a change associated with a data model.

60. The apparatus of claim 57, wherein an event comprises a canonicalization event where the dialog manager requests canonicalization to be performed by the application server.

61. The apparatus of claim 57, wherein an event comprises a disambiguation event where the dialog manager requests disambiguation to be performed by the application server.

62. The apparatus of claim 57, wherein an event comprises a backend submit event where the dialog manager indicates to the interaction manager that a particular instance be sent to business logic associated with the application.

63. The apparatus of claim 57, wherein an event comprises a parser event that contains attribute value pairs derived from user input.

64. The apparatus of claim 57, wherein an event comprises a canonicalizer response event that contains results of canonicalization of user input.

65. The apparatus of claim 57, wherein an event comprises a backend submit response event that indicates whether or not a backend request has successfully completed.

66. The apparatus of claim 57, wherein an event comprises a graphical user interface (GUI) focus event that indicates whether or not focus has been set explicitly by the user via a GUI browser.

67. The apparatus of claim 57, wherein an event comprises a slot mutation event that contains one or more updated data model values that the interaction manager has received from a graphical user interface (GUI) browser.

68. The apparatus of claim 38, wherein a prompt is generated via a natural language generation (NLG) technique.

69. The apparatus of claim 38, wherein the application comprises forms and the dialog manager scores attribute value pairs generated from user input against each form and determines a top scoring form.

70. The apparatus of claim 69, wherein a scoring technique used by the dialog manager is based on a default generic scoring algorithm.

71. The apparatus of claim 69, wherein a scoring technique used by the dialog manager is based on a scoring algorithm pointed to by a form.

72. The apparatus of claim 71, wherein the form points to Extensible Markup Language (XML) arguments for the scoring algorithm.

73. The apparatus of claim 69, wherein a form specifies an algorithm to be used to create or reuse one or more forms and a data model.

74. A conversational browser apparatus for providing access to an application by a user, the apparatus comprising:

an interaction manager function;

at least one modality based browser function operatively coupled to the interaction manager function; and

a dialog manager function operatively coupled to the interaction manager function;

wherein interaction between the user, the interaction manager, the browser and the the dialog manager functions comprises;

(i) the interaction manager obtains the application from a server in response to a user request, the application representing interactions that the user may have with the application as a data model and one or more user interaction elements that populate an application state of the data model and that are bound thereto;

(ii) the dialog manager receives attribute value pairs generated from user input and processes the attribute value pairs to populate the data model and determine a next interaction focus; and

(iii) upon determination of the next interaction focus by the dialog manager, the application state and context are updated and an output is generated by the interaction manager and provided to the user for interaction via the browser.

75. The apparatus of claim 74, wherein determination of the next interaction focus uses at least one of interaction histories and contexts.

76. The apparatus of claim 75, wherein the at least one of interaction histories and contexts comprise time ordered sets of input and output attribute value pairs and the resulting data model changes.

77. The apparatus of claim 76, wherein the attribute value pairs are from different devices or modalities.

78. The apparatus of claim 74, wherein the dialog manager defers data model updates to disambiguate intent by seeking more at least one of details, confirmations and correction from at least one of the user and a knowledge source.

79. The apparatus of claim 78, wherein the dialog manager employs at least one of sub-dialogs and jumps to other dialogs within the application.

80. The apparatus of claim 78 wherein the knowledge source comprises at least one of a backend database and business logic.

81. The apparatus of claim 74, wherein the application is declaratively authored.

82. The apparatus of claim 74, wherein the output generating operation comprises transcoding.

83. The apparatus of claim 74, wherein the output generating operation comprises selecting a precompiled presentation snippet.

84. The apparatus of claim 74, wherein the output generating operation comprises use of a manually authored presentation.

85. The apparatus of claim 74, wherein the browser is a voice-based browser.

86. The apparatus of claim 74, wherein the browser is a HyperText Markup Language (HTTP) based browser.

87. The apparatus of claim 74, wherein communication between the interaction manager, the dialog manager and the application server is event-based.

88. The apparatus of claim 74, wherein the interaction manager generates a document object model (DOM) representation of at least the data model.

89. The apparatus of claim 74, wherein the browser supports Voice Extensible Markup Language (VoiceXML).

90. The apparatus of claim 89, wherein VoiceXML is extended to allow specification of a dialog management algorithm.

91. The apparatus of claim 90, wherein the dialog management algorithm performs at least one of scoring, focus determination and disambiguation.

92. The apparatus of claim 90, wherein the dialog management algorithm overwrites the form interpretation algorithm in VoiceXML.

93. The apparatus of claim 74, wherein the attribute value pairs are annotated.

94. The apparatus of claim 74, wherein the functions support a protocol for annotating the attribute value pairs.

95. A method of authoring an application accessible by a user in accordance with a conversational browser, the method comprising the steps of:

an interaction manager of the conversational browser obtaining the application from a server in response to a user request, the application representing interactions that the user may have with the application as a data model and one or more user interaction elements that populate an application state of the data model and that are bound thereto;

a dialog manager of the conversational browser receiving attribute value pairs generated from user input and processing the attribute value pairs to populate the data model and determine a next interaction focus; and

upon determination of the next interaction focus by the dialog manager, the application state and context being updated and an output being generated by the interaction manager and provided to the user for interaction via a browsing mechanism of the conversational browser.

96. An article of manufacture comprising a machine readable medium containing one or more programs which when executed implement the steps of:

representing interactions that the user may have with a dialog system as a data model and one or more user interaction elements that populate an application state of the data model and that are bound thereto, such that a dialog that the user has with the dialog system may be a mixed-initiative dialog;

wherein at least a portion of the one or more user interaction elements can be transformed or associated to one or more modality-specific renderings associated with the representation which are presentable to the user and are one of selected and generated by a dialog manager algorithm.