Methods and systems for authoring of mixed-initiative multi-modal interactions and related browsing mechanisms

• US 8,572,209 B2
• Filed: 10/12/2007
• Issued: 10/29/2013
• Est. Priority Date: 05/28/2002
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Healey, Jennifer A., Hosn, Rafah A., Maes, Stephane Herman, Raman, Thiruvilvama Lai V., Tiwari, Alpana
• Primary Examiner(s): BLAIR, DOUGLAS B
• Application Number:
  US11/871,571
• Publication Number:
  US 20080034032A1
• Time in Patent Office: 2,209 Days
• Field of Search: 709/219, 709/246
• US Class Current: 709/219
• CPC Class Codes: G06F 8/00 Arrangements for software engineering testing or debugging G06F11/36; administrative, planning or organisation aspects of software project management G06Q10/06

Abstract

Application authoring techniques, and information browsing mechanisms associated therewith, which employ programming in association with mixed-initiative multi-modal interactions and natural language understanding for use in dialog systems. Also, a conversational browsing architecture is provided for use with these and other authoring techniques.

Claims

1. A conversational browser apparatus for providing access to an application by a user, the apparatus comprising:

a processor; and

a memory for storing program instructions that are executed by the processor for implementing;

an interaction manager;

at least one modality based browser operatively coupled to the interaction manager, the modality based browser being a voice-based browser; and

a dialog, manager operatively coupled to the interaction manager;

wherein interaction between the user, the interaction manager, the voice-based browser and the dialog manager comprises;

(i) the interaction manager obtains the application from a server in response to a user request and passes a representation of the application to the dialog manger;

(ii) the dialog manager determines a focus of the application and an intent of the user in accordance with the representation of the application, and informs the interaction manager of at least one of the focus of the application and the intent of the user;

(iii) the interaction manager then designates a particular portion of the application for presentation to the user in accordance with the voice-based browser based on the determination of at least one of the focus of the application and the intent of the user, the portion comprising at least a prompt and an input field, wherein the prompt is presented to the user and the input field specifies user input receivable in response to the prompt for use in predetermining at least one of the focus of the application and the intent of the user;

(iv) the interaction manager parses the application into a data model and one or more user interaction elements that are bound to the data model, and

(v) the dialog manager manages mixed-initiative dialog interaction between the user and the application by populating an application state of the data model through interaction with the user, wherein populating the application state comprises filling one or more slots of a form, in any order, in a single user interaction, and wherein the dialog manager uses the data model to keep track of a conversation context to determine which slots of the form have been filled with information from the user and which slots of the form need to be resented to the user to complete the form.

2. The apparatus of claim 1, wherein the interaction manager parses the application into a data model and one or more user interaction elements bound to the data model, generates a document object model (DOM) representation of at least the data model, and passes the DOM representation of the data model to the dialog manager.

3. The apparatus of claim 1, wherein the memory further stores program instructions that are executed by the processor for implementing a natural language attribute value (AV) pair extractor operatively coupled between the voice-based browser and the interaction manager for extracting AV pairs associated with one or more user utterances for use by the dialog manager in determining at least one of the focus of the application and the intent of the user.

4. The apparatus of claim 1, wherein the designated portion further comprises a submit instruction to submit a result of the user input to the interaction manager.

5. The apparatus of claim 1, wherein the designated portion specifies a type of processing, to be performed in accordance with at least one of the prompt and the user input.

6. The apparatus of claim 1, wherein the dialog manager operates independent of modalities.

7. The apparatus of claim 1, wherein the application server attempts to disambiguate an original user input provided thereto and, upon failing to do so, generates a sub-dialog to attempt to clarify the input such that the application server can reattempt to disambiguate the original user input based on additional input received from the user in response to the sub-dialog.

8. The apparatus of claim 1, wherein the designated portion is a Voice Extensible Markup Language (VoiceXML) snippet.

9. The apparatus of claim 1, wherein a menu is presented in an Extensible HyperText Markup Language (XHTML) frame or as part of a page of the application.

10. The apparatus of claim 1, wherein the voice-based browser supports Voice Extensible Markup Language (VoiceXML).

11. The apparatus of claim 1, wherein the memory further stores program instructions that are executed by the processor for implementing a graphical user interface (GUI) based browser operatively coupled to the interaction manager for providing the use with the ability to interact with the application via, at least a second modality.

12. The apparatus of claim 1, wherein communication between the interaction manager, the dialog manager and the application server is event-based.

13. The apparatus of claim 1, wherein a prompt is generated via a natural language generation (NLG) technique.

14. The apparatus of claim 1, wherein the application comprises forms and the dialog manager scores attribute value pairs generated from user input against each form and determines a top scoring form.

15. The apparatus of claim 1, wherein populating the application state by the dialog manager comprises:

scoring each of the one or more user interaction elements against one or more slots of each of a plurality of forms in accordance with a scoring algorithm;

selecting at least one of the plurality of forms to represent the application state based at least in pan on the scoring; and

populating the one or more slots of the selected form by the one or more user interaction elements in accordance with the user'"'"'s interaction with the dialog system.

16. A conversational browser apparatus for providing access to an application by a user, the apparatus comprising:

a processor; and

a memory for storing program instructions that are executed by the processor for implementing;

an interaction manager;

at least one modality based browser operatively coupled to the interaction manager; and

a dialog manager operatively coupled to the interaction manager;

wherein interaction between the user, the interaction manager, the browser and the dialog manager comprises;

(i) the interaction manager obtains the application from a server in response to a user request, the application representing interactions that the user may have with the application as a data model and one or more user interaction elements that populate an application state of the data model and that are bound thereto;

(ii) the dialog manager receives attribute value pairs generated from user input and processes the attribute value pairs to populate the data model and determine a next interaction focus, wherein the dialog manager manages mixed-initiative dialog interaction between the user and the application by populating the application state of the data model through interaction with the user, wherein populating the application state comprises using the attribute value pairs to fill one or more slots of a form, in any order, in a single user interaction, and wherein the dialog manager uses the data model to keep track of the conversation context to determine which slots of the form have been filled with information from the user and which slots of the form need to be presented to the user to complete the form; and

(iii) upon determination of the next interaction focus by the dialog manager, the application state and context are updated and an output is generated by the interaction manager and provided to the user for interaction via the browser.

17. The apparatus of claim 16, wherein populating the application state by the dialog manager comprises:

scoring each attribute value pair against each of a plurality of forms in accordance with a scoring algorithm;

selecting at least one of the plurality of forms to represent the application state based at least in part on the scoring; and

populating the one or more slots of the selected form with values from the attribute value pairs in accordance with the user'"'"'s interaction with the dialog system.

18. A method of authoring an application accessible by a user in accordance with a conversational browser, the method comprising the steps of:

an interaction manager of the conversational browser obtaining the application from a server in response to a user request, the application representing interactions that the user may have with the application as a data model and one or more user interaction elements that populate an application state of the data model and that are bound thereto;

a dialog manager of the conversational browser managing mixed-initiative dialog interaction between the user and the application, wherein the dialog manager manages mixed-initiative dialog by receiving attribute value pairs generated from user input and processing the attribute value pairs to populate an application state of the data model and determine a next interaction focus, wherein populating the application state comprises using the attribute value pairs to fill one or more slots of a form, in any order, in a single user interaction, and wherein the dialog manager uses the data model to keep track of the conversation context to determine which slots of the form have been filled with information from the user and which slots of the form need to be presented to the user to complete the form; and

upon determination of the next interaction focus by the dialog manager, the application state and context being updated and an output being generated by the interaction manager and provided to the user for interaction via a browsing mechanism of the conversational browser.

19. The method of claim 18, wherein populating the application state by the dialog manager comprises:

scoring each attribute value pair against each of a plurality of forms in accordance with a scoring algorithm;

selecting at least one of the plurality of forms to represent the application state based at least in part on the scoring; and

populating the one or more slots of the selected form with values from the attribute value pairs in accordance with the user'"'"'s interaction with the dialog system.