USING HUMAN WIZARDS IN A CONVERSATIONAL UNDERSTANDING SYSTEM

• US 20140250378A1
• Filed: 03/04/2013
• Published: 09/04/2014
• Est. Priority Date: 03/04/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Stifelman, Lisa, Hakkani-Tur, Dilek, Heck, Larry, Chinthakunta, Madhusudan
• Granted Patent: US 9,830,039 B2
• Time in Patent Office: Days
• US Class Current: 715/708
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

A wizard control panel may be used by a human wizard to adjust the operation of a Natural Language (NL) conversational system during a real-time dialog flow. Input to the wizard control panel is detected and used to interrupt/change an automatic operation of one or more of the NL conversational system components used during the flow. For example, the wizard control panel may be used to adjust results determined by an Automated Speech Recognition (ASR) component, a Natural Language Understanding (NLU) component, a Dialog Manager (DM) component, and a Natural Language Generation (NLG) before the results are used to perform an automatic operation within the flow. A timeout may also be set such that when the timeout expires, the conversational system performs an automated operation by using the results shown in the wizard control panel (edited/not edited).

Claims

10. A computer-readable medium storing computer-executable instructions for using a wizard control panel in a natural language (NL) conversational system, comprising:

receiving a user interaction with the NL conversational system that includes the use of an Automated Speech Recognition (ASR) component, a Natural Language Understanding (NLU) component, a Dialog Manager (DM) component, and a Natural Language Generation (NLG) component during a dialog flow;

displaying the wizard control panel including a display of elements that are used for affecting an automatic operation of at least one of the different components of the NL conversational system during the dialog flow;

displaying a timer that indicates an amount of time left of a predetermined time before the conversational system automatically performs actions using results currently shown in the wizard control panel;

determining when an input is received within the predetermined time that is associated with one of the elements of the wizard control panel

determining a current component of the dialog flow; and

submitting to the current component updated results based on the input that modify results automatically determined by the current component of the NL conversational system.

11. The computer-readable medium of claim 10, wherein displaying the wizard control panel, comprises:

displaying text that is automatically recognized by an automatic speech recognition (ASR) component of the conversational system within an ASR element of the wizard that is changeable.

12. The computer-readable medium of claim 10, wherein displaying the wizard control panel, comprises:

displaying a Natural Language Understanding element of the wizard that is configured for;

viewing an intent and changing the intent;

displaying a slot and a value and changing the slot and the value;

wherein the Natural Language Understanding element is populated using results determined by the NLU component.

13. The computer-readable medium of claim 10, wherein displaying the wizard control panel, comprises:

displaying an Natural Language Generation element of the wizard that is configured for viewing and changing feedback that is populated using results automatically determined by the NLG component.

14. The computer-readable medium of claim 10, wherein displaying the wizard control panel, comprises:

displaying shortcut elements that when selected instruct the conversational system to perform a command comprising one or more of;

a back command, a help command, a home command, a scroll up command, a scroll down command, and a search command.

15. The computer-readable medium of claim 10, wherein displaying the wizard control panel, comprises:

displaying one or more of;

a skip element that when selected instructs the NL conversational system to ignore the current user interaction;

a repeat element that when selected instructs the NL conversational system to request a user to repeat the user interaction; and

an unknown element that when selected instructs the NL conversational system to ignore the user interaction as unknown.

16. The computer-readable medium of claim 10, wherein displaying the wizard control panel, comprises:

displaying a GO element that when selected uses results currently shown in the wizard control panel to perform the operation.

17. A system for using a wizard control panel in a natural language (NL) conversational system, comprising:

a processor and memory;

an operating environment executing using the processor;

a display; and

a wizard manager that is configured to perform actions comprising;

receiving a user interaction with the NL conversational system that includes the use of an Automated Speech Recognition (ASR) component, a Natural Language Understanding (NLU) component, a Dialog Manager (DM) component, and a Natural Language Generation (NLG) component during a dialog flow;

displaying the wizard control panel including a display of elements including ASR elements including a skip element that when selected skips the utterance and a repeat element that when selected requests a repeat of the utterance, NLU elements, NLG elements and shortcut elements that are used for affecting an operation of at least one of the different components of the NL conversational system during the dialog flow;

displaying a timer that indicates an amount of time left of a predetermined time before the conversational system automatically performs actions using results currently shown in the wizard control panel;

determining when an input is received within the predetermined time that is associated with one of the elements of the wizard control panel

determining a current component of the dialog flow; and

submitting to the current component updated results based on the input that modify results automatically determined by the current component of the NL conversational system.

1. A method for using a wizard control panel in a natural language (NL) conversational system, comprising:

receiving a user interaction with the NL conversational system that includes the use of an Automated Speech Recognition (ASR) component, a Natural Language Understanding (NLU) component, a Dialog Manager (DM) component, and a Natural Language Generation (NLG) component during a dialog flow;

displaying the wizard control panel including a display of elements that are used for affecting an automatic operation of at least one of the different components of the NL conversational system during the dialog flow;

determining when an input is received that is associated with one of the elements of the wizard control panel;

determining a current component of the dialog flow; and

submitting to the current component updated results based on the input that modify results automatically determined by the current component of the NL conversational system.

2. The method of claim 1, wherein displaying the wizard control panel, comprises:

displaying text that is automatically recognized by an automatic speech recognition (ASR) component of the conversational system within an ASR element of the wizard that is changeable.

3. The method of claim 1, wherein displaying the wizard control panel, comprises:

displaying a Natural Language Understanding element of the wizard that is configured for;

viewing an intent and changing the intent;

displaying a slot and a value and changing the slot and the value;

wherein the Natural Language Understanding element is populated using results determined by the NLU component.

4. The method of claim 1, wherein displaying the wizard control panel, comprises:

displaying an Natural Language Generation element of the wizard that is configured for viewing and changing feedback that is populated using results automatically determined by the NLG component.

5. The method of claim 1, wherein displaying the wizard control panel, comprises:

displaying shortcut elements that when selected instruct the conversational system to perform a command.

6. The method of claim 1, wherein displaying the wizard control panel, comprises:

displaying a timer that indicates an amount of time left before the conversational system automatically performs actions using results currently shown in the wizard control panel.

7. The method of claim 1, further comprising starting a timer in response to receiving the user interaction and using the timer to determine when input is receivable using the wizard control panel for the user interaction.

8. The method of claim 1, wherein displaying the wizard control panel, comprises:

displaying a skip element that when selected instructs the NL conversational system to ignore the current user interaction.

9. The method of claim 1, wherein displaying the wizard control panel, comprises:

displaying a repeat element that when selected instructs the NL conversational system to request a user to repeat the user interaction; and

an unknown element that when selected instructs the NL conversational system to ignore the user interaction as unknown.

18. The system of claim 17, wherein displaying the wizard control panel, comprises:

displaying text that is automatically recognized by an automatic speech recognition (ASR) component of the conversational system within at least one of the ASR elements of the wizard that is changeable.

20. The system of claim 17, wherein displaying the wizard control panel, comprises:

displaying feedback within at least one of the NLG elements that is populated using results automatically determined by the NLG component.

18-1. The system of claim 17, wherein displaying the wizard control panel, comprises:

displaying an intent, a slot and a value within the NLU elements.