Using human wizards in a conversational understanding system

• US 9,830,039 B2
• Filed: 03/04/2013
• Issued: 11/28/2017
• Est. Priority Date: 03/04/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Stifelman, Lisa, Hakkani-Tur, Dilek, Heck, Larry, Chinthakunta, Madhusudan
• Primary Examiner(s): TRAPANESE, WILLIAM C
• Application Number:
  US13/784,449
• Publication Number:
  US 20140250378A1
• Time in Patent Office: 1,730 Days
• Field of Search: None
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

A wizard control panel may be used by a human wizard to adjust the operation of a Natural Language (NL) conversational system during a real-time dialog flow. Input to the wizard control panel is detected and used to interrupt/change an automatic operation of one or more of the NL conversational system components used during the flow. For example, the wizard control panel may be used to adjust results determined by an Automated Speech Recognition (ASR) component, a Natural Language Understanding (NLU) component, a Dialog Manager (DM) component, and a Natural Language Generation (NLG) before the results are used to perform an automatic operation within the flow. A timeout may also be set such that when the timeout expires, the conversational system performs an automated operation by using the results shown in the wizard control panel (edited/not edited).

Claims

1. A method for using a wizard control panel in a conversational system, comprising:

receiving a user interaction with the conversational system during a dialog flow;

receiving information associated with graphical entities of a user'"'"'s display;

displaying the wizard control panel including a display of elements that are used for affecting an automatic operation of the conversational system during the dialog flow and a display of a representative state of the user'"'"'s display for interacting with the user'"'"'s display by a human wizard;

processing the user interaction to display results for the user interaction within the wizard control panel;

determining receipt of an update to the results in the wizard control panel; and

in response to determining the results are updated in the wizard control panel, submitting updated results to the conversational system, wherein submission of the updated results modifies operation of the conversational system before the automatic operation is processed.

2. The method of claim 1, wherein displaying the wizard control panel, comprises:

displaying text that is automatically recognized by an input recognition component of the conversational system within an element of the wizard that is changeable.

3. The method of claim 1, wherein displaying the wizard control panel, comprises:

displaying a Natural Language Understanding element of the wizard that is configured for;

viewing an intent and changing the intent;

displaying a slot and a value and changing the slot and the value;

wherein the Natural Language Understanding element is populated using results determined by an input understanding component of the conversational system.

4. The method of claim 1, wherein displaying the wizard control panel, comprises:

displaying a Natural Language Generation element of the wizard that is configured for viewing and changing feedback that is populated using results automatically determined by an input understanding component of the conversational system.

5. The method of claim 1, wherein displaying the wizard control panel, comprises:

displaying shortcut elements that when selected instruct the conversational system to perform a command.

6. The method of claim 1, wherein displaying the wizard control panel, comprises:

displaying a timer that indicates an amount of time left before the conversational system automatically performs actions using results currently shown in the wizard control panel.

7. The method of claim 1, wherein displaying the wizard control panel, comprises:

displaying a skip element that when selected instructs the conversational system to ignore the user interaction.

8. The method of claim 1, wherein displaying the wizard control panel, comprises:

displaying a repeat element that when selected instructs the conversational system to request a user to repeat the user interaction; and

an unknown element that when selected instructs the conversational system to ignore the user interaction as unknown.

9. The method of claim 1, wherein submission of the updated results further comprises transmitting a control action to manipulate an entity on the user'"'"'s display.

10. A computer-readable hardware storage device storing computer-executable instructions that, when executed on a processor, causes the processor to perform a method comprising:

receiving a user interaction with a conversational system during dialog flow processing;

receiving information associated with graphical entities of a user'"'"'s display;

displaying a wizard control panel including a display of elements that are used for affecting an automatic operation of the conversational system during the dialog flow processing and a display of a representative state of the user'"'"'s display for interacting with the user'"'"'s display by a human wizard;

processing the user interaction to display results for the user interaction within the wizard control panel;

displaying a timer that indicates an amount of time left of a predetermined time before the conversational system executes the automatic operation using results currently shown in the wizard control panel;

determining when an input is received into the wizard control panel within the predetermined time, wherein the input updates the results displayed in the wizard control panel; and

in response to determining that the input is received into the wizard control panel, submitting updated results based on the input, wherein the updated results modifies operation of the conversational system before the automatic operation is processed.

11. The computer-readable hardware storage device of claim 10, wherein displaying the wizard control panel, comprises:

displaying text that is automatically recognized by an input recognition component of the conversational system within an element of the wizard that is changeable.

12. The computer-readable hardware storage device of claim 10, wherein displaying the wizard control panel, comprises:

displaying a Natural Language Understanding element of the wizard that is configured for;

viewing an intent and changing the intent;

displaying a slot and a value and changing the slot and the value;

wherein the Natural Language Understanding element is populated using results determined by an input understanding component of the conversational system.

13. The computer-readable hardware storage device of claim 10, wherein displaying the wizard control panel, comprises:

displaying a Natural Language Generation element of the wizard that is configured for viewing and changing feedback that is populated using results automatically determined by an input understanding component of the conversational system.

14. The computer-readable hardware storage device of claim 10, wherein displaying the wizard control panel, comprises:

displaying shortcut elements that when selected instruct the conversational system to perform a command comprising one or more of;

a back command, a help command, a home command, a scroll up command, a scroll down command, and a search command.

15. The computer-readable hardware storage device of claim 10, wherein displaying the wizard control panel, comprises:

displaying one or more of;

a skip element that when selected instructs the conversational system to ignore the user interaction;

a repeat element that when selected instructs the conversational system to request a user to repeat the user interaction; and

an unknown element that when selected instructs the conversational system to ignore the user interaction as unknown.

16. The computer-readable storage device of claim 10, wherein submission of the updated results further comprises transmitting a control action to manipulate an entity on the user'"'"'s display.

17. A system for using a wizard control panel in a natural language (NL) conversational system, comprising:

at least one processor;

a display; and

a memory operatively connected with the at least one processor, the memory storing instructions, that when executed on the at least one processor, causes the at least one processor to perform operations comprising;

receiving a user interaction with the NL conversational system during processing of a dialog flow;

receiving graphical entities from a user'"'"'s display;

displaying the wizard control panel comprising a display of a representative state of the user'"'"'s display for interacting with the user'"'"'s display by a human wizard;

processing the user interaction to display results for the user interaction within the wizard control panel and display the graphical entities;

displaying a timer that indicates an amount of time left of a predetermined time before the NL conversational system performs an automatic operation using results currently shown in the wizard control panel;

determining when an input is received into the wizard control panel within the predetermined time, wherein the input updates the results displayed in the wizard control panel; and

in response to determining that the input is received into the wizard control panel, submitting updated results based on the input, wherein the updated results modifies operation of the NL conversational system before the automatic operation is processed.

18. The system of claim 17, wherein displaying the wizard control panel, comprises:

displaying text that is automatically recognized by an automatic speech recognition (ASR) component of the NL conversational system within at least one of the ASR elements of the wizard that is changeable.

19. The system of claim 17, wherein displaying the wizard control panel, comprises:

displaying feedback within a user interface for the NL conversational system that is populated using the updated results.

20. The system of claim 17, wherein submission of the updated results further comprises transmitting a control action to manipulate an entity on the user'"'"'s display.