Data collection for a new conversational dialogue system

• US 10,824,798 B2
• Filed: 11/06/2017
• Issued: 11/03/2020
• Est. Priority Date: 11/04/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Semantic Machines, Inc. (Microsoft Corporation)
• Inventors: Liang, Percy Shuo, Klein, Daniel, Gillick, Laurence Steven, Cohen, Jordan Rian, Arsenault, Linda Kathleen, Clausman, Joshua James, Pauls, Adam David, Hall, David Leo Wright
• Primary Examiner(s): Lerner, Martin
• Application Number:
  US15/804,093
• Publication Number:
  US 20180203833A1
• Time in Patent Office: 1,093 Days
• Field of Search: 704 9, 704243, 704257, 704275
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 7/08 Sorting, i.e. grouping record carriers in numerical or other ordered sequence according to the classification of at least some of the information they carry by merging two or more sets of carriers in ordered sequence G06F7/16
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06Q 10/02 Reservations, e.g. for tickets, services or events
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: G06Q 30/06 Buying, selling or leasing transactions
• CPC Class Codes: G10L 15/06 Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice G10L15/14 takes precedence
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/0638 Interactive procedures

Abstract

A data collection system is based on a general set of dialogue acts which are derived from a database schema. Crowd workers perform two types of tasks: (i) identification of sensical dialogue paths and (ii) performing context-dependent paraphrasing of these dialogue paths into real dialogues. The end output of the system is a set of training examples of real dialogues which have been annotated with their logical forms. This data can be used to train all three components of the dialogue system: (i) the semantic parser for understanding context-dependent utterances, (ii) the dialogue policy for generating new dialogue acts given the current state, and (iii) the generation system for both deciding what to say and how to render it in natural language.

Claims

1. A method for training an annotated dialogue system, comprising:

generating, by a data generation application executing on a machine, a first list of canonical utterances in a dialogue tree of possible dialogues, the first list of canonical utterances generated in response to an input received from an annotating user;

receiving from the annotating user a first selection of a first canonical utterance from the first list, the selection indicating a first step in a multi-step dialogue;

generating a second list of canonical utterances, the second list of canonical utterances generated in response to the first selection from the first list;

receiving from the user a second selection of a second canonical utterance from the second list, the second selection indicating a second step in the multi-step dialogue that also includes the first step;

presenting a user interface indicating a dialogue path including both the first step and the second step;

receiving from the annotating user via the user interface a compound paraphrase for the dialogue path including both the first step and the second step; and

training the annotated dialogue system on annotated data including the compound paraphrase, the first step, and the second step.

2. The method of claim 1, wherein the compound paraphrase represents paraphrasing for the whole dialogue path, wherein the first selection from the first list and the second selection from the second list are two canonical utterances of a plurality of three or more selected canonical utterances in the dialogue path.

3. The method of claim 1, further comprising receiving, through the user interface, a ranking from the annotating user for one or more canonical utterances of the first list and the second list.

4. The method of claim 1, wherein further comprising storing the annotated training data in a format suitable for training the annotated dialogue system.

5. The method of claim 1, wherein generating a first list of a plurality of canonical utterances includes:

querying a database of possible actions,

generating a logical form at least in part from each action; and

generating a canonical utterance from each logical form.

6. The method of claim 5, wherein a logical form generated from an action includes computer code associated with the action, and wherein a canonical utterance generated from the logical form is a pseudocode representation of the computer code associated with the action.

7. The method of claim 1, wherein the first list and second list are each ranked according to a ranking model.

8. The method of claim 7, wherein the ranking model is based at least in part from user rankings of previous canonical utterances.

9. The method of claim 1, wherein the first list and the second list are provided via the user interface.

10. The method of claim 9, wherein the user interface includes state information for the current dialogue.

11. The method of claim 9, wherein the user interface includes a search box.

12. The method of claim 9, further comprising, in response to receiving the first input from the list, removing the one or more canonical utterances of the list that were not selected by the annotating user from the user interface.

13. The method of claim 9, further comprising, in response to receive the first input, adding the selected canonical utterance to a list of selected canonical utterances.

14. A system for training an annotated dialogue system, comprising:

a processor;

memory;

one or more modules stored in memory and executable by the processor to;

generate, by a data generation application, a first list of canonical utterances in a dialogue tree of possible dialogues, the first list of canonical utterances generated in response to an input received from an annotating user,

receive from the annotating user a first selection of a first canonical utterance from the first list, the first selection indicating a first step in a multi-step dialogue,

generate a second list of canonical utterances, the second list of canonical utterances generated in response to the first selection from the first list,

receive from the annotating user a second selection of a second canonical utterance from the second list, the second selection indicating a second step in the multi-step dialogue that also includes the first step,

present a user interface indicating a dialogue path including both the first step and the second step,

receive from the annotating user via the user interface a compound paraphrase for the dialogue path including both the first step and the second step, and

train the annotated dialogue system on annotated data including the compound paraphrase, the first step, and the second step.

15. The system of claim 14, wherein the one or more modules stored in memory are further executable by the processor to receive, through the user interface, a ranking from the annotating user for one or more canonical utterances of the first list and the second list.

16. The system of claim 14, further comprising storing the annotated data in a format suitable for training a dialogue system.

17. The system of claim 14, wherein the first list and second list are each ranked according to a ranking model.

18. The system of claim 17, wherein the ranking model is based at least in part on user rankings of previous canonical utterances.

19. The system of claim 18, wherein the first list and second list are provided within the interface.

20. A computer system including a processor and memory holding instructions executable by the processor to perform a method for training an annotated dialogue system, the method comprising:

generating a first list of canonical utterances in a dialogue tree of possible dialogues, the first list of canonical utterances generated in response to an input received from an annotating user;

receiving from the annotating user a first selection of a first canonical utterance from the first list, the first selection indicating a first step in a multi-step dialogue;

generating a second list of canonical utterances, the second list of canonical utterances generated in response to the first selection from the first list;

receiving from the annotating user a second selection of a second canonical utterance from the second list, the second selection indicating a second step in the multi-step dialogue that also includes the first step;

presenting a user interface indicating a dialogue path including both the first step and the second step and an input box configured for receiving a compound paraphrase for the dialogue path including both the first step and the second step;

receiving from the annotating user via the input box of the user interface a compound paraphrase for the dialogue path including both the first step and the second step; and

training the annotated dialogue system on annotated data including the compound paraphrase, the first step, and the second step.