DIALOG STATE TRACKING USING WEB-STYLE RANKING AND MULTIPLE LANGUAGE UNDERSTANDING ENGINES

• US 20150363393A1
• Filed: 06/12/2014
• Published: 12/17/2015
• Est. Priority Date: 06/12/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Williams, Jason D., Zweig, Geoffrey G.
• Granted Patent: US 10,108,608 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/0635 updating or merging of old and new templates; Mean values; Weighting

Abstract

A dialog state tracking system. One aspect of the system is the use of multiple utterance decoders and/or multiple spoken language understanding (SLU) engines generating competing results that improve the likelihood that the correct dialog state is available to the system and provide additional features for scoring dialog state hypotheses. An additional aspect is training a SLU engine and a dialog state scorer/ranker DSR engine using different subsets from a single annotated training data set. A further aspect is training multiple SLU/DSR engine pairs from inverted subsets of the annotated training data set. Another aspect is web-style dialog state ranking based on dialog state features using discriminative models with automatically generated feature conjunctions. Yet another aspect is using multiple parameter sets with each ranking engine and averaging the rankings. Each aspect independently improves dialog state tracking accuracy and may be combined in various combinations for greater improvement.

Claims

1. A method for improving dialog state tracking accuracy in a dialog system, the method comprising:

determining meaning representations for conversational inputs received from a user;

enumerating dialog state hypotheses from the meaning representations;

extracting the features associated with each dialog state hypothesis using spoken language processing;

ranking the dialog state hypotheses; and

using at least one member of the ranked set of dialog states hypotheses to determine what action the dialog system should take next.

2. The method of claim 1 characterized in that the act of ranking the dialog state hypotheses further comprises the acts of:

determining component scores for each dialog state hypothesis based on features associated with the dialog state hypothesis using a forest of decision trees;

computing a final score for each dialog state hypothesis from a weighted sum of the component scores; and

ranking the dialog state hypotheses based on the final score.

3. The method of claim 1 characterized in that the act of ranking the dialog state hypotheses further comprises the acts of:

enumerating a plurality of scores for each dialog state hypothesis using a plurality of dialog state ranking engines;

averaging the scores for each dialog hypothesis to produce a final score for each dialog hypothesis;

ranking the dialog state hypotheses based on the final score.

4. The method of claim 1 characterized in that the decision trees include cascading binary branches for leading to leaf nodes, each leaf node having a real value that is added to the score for a dialog state hypothesis when decisions of the binary branches based on features associated with the dialog state hypothesis lead to that leaf node.

5. The method of claim 4 characterized in that each binary branch applies a threshold to feature scoring features associated with the dialog state hypothesis.

6. The method of claim 1 further comprising the act of using a web-style ranking algorithm to automatically build ranking models having conjunctions.

7. The method of claim 6 characterized in that the web-style ranking algorithm is lambdaMart.

8. The method of claim 1 further comprising the acts of:

dividing a training data set into a plurality of parts, including a first part and a second part;

training a first spoken language understanding engine with the first part of the training data set; and

training a first ranking engine with the second part of the training data set using the first training engine and a first training parameter set.

9. The method of claim 8 further comprising the acts of:

training a second spoken language understanding engine with the second part of the training data set; and

training a second ranking engine with the first part of the training data set using the second training engine and a second training parameter set.

10. The method of claim 9 further comprising the acts of:

training a third ranking engine with the second part of the training data set using the first training engine and a third training parameter set, the third training parameter set being different than the first training parameter set; and

training a fourth ranking engine with the first part of the training data set using the second training engine and a fourth training parameter set, the fourth training parameter set being different than the second training parameter set.

11. The method of claim 1 characterized in that the features associated with each dialog state hypothesis include additional features derived from competitive outputs obtained by processing dialog state hypotheses using a plurality of spoken language processing engines.

12. The method of claim 1 characterized in that the conversational inputs are spoken utterances decoded using automatic speech recognition.

13. A dialog state tracking system comprising:

an input device operable to collect conversational inputs from a user;

an input decoder in communication with the input device, the input decoder operable to receive convert the conversational inputs into computer readable text;

a plurality of spoken language understanding processors in communication with the input decoder, the spoken language understanding processors operable to translate the computer readable text into a dialog state hypothesis set comprising at least one dialog state; and

a dialog manager operable to score each dialog state hypothesis based on features associated with the dialog state hypothesis and select the highest scoring dialog state hypothesis as the correct dialog state.

14. The dialog state tracking system of claim 13 characterized in that the dialog manager uses web-style ranking to score each dialog state hypothesis.

15. The dialog state tracking system of claim 13 characterized in that dialog manager includes a plurality of dialog state rankers operable to score each dialog state hypothesis using a forest of decision trees including automatically built conjunctions.

16. The dialog state tracking system of claim 15 characterized in that spoken language understanding processors are paired with dialog state rankers, each pair trained using a single training data set, the spoken language understanding processor and the dialog state ranker in each pair being trained with different portions of the training data set.

17. The dialog state tracking system of claim 16 comprising a first pair and a second pair, the spoken language understanding processor of the first pair and the dialog state ranker of the second pair trained with one portion of training data set, and the spoken language understanding processor of the second pair and the dialog state ranker of the first pair being trained with a different portion of training data set.

18. The dialog state tracking system of claim 13 characterized in that the conversational inputs are spoken utterances and the input decoder includes an automatic speech recognizer operable to decode the spoken utterance into computer readable text.

19. The dialog state tracking system of claim 13 characterized in that the input decoder comprises a plurality of automatic speech recognizers, each automatic speech recognizer in communication with one or more of the spoken language understanding processors.

20. A computer readable medium containing computer executable instructions which, when executed by a computer, perform a method of improving dialog state tracking accuracy in a human-computer interaction system, the method comprising:

determining meaning representations for conversational inputs received from a user;

enumerating dialog state hypotheses and extracting the features associated with each dialog state hypothesis using multiple spoken language understanding engines;

determining scores for each dialog state hypothesis based on dialog state hypothesis features using multiple dialog state ranking engines, each dialog state ranking engine having a forest of decision trees including automatically built conjunctions;

averaging the scores from each dialog state ranking engines to produce a final score for each dialog state hypothesis;

ranking the dialog state hypotheses based on the final scores; and

updating a dialog session with the highest ranking dialog state hypothesis.