ORPHANED UTTERANCE DETECTION SYSTEM AND METHOD

• US 20160055240A1
• Filed: 08/22/2014
• Published: 02/25/2016
• Est. Priority Date: 08/22/2014
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Deoras, Anoop, Tur, Gokhan, Hakkani-Tur, Dilek
• Application Number:
  US14/466,642
• Publication Number:
  US 20160055240A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/225 Feedback of the input speech

Abstract

An orphan detector. The orphan detector processes out-of-domain utterances from a targeted language understanding dialog system to determine whether the out-of-domain utterance expresses a specific intent to have the targeted language understanding dialog system to take a certain action where fallback processing, such as performing a generic web search, is unlikely to be satisfied by web searches. Such utterances are referred to as orphans because they are not appropriately handled by any of the task domains or fallback processing. The orphan detector distinguishes orphans from web search queries and other out-of-domain utterances by focusing primarily on the structure of the utterance rather than the content. Orphans detected by the orphan detector may be used both online and offline to improve user experiences with targeted language understanding dialog systems. The orphan detector may also be used to mine structurally similar queries or sentences from the web search engine query logs.

Claims

1. A method of classifying an out-of-domain utterance as a web search query or an orphan of a targeted language understanding dialog system, the method comprising:

receiving an out-of-domain utterance rejected by a domain classifier of the targeted language system;

evaluating the out-of-domain utterance for one or more semantic, syntactic, part-of-speech tag, or lexical features determinative of whether the out-of-domain utterance is a web search query or an orphan of the targeted language understanding dialog system;

determining that the out-of-domain utterance is an orphan;

generating a response indicating that the out-of-domain utterance is understood as a request to perform a machine action addressed to the targeted language understanding dialog system.

2. The method of claim 1 further comprising the acts of:

receiving an utterance supplied as an input to the targeted language understanding dialog system; and

determining that the utterance is an out-of-domain utterance not covered by a task domain associated with the targeted language understanding dialog system.

3. The method of claim 1 further comprising the acts of:

generating a message reporting that the out-of-domain utterance is understood as a request to perform a machine action addressed to the targeted language understanding dialog system; and

rendering the message to an output device associated with the targeted language understanding dialog system.

4. The method of claim 3 wherein the response does not comprise results from a web search query based on the orphan.

5. The method of claim 1 wherein the act of determining that the out-of-domain utterance is an orphan further comprises the act of classifying the out-of-domain utterance as an orphan using an orphan classifier based on features of the out-of-domain utterance supplied as inputs to the orphan classifier.

6. The method of claim 1 further comprising the act of extracting features from the utterance prior to determining that the out-of-domain utterance is an orphan.

7. The method of claim 1 further comprises the acts of:

checking the utterance for a predicate and an argument; and

using the existence of a predicate and an argument as an orphan classification feature.

8. The method of claim 1 further comprising the acts of:

semantically clustering a plurality of orphans; and

identifying a semantic cluster containing a selected measure of orphans that indicates users frequent make requests relating to an unsupported task domain.

9. The method of claim 8 further comprising the acts of:

building a new task domain classification model for the unsupported task domain using the selected semantic cluster of orphans as the positive class of training data; and

adding the new task domain classification model to the targeted language understanding dialog system.

10. The method of claim 1 wherein the act of determining that the utterance is an out-of-domain utterance not covered by a task domain associated with the targeted language understanding dialog system further comprises the acts of:

processing the utterance using one or more domain classifiers associated with the one or more task domains; and

classifying the out-of-domain utterance as an orphan when the utterance is not accepted by any of the one or more task domains.

11. The method of claim 1 further comprising the act of determining whether the out-of-domain utterance is a factoid question or chit-chat prior to determining that the out-of-domain utterance is an orphan.

12. The method of claim 1 wherein the orphan classifier applies a classification model built from a positive class of training data comprising a plurality of in-domain utterances used to train at least one task domain of the targeted understanding system and a negative class of training data comprising a plurality of web search queries.

13. A targeted language understanding dialog system having at least one domain, the targeted language understanding dialog system comprising:

a feature extractor operable to extract features from an utterance;

a classifier operable to receive the features as inputs, determine whether the utterance is an out-of-domain utterance not belonging to any domain associated with the classifier;

an orphan detector operable to receive the out-of-domain utterance, determine whether the out-of-domain utterance is an orphan; and

an interaction manager operable to process the orphans without sending the orphan to a general web search.

14. The targeted language understanding dialog system of claim 13 wherein the orphan detector comprises a semantic parser operable to semantically parse the out-of-domain utterance.

15. The targeted language understanding dialog system of claim 14 wherein the orphan detector comprises a discriminative classifier operable to apply a classification model using a feature associated with the out-of-domain utterance obtained using an output of the semantic parser to determine if the out-of-domain utterance is an orphan.

16. The targeted language understanding dialog system of claim 14 wherein the orphan detector further comprises at least one of a syntactic parser, a part-of-speech tagger, and a lexical parser.

17. The targeted language understanding dialog system of claim 13 wherein the orphan detector comprises a discriminative classifier operable to apply a semantic classification model.

18. The targeted language understanding dialog system of claim 13 further comprising:

an input device operable to collect utterances from a user; and

an input decoder in communication with the input device, the input decoder operable to receive convert the utterances that are not text into text processable by a machine; and

an output device for rendering a result indicating the orphan specifies an intent that does not have a corresponding dialog act known to the interaction manager.

19. The targeted language understanding dialog system of claim 13 wherein the targeted language understanding dialog system is a production system contemporaneously responding to utterances from users, the interaction manager further operable to generate an output notifying the user that the intent cannot be performed.

20. A computer readable medium containing computer executable instructions which, when executed by a computer, perform a method of detecting orphans from out-of-domain utterances in a targeted language understanding dialog system having one or more task domains, the method comprising:

receiving an utterance via an input device associated with the targeted language understanding dialog system;

processing the utterance using a set of a domain classifiers to determine if the utterance is belongs to a task domain supported by the targeted language understanding dialog system;

classifying the utterance as an out-of-domain utterance when the utterance does not belong to any task domain supported by the targeted language understanding dialog system;

a set of a domain classifiers to determine if the out-of-domain utterance belongs to a task domain supported by the targeted language understanding dialog system;

determining a semantic shape for the out-of-domain utterance;

classifying the out-of-domain utterance as an orphan when the semantic shape of the out-of-domain utterance corresponds to a reference semantic shape having a predicate and an object;

generating an output indicating that the orphan is understood as not being a web search query but is not supported by the targeted language understanding dialog system; and

rendering an output via an output device associated with the targeted language understanding dialog system.