Discriminating between natural language and keyword language items

• US 9,558,176 B2
• Filed: 01/14/2014
• Issued: 01/31/2017
• Est. Priority Date: 12/06/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Tur, Gokhan, Celikyilmaz, Fethiye Asli, Hakkani-Tr, Dilek, Heck, Larry P.
• Primary Examiner(s): Tieu, Benny Q
• Assistant Examiner(s): Chacko, Sunil
• Application Number:
  US14/155,097
• Publication Number:
  US 20150161107A1
• Time in Patent Office: 1,113 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 40/20 Natural language analysis semantic analysis of natural language G06F40/30
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

This disclosure pertains to a classification model, and to functionality for producing and applying the classification model. The classification model is configured to discriminate whether an input linguistic item (such as a query) corresponding to either a natural language (NL) linguistic item or a keyword language (KL) linguistic item. An NL linguistic item expresses an intent using a natural language, while a KL linguistic item expresses the intent using one or more keywords. In a training phase, the functionality produces the classification model based on query click log data or the like. In an application phase, the functionality may, among other uses, use the classification model to filter a subset of NL linguistic items from a larger set of items, and then use the subset of NL linguistic items to train a natural language interpretation model, such as a spoken language understanding model.

Claims

1. A computer system comprising:

one or more processing devices; and

one or more storage resources storing instructions which, when executed by the one or more processing devices, cause the one or more processing devices to;

receive an input linguistic item;

use a classification model to produce a label which identifies the input linguistic item as;

a natural language linguistic item, which expresses an intent using a natural language, or

a keyword language linguistic item, which expresses the intent using one or more keywords; and

perform an action on the input linguistic item based at least on the label,

the classification model being trained using training sets, individual training sets comprising an individual natural language linguistic item and an individual keyword language linguistic item that is identified as being related to the individual natural language linguistic item based at least on user behavior as reflected in a query click log.

2. The computer system of claim 1, wherein the instructions, when executed by the one or more processing devices, cause the one or more processing devices to:

receive a set of input linguistic items;

use the classification model to classify each of the set of input linguistic items as either a natural language linguistic item or a keyword language linguistic item, yielding, overall, a subset of natural language linguistic items and a subset of keyword language linguistic items; and

use a machine learning technique to generate a natural language interpretation model, based at least on the subset of natural language linguistic items.

3. The computer system of claim 2, wherein the natural language interpretation model is a spoken language understanding model.

4. The computer system of claim 1, wherein the instructions, when executed by the one or more processing devices, cause the one or more processing devices to:

produce a collection of linguistic items that match a filtering condition specified in a filter request; and

classify each of the collection of linguistic items as either a natural language linguistic item or a keyword language linguistic item, yielding, overall, a subset of natural language linguistic items and a subset of keyword language linguistic items.

5. The computer system of claim 4, wherein the instructions, when executed by the one or more processing devices, cause the one or more processing devices to:

use a knowledge resource to generate the collection of linguistic items, the knowledge resource specifying a plurality of entities and relations among the entities.

6. The computer system of claim 1, wherein the instructions, when executed by the one or more processing devices, cause the one or more processing devices to:

receive a set of input linguistic items;

classify each of the set of input linguistic items as either a natural language linguistic item or a keyword language linguistic item, yielding, overall, a subset of natural language linguistic items and a subset of keyword language linguistic items; and

filter the subset of natural language linguistic items and/or the subset of keyword language linguistic items based at least on a filtering condition specified in a filter request.

7. The computer system of claim 1, wherein the instructions, when executed by the one or more processing devices, cause the one or more processing devices to:

perform a first action when the input linguistic item is a natural language linguistic item, and perform a second action when the input linguistic item is a keyword language linguistic item.

8. The computer system of claim 7,

wherein the first action comprises processing the input linguistic item with a first processing resource, and

wherein the second action comprises processing the input linguistic item with a second processing resource.

9. The computer system of claim 7,

wherein the first action comprises routing the input linguistic item to a first destination, and

wherein the second action comprises routing the input linguistic item to a second destination.

10. The computer system of claim 1, wherein the training sets comprise training pairs.

11. A method performed by one or more processing devices, the method comprising:

identifying training sets comprising natural language linguistic items and related keyword language linguistic items, the training sets being identified based at least on similarity of user behavior by users that submit the natural language linguistic items and the related keyword language linguistic items as queries to a search engine;

training a classification model to distinguish between natural language input and keyword language input, the classification model being trained using the training sets;

receiving an input linguistic item; and

using the trained classification model to produce a label which identifies the input linguistic item as;

an input natural language linguistic item, which expresses an intent using natural language;

or

an input query language linguistic item, which expresses the intent using one or more keywords.

12. A method implemented using one or more computing devices, the method comprising:

producing and storing a set of pair items based at least on relative similarity of the pair items as determined using a query click log, each pair item in the set of pair items including;

(a) a natural language (NL) linguistic item, which expresses an intent using natural language; and

(b) a keyword language (KL) linguistic item, which expresses the intent using one or more keywords, and is related to the NL linguistic item; and

using a machine learning technique to generate and store an NL-KL classification model based, in part, on the set of pair items,

the NL-KL classification model being configured to identify whether an input linguistic item corresponds to either an NL linguistic item or a KL linguistic item.

13. The method of claim 12, wherein said producing of the set of pair items comprises:

receiving a set of domain-labeled linguistic items;

determining a set of domain-independent linguistic items based at least on the set of domain-labeled linguistic items;

determining a set of matching linguistic items that contain at least one domain-independent linguistic item from the set of domain-independent linguistic items;

determining a set of linked linguistic items that are linked to the set of matching linguistic items, based at least on common actions taken by users, as reflected in the query click log; and

formulating the set of pair items based at least on the set of matching linguistic items and the set of linked linguistic items.

14. The method of claim 13, wherein the set of linked linguistic items are associated with the set of matching linguistic items based at least on common selections made by users in response to submitting the set of matching linguistic items and the set of linked linguistic items.

15. The method of claim 12, wherein generating of the NL-KL classification model comprises:

applying a first machine learning technique to the set of pair items, to produce an initial classification model;

receiving a set of new linguistic items;

using the initial classification model to determine whether each new linguistic item in the set of new linguistic items is an NL linguistic item or a KL linguistic item, to produce a set of initially-labeled linguistic items;

removing noise from the set of initially-labeled linguistic items, to produce a set of filtered linguistic items;

receiving additional labels for at least part of the set of filtered linguistic items provided by human annotators, to produce a set of refined linguistic items; and

applying a second machine learning technique to the set of refined linguistic items to produce the NL-KL classification model.

16. The method of claim 15,

wherein the first machine learning technique uses a first set of features, and the second machine learning techniques uses a second set of features,

wherein the first set of features includes lexical features and structural features, and

wherein the second set of features includes lexical features, structural features, and semantic features.

17. The method of claim 16, wherein at least one semantic feature identifies a presence of non-NL content in an input linguistic item, where that non-NL content resembles NL content.

18. The method of claim 12, further comprising applying the NL-KL classification model by:

receiving a set of input linguistic items;

using the NL-KL classification model to classify each of the input linguistic items as either an NL linguistic item or a KL linguistic item, to produce, overall, a subset of NL linguistic items and a subset of KL linguistic items; and

using a machine learning technique to generate a natural language interpretation model, based at least on the subset of NL linguistic items.

19. The method of claim 12, further comprising applying the NL-KL classification model by:

receiving a filter request that specifies a filtering condition;

producing a collection of linguistic items that match the filtering condition, based, at least in part, on a knowledge resource; and

classifying each of the collection of linguistic items as either an NL linguistic item or a KL linguistic item, to produce, overall, a subset of NL linguistic items and a subset of KL linguistic items.

20. The method of claim 12, further comprising applying the NL-KL classification model by:

receiving an input linguistic item;

using the NL-KL classification model to classify the input linguistic item as either an NL linguistic item or a KL linguistic item;

performing a first action when the input linguistic item is an NL linguistic item; and

performing a second action when the input linguistic item is a KL linguistic item.