Ontology-crowd-relevance deep response generation

• US 10,061,848 B2
• Filed: 05/22/2015
• Issued: 08/28/2018
• Est. Priority Date: 05/22/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Basu, Sumit, Vanderwende, Lucretia, Labutov, Igor I.
• Primary Examiner(s): Tran, Anh Tai
• Application Number:
  US14/720,278
• Publication Number:
  US 20160342685A1
• Time in Patent Office: 1,194 Days
• Field of Search: None
• CPC Class Codes: G06F 16/367 Ontology

Abstract

Generating responses to input utilizing an ontology-crowd-relevance methodology is described. The techniques described herein access a plurality of data items and determine an ontology associated with the plurality of data items. The ontology includes one or more ontological elements. Furthermore, the techniques describe sending, to a plurality of devices, a request to generate response templates based on the one or more ontological elements and receiving, from the plurality of devices, the response templates directed to the one or more ontological elements.

Claims

1. A system comprising:

one or more processors;

memory; and

one or more modules stored in the memory and executable by the one or more processors to perform operations comprising;

accessing a plurality of data items, the plurality of data items being associated with one or more labels and one or more sub-labels;

determining, using at least one of latent Dirichlet allocation, a word cloud, and a classifier model, an ontology for the plurality of data items the ontology including one or more ontological elements, the ontology including a product of the label and the sub-label, the one or more ontological elements comprise a label of the one or more labels that is semantically associated with a data item of the data items and a sub-label of the sub-labels that is semantically associated with a portion of the data item;

crowdsourcing response templates including sending, to a plurality of devices with associated different users, a request for a respective user of the different users to generate one or more response templates based on the one or more ontological elements, one or more of the response templates including a question with a subject of the question as a slot and an ontological element of the one or more ontological elements in the question;

receiving, from the plurality of devices, the response templates generated by the different users and directed to the one or more ontological elements

storing the response templates in a database;

receiving a new data item associated with the one or more ontological elements; and

based at least in part on receiving the new data item, accessing an individual response template of the response templates stored in the database, the individual response template mapped to the one or more ontological elements.

2. The system as claim 1 recites, wherein the response templates are applicable data items that are associated with the one or more ontological elements.

3. The system as claim 1 recites, the operations further comprising generating a response based at least in part on an individual response template of the response templates directed to the one or more ontological elements, the generating comprising:

accessing a data item from the plurality of data items, the data item associated with the one or more ontological elements;

accessing, from a database storing the response templates, the individual response template; and

supplementing the individual response template with corresponding information specific to the data item to generate the response.

4. The system as claim 3 recites, the operations further comprising:

sending the response to a second plurality of devices with associated second different users; and

receiving, from the second different users and through the second plurality of devices, feedback including a relevance score associated with the individual response template, the relevance score based at least in part on determining a relatedness between the response and the data item.

5. The system as claim 4 recites, the operations further comprising training a relevance model based at least in part on the relevance score, the data item, and the response.

6. A computer-implemented method comprising:

accessing a plurality of data items, the plurality of data items being associated with one or more labels and one or more sub-labels;

determining, using at least one of latent Dirichlet allocation, a word cloud, and a classifier model, an ontology for the plurality of data items, the ontology including one or more ontological elements, the ontology including a product of one or more labels of the one or more labels and one or more sub-labels of the one or more sub-labels, the one or more ontological elements comprise a label of the one or more labels that is semantically associated with a data item of the data items, and a sub-label of the one or more sub-labels that is semantically associated with a portion of the data item;

crowdsourcing response templates including sending, to a plurality of devices, a request for respective users of the plurality of devices to generate a response template associated with a label of the one or more labels and a sub-label of the one or more sub-labels;

receiving, from the plurality of devices, a plurality of response templates generated by the respective users and associated with the label and the sub-label, wherein the plurality of response templates are applicable to data items associated with the label and sub-label, one or more of the plurality of response templates including a question with a subject of the question as a slot and the label and the sub-label in the question;

storing at least some response templates of the plurality of response templates in a database;

receiving a new data item associated with the one or more ontological elements; and

based at least in part on receiving the new data item, accessing individual response templates of the response templates stored in the database, the individual response templates mapped to the one or more ontological elements.

7. A computer-implemented method as claim 6 recites, wherein the database maps each of the plurality of response templates to the label and the sub-label.

8. A computer-implemented method as claim 6 recites, wherein:

individual data items of the plurality of data items are text documents;

the label corresponds to a category that is semantically associated with a majority of a text document of the text documents; and

the sub-label corresponds to a section that is semantically associated with a portion of the text document, the portion being less than the majority of the text document.

9. A computer-implemented method as claim 6 recites, further comprising training a ontology model based at least in part on the plurality of data items and a corresponding label and sub-label, the ontology model being trained to infer the label and the sub-label from new data items.

10. A computer-implemented method as claim 6 recites, further comprising:

accessing a data item of the plurality of data items;

generating a response associated with the data item based at least in part on supplementing an individual response template of the plurality of response templates with corresponding information specific to the data item;

sending the response to one or more devices; and

receiving, from the one or more devices, feedback including a relevance score associated with the response, the relevance score based at least in part on a relatedness between the response and the data item.

11. A computer-implemented method as claim 10 recites, further comprising training a relevance model based at least in part on the relevance score, the response, and the data item, the training including:

determining a first feature vector based at least in part on a distance between the data item and the response;

accessing a second data item with a second relevance score that is above a predetermined threshold, the second relevance score based at least in part on the relatedness between the second data item and the response;

determining a second feature vector based at least in part on a distance between the data item and the second data item;

combining the first feature vector and the second feature vector to determine a combined feature vector; and

training the relevance model based at least in part on the combined feature vector, the response, and the data item.