Measuring entity extraction complexity

• US 8,140,567 B2
• Filed: 04/13/2010
• Issued: 03/20/2012
• Est. Priority Date: 04/13/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Padovitz, Amir J., Nagarajan, Bala Meenakshi
• Primary Examiner(s): Morrison, Jay
• Assistant Examiner(s): Ahn, Sangwoo
• Application Number:
  US12/759,513
• Publication Number:
  US 20110252034A1
• Time in Patent Office: 707 Days
• Field of Search: None
• US Class Current: 707/769
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: G06F 16/93 Document management systems

Abstract

A named entity input is received and a target sense for which the named entity input is to be extracted from a set of documents is identified. An extraction complexity feature is generated based on the named entity input, the target sense, and the set of documents. The extraction complexity feature indicates how difficult or complex it is deemed to be to identify the named entity input for the target sense in the set of documents.

Claims

1. A method implemented in a computing device, the method comprising:

receiving a named entity input;

identifying a target sense for which the named entity input is to be extracted from a set of documents; and

generating by the computing device, based at least in part on both the named entity input and the set of documents, an extraction complexity feature indicating how difficult it is deemed to be to identify the named entity input for the target sense in the set of documents by;

performing a graph-based spreading activation technique to generate a language model including;

building an undirected graph based on the named entity input and the set of documents, the undirected graph including multiple vertices and multiple edges,

incrementing scores of selected ones of the multiple vertices by propagating a relevance of one or more of the multiple vertices through the undirected graph, and

normalizing, after incrementing the scores of the selected ones of the multiple vertices, scores of the multiple vertices to obtain the language model;

performing a clustering technique to refine the language model resulting in a refined language model that includes multiple clusters each including one or more documents; and

determining an extraction complexity measurement for the named entity input by;

determining a relatedness of each of the multiple clusters in the refined language model to the target sense;

assigning, for each of the multiple clusters, a score to the cluster based on the relatedness of the cluster to the target sense;

determining an average cluster score that is an average of the scores of the multiple clusters;

identifying, as a value |C*|, a number of documents in clusters having a score greater than the average cluster score;

identifying, as a value |D|, a number of documents in the set of documents; and

determining the extraction complexity measurement as;

2. A method as recited in claim 1, further comprising providing the extraction complexity feature to a named entity recognition module that identifies the named entity input in the set of documents based at least in part on the extraction complexity feature.

3. A method as recited in claim 1, wherein the named entity input is received from a source, and the method further comprises receiving the target sense from the source.

4. A method as recited in claim 1, wherein the computing device is included in an open system having no knowledge of all the different senses in which the named entity input can be used.

5. A method implemented in a computing device, the method comprising:

receiving a named entity input from a source;

identifying a target sense for the named entity input, wherein the target sense is a particular desired usage of the named entity input in a set of documents; and

generating by the computing device, based at least in part on both the named entity input and the set of documents, an extraction complexity measurement that indicates how difficult it is deemed to be to identify the named entity input in the set of documents for the target sense, the generating the extraction complexity measurement comprising;

performing a graph-based spreading activation technique to generate a language model by;

building an undirected graph based on the named entity input and the set of documents, the undirected graph including multiple vertices and multiple edges,

incrementing scores of selected ones of the multiple vertices by propagating a relevance of one or more of the multiple vertices through the undirected graph, and

normalizing, after incrementing the scores of the selected ones of the multiple vertices, scores of the multiple vertices to obtain the language model;

performing a graph-based clustering technique to refine the language model, wherein the refined language model includes multiple clusters; and

determining the extraction complexity measurement based on the refined language model, wherein the determining the extraction complexity measurement comprises identifying a number of documents in clusters of the multiple clusters having a score greater than an average cluster score.

6. A method as recited in claim 5, further comprising providing the extraction complexity measurement to a named entity recognition module that identifies the named entity input in the set of documents based at least in part on the extraction complexity measurement.

7. A method as recited in claim 5, wherein the identifying the target sense comprises receiving the target sense from the source.

8. A method as recited in claim 5, wherein the computing device is included in an open system in which the computing device has no knowledge of all the different senses in which the named entity input can be used.

9. One or more computer storage media having stored thereon multiple instructions that, when executed by one or more processors of a computing device, cause the one or more processors to:

receive a named entity input from a source;

identify a target sense for the named entity input, wherein the target sense is a particular desired usage of the named entity input in a set of documents; and

generate, based at least in part on both the named entity input and the set of documents, an extraction complexity measurement that indicates how difficult it is deemed to be to identify the named entity input in the set of documents for the target sense, wherein to generate the extraction complexity measurement is to;

perform a graph-based spreading activation technique to generate a language model by;

building an undirected graph based on the named entity input and the set of documents, the undirected graph including multiple vertices and multiple edges,

incrementing scores of selected ones of the multiple vertices by propagating a relevance of one or more of the multiple vertices through the undirected graph, and

normalizing, after incrementing the scores of the selected ones of the multiple vertices, scores of the multiple vertices to obtain the language model;

perform a graph-based clustering technique to refine the language model, wherein the refined language model includes multiple clusters; and

determine the extraction complexity measurement based on the refined language model, wherein to determine the extraction complexity measurement comprises to identify a number of documents in clusters of the multiple clusters having a score greater than an average cluster score.

10. One or more computer storage media as recited in claim 9, wherein the multiple instructions further cause the one or more processors to provide the extraction complexity measurement to a named entity recognition module that identifies the named entity input in the set of documents based at least in part on the extraction complexity measurement.

11. One or more computer storage media as recited in claim 9, wherein to identify the target sense is to receive the target sense from the source.

12. One or more computer storage media as recited in claim 9, wherein the computing device is included in an open system in which the computing device has no knowledge of all the different senses in which the named entity input can be used.