Automatic evaluation and improvement of ontologies for natural language processing tasks

• US 9,336,306 B2
• Filed: 03/21/2014
• Issued: 05/10/2016
• Est. Priority Date: 03/21/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: McAteer, Seamus R., McCloskey, Daniel J., Sogrin, Mikhail
• Primary Examiner(s): Neway, Samuel G
• Application Number:
  US14/221,392
• Publication Number:
  US 20150269139A1
• Time in Patent Office: 781 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/367 Ontology
• CPC Class Codes: G06F 40/30 Semantic analysis

Abstract

Mechanisms are provided for modifying an ontology for use with a natural language processing (NLP) task. A model of the ontology is obtained that comprises nodes representing concepts of the ontology and edges between nodes representing relationships between associated nodes of the ontology. A spreading activation operation is performed on the model of the ontology with the spreading activation operation matching a portion of a textual source to a matching node of the model and identifying related nodes to the matching node through edges of the model associated with the matching node to thereby generate an activation network. The activation network is evaluated with regard to a chosen NLP task to determine a performance metric for the NLP task associated with the nodes of the model. Based on the results, one of the model or a configuration of the activation network may be modified.

Claims

1. A method, in a data processing system comprising a processor and a memory, for modifying an ontology for use with a natural language processing (NLP) task, the method comprising:

obtaining, by the data processing system, a model of the ontology, wherein the model of the ontology comprises nodes representing concepts of the ontology and edges between nodes representing relationships between associated nodes of the ontology;

performing, by the data processing system, a spreading activation operation on the model of the ontology, wherein the spreading activation operation matches a portion of a textual source to a matching node of the model and identifies related nodes to the matching node through edges of the model associated with the matching node to thereby generate an activation network;

evaluating, by the data processing system, the activation network with regard to a chosen NLP task to determine a performance metric for the NLP task associated with the nodes of the model; and

in response to results of the evaluating indicating that performance of a node does not meet a desired performance for the NLP task, modifying at least one of the model or a configuration of the activation network based on results of the evaluation of the activation network.

2. The method of claim 1, wherein the chosen NLP task is one of word sense disambiguation, named entity detection, named entity resolution, named entity inference, or word sense inference.

3. The method of claim 1, wherein modifying the configuration of the activation network comprises modifying one or more weight values associated with a portion of the activation network.

4. The method of claim 1, wherein evaluating the activation network with regard to the chosen NLP task comprises performing differential analysis of characteristics of nodes in the model that are determined to be relatively higher performing in the activation network and nodes in the model that are determined to be relatively lower performing in the activation network.

5. The method of claim 4, wherein the characteristics of a node comprise at least one of a number of in-bound links of the node, a number of out-bound links of the node, or an identification of the node as being an isolated node.

6. The method of claim 1, wherein evaluating the activation network with regard to the chosen NLP task comprises performing static analysis of a subset of nodes in the model, identified in the activation network, to identify characteristics of the subset of nodes, and wherein modifying at least one of the model or the activation network comprises modifying a starting weight for a spreading activation signal of the spreading activation operation, for a node in the subset of nodes.

7. The method of claim 1, wherein evaluating the activation network with regard to the chosen NLP task comprises comparing the activation network to a golden standard corpus of text content to identify nodes in the activation network corresponding to instances of terms in the golden standard corpus of text and evaluating characteristics of the identified nodes to determine modifications to be performed to at least one of the model or the activation network.

8. The method of claim 1, wherein performing, by the data processing system, a spreading activation operation on the model comprises annotating the textual source using a pre-determined lexicon data structure and utilizing the annotated textual source to identify nodes in the model to activate as part of the spreading activation signal, and wherein the textual source comprises a set of test text for use in testing the ontology.

9. The method of claim 1, wherein the performance metric for the NLP task comprises at least one of a precision metric, a recall metric, an F-measure metric, or an accuracy metric.

10. The method of claim 1, further comprising:

performing the NLP task utilizing the modified graph representation or activation network.

11. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:

obtain a model of an ontology, wherein the model of the ontology comprises nodes representing concepts of the ontology and edges between nodes representing relationships between associated nodes of the ontology;

perform a spreading activation operation on the model of the ontology, wherein the spreading activation operation matches a portion of a textual source to a matching node of the model and identifies related nodes to the matching node through edges of the model associated with the matching node to thereby generate an activation network;

evaluate the activation network with regard to a chosen NLP task to determine a performance metric for the NLP task associated with the nodes of the model; and

in response to results of the evaluating indicating that performance of a node does not meet a desired performance for the NLP task, modify at least one of the model or a configuration of the activation network based on results of the evaluation of the activation network.

12. The computer program product of claim 11, wherein the chosen NLP task is one of word sense disambiguation, named entity detection, named entity resolution, named entity inference, or word sense inference.

13. The computer program product of claim 11, wherein the computer readable program further causes the processor to modify the configuration of the activation network at least by modifying one or more weight values associated with a portion of the activation network.

14. The computer program product of claim 11, wherein the computer readable program further causes the processor to evaluate the activation network with regard to the chosen NLP task at least by performing differential analysis of characteristics of nodes in the model that are determined to be relatively higher performing in the activation network and nodes in the model that are determined to be relatively lower performing in the activation network.

15. The computer program product of claim 14, wherein the characteristics of a node comprise at least one of a number of in-bound links of the node, a number of out-bound links of the node, or an identification of the node as being an isolated node.

16. The computer program product of claim 11, wherein the computer readable program further causes the processor to evaluate the activation network with regard to the chosen NLP task at least by performing static analysis of a subset of nodes in the model, identified in the activation network, to identify characteristics of the subset of nodes, and wherein the computer readable program further causes the processor to modify at least one of the model or the activation network at least by modifying a starting weight for a spreading activation signal of the spreading activation operation, for a node in the subset of nodes.

17. The computer program product of claim 11, wherein the computer readable program further causes the processor to evaluate the activation network with regard to the chosen NLP task at least by comparing the activation network to a golden standard corpus of text content to identify nodes in the activation network corresponding to instances of terms in the golden standard corpus of text and evaluating characteristics of the identified nodes to determine modifications to be performed to at least one of the model or the activation network.

18. The computer program product of claim 11, wherein the computer readable program further causes the processor to perform a spreading activation operation on the model at last by annotating the textual source using a pre-determined lexicon data structure and utilizing the annotated textual source to identify nodes in the model to activate as part of the spreading activation signal, and wherein the textual source comprises a set of test text for use in testing the ontology.

19. The computer program product method of claim 11, wherein the performance metric for the NLP task comprises at least one of a precision metric, a recall metric, an F-measure metric, or an accuracy metric.

20. An apparatus comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

obtain a model of an ontology, wherein the model of the ontology comprises nodes representing concepts of the ontology and edges between nodes representing relationships between associated nodes of the ontology;

perform a spreading activation operation on the model of the ontology, wherein the spreading activation operation matches a portion of a textual source to a matching node of the model and identifies related nodes to the matching node through edges of the model associated with the matching node to thereby generate an activation network;

evaluate the activation network with regard to a chosen NLP task to determine a performance metric for the NLP task associated with the nodes of the model; and

in response to results of the evaluating indicating that performance of a node does not meet a desired performance for the NLP task, modify at least one of the model or a configuration of the activation network based on results of the evaluation of the activation network.