System and method to extract models from semi-structured documents

• US 10,089,390 B2
• Filed: 09/24/2010
• Issued: 10/02/2018
• Est. Priority Date: 09/24/2010
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Ananthanarayanan, Rema, Bhamidipaty, Anuradha, Kummamuru, Krishna, Mukherjee, Debdoot, Padmanabhan, Deepak S., Sinha, Vibha Singhal, Srivastava, Biplav
• Primary Examiner(s): Adams, Charles D
• Application Number:
  US12/890,060
• Publication Number:
  US 20120078969A1
• Time in Patent Office: 2,930 Days
• Field of Search: 707798
• CPC Class Codes: G06F 16/355 Class or cluster creation or modification

Abstract

Systems and associated methods for automated and semi-automated building of domain models for documents are described. Embodiments provide an approach to discover an information model by mining documentation about a particular domain captured in the documents. Embodiments classify the documents into one or more types corresponding to concepts using indicative words, identify candidate model elements (concepts) for document types, identify relationships both within and across document types, and consolidate and learn a global model for the domain.

Claims

1. A method for producing a global model describing a collection of documents comprising:

executing with one or more processors one or more modules of computer program code configured for accessing a collection of documents, the collection of documents comprising labeled documents and unlabeled documents;

receiving input of at least one indicative word, wherein the at least one indicative word comprises a descriptive word for classification and wherein the at least one indicative word indicates a probability of belonging to a classification based upon the indicative word occurring in a document during classification;

classifying both labeled documents and unlabeled documents of the collection of documents to produce classified documents of one or more types, wherein the classifying comprises producing a domain sub-model for each document type, wherein the domain sub-model represents a graphical representation of a set of concepts contained within each document type and wherein the domain sub-model is generated using the labeled documents and the at least one indicative word;

wherein the producing a domain sub-model for each document type comprises extracting concepts from each of the documents and determining relationships between the concepts, wherein the extracting concepts comprises producing concept pairs by identifying, within the collection of documents, co-occurring candidate concepts and wherein the determining relationships between the concepts comprises identifying relationship links between source and destination candidate concepts, wherein the identifying relationship links comprises extracting, from each of the documents of the collection of documents, a hierarchical structure, searching for adjacent container pairs within the hierarchical structures, and inferring directed relationships between elements within the adjacent container pairs;

thereupon generating a global domain model for the documents of the collection of the documents by merging the produced domain sub-models, based on the relationships between the concepts;

said generating of a global domain model comprising aggregating identified relationship links and corresponding concepts of each of the domain sub-models across the produced domain sub-models, wherein the relationship links and corresponding concepts selected for aggregation are based upon a strategy identified based upon a level of manual review;

thereupon outputting the global model as a graphical representation comprising the aggregated concepts and relationship links between concepts;

ascertaining one or more changes to the collection of documents; and

generating a new global model based on the one or more changes to the collection of documents by reclassifying the collection of documents and generating a new global model using the new domain sub-models generated during reclassification of the collection of documents.

2. The method according to claim 1, wherein:

the documents correspond to a plurality of document types; and

said extracting of concepts from the classified documents further comprises identifying concepts and links for each document type.

3. The method according to claim 2, wherein the document types correspond to at least one of:

differing domains; and

differing classification models.

4. The method according to claim 1, further comprising accepting user input indicating one or more of threshold input and validation input.

5. The method according to claim 1, wherein the global model is output as a graph, the graph comprising a tree structure wherein nodes represent concepts and edges represent relationships between concepts.

6. A computer program product for producing a global model describing a collection of documents comprising:

a non-transitory computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising;

computer readable program code configured to access a collection of documents, the collection of documents comprising labeled documents and unlabeled documents;

computer readable program code configured to receive input of at least one indicative word, wherein the at least one indicative word comprises a descriptive word for classification and wherein the at least one indicative word indicates a probability of belonging to a classification based upon the indicative word occurring in a document during classification;

computer readable program code configured to classify both labeled documents and unlabeled documents of the collection of documents to produce classified documents of one or more types, via producing a domain sub-model for each document type, wherein the domain sub-model represents a graphical representation of a set of concepts contained within each document type and wherein the domain sub-model is generated using the labeled documents and the at least one indicative word;

wherein the producing a domain sub-model for each document type comprises extracting concepts from each of the documents and determine relationships between the concepts, wherein the extracting concepts comprises producing concept pairs by identifying, within the collection of documents, co-occurring candidate concepts and wherein the determining relationships between the concepts comprises identifying relationship links between source and destination candidate concepts, wherein the identifying relationship links comprises extracting, from each of the documents of the collection of documents, a hierarchical structure, searching for adjacent container pairs within the hierarchical structures, and inferring directed relationships between elements within the adjacent container pairs;

computer readable program code configured to thereupon generate a global domain model for the documents of the collection of the documents by merging the produced domain sub-models, based on the relationships between the concepts, via aggregating identified relationship links and corresponding concepts of each of the domain sub-models across the produced domain sub-models, wherein the relationship links and corresponding concepts selected for aggregation are based upon a strategy identified based upon a level of manual review;

computer readable program code configured to thereupon output the global model as a graphical representation comprising the aggregated concepts and relationship links between concepts;

computer readable program code configured to ascertain one or more changes to the collection of documents; and

computer readable program code configured to generate a new global model based on the one or more changes to the collection of documents by reclassifying the collection of documents and generating a new global model using the new domain sub-models generated during reclassification of the collection of documents.

7. The computer product according to claim 6, wherein the documents correspond to a plurality of document types, and said computer readable program code is configured to extract concepts from the classified documents via identifying concepts and links for each document type.

8. The computer program product according to claim 6, further comprising computer readable program code configured to ascertain user input indicating one or more of threshold input and validation input.

9. The computer program product according to claim 6, wherein the global model is output as a graph, the graph comprising a tree structure wherein nodes represent concepts and edges represent relationships between concepts.

10. A system for producing a global model describing a collection of documents comprising:

one or more processors; and

a memory operatively connected to the one or more processors;

wherein, responsive to execution of computer readable program code accessible to the one or more processors, the one or more processors are configured to;

access a collection of documents, the collection of documents comprising labeled documents and unlabeled documents;

receive input of at least one indicative word, wherein the at least one indicative word comprises a descriptive word for classification and wherein the at least one indicative word indicates a probability of belonging to a classification based upon the indicative word occurring in a document during classification;

classify both labeled documents and unlabeled documents of the collection of documents to produce classified documents of one or more types, via producing a domain sub-model for each document type, wherein the domain sub-model represents a graphical representation of a set of concepts contained within each document type and wherein the domain sub-model is generated using the labeled documents and the at least one indicative word;

wherein the producing a domain sub-model for each document type comprises extracting concepts from each of the documents and determine relationships between the concepts, wherein to extract concepts comprises producing concept pairs by identifying, within the collection of documents, co-occurring candidate concepts and wherein the determining relationships between the concepts comprises identifying relationship links between source and destination candidate concepts, wherein the identifying relationship links comprises extracting, from each of the documents of the collection of documents, a hierarchical structure, searching for adjacent container pairs within the hierarchical structures, and inferring directed relationships between elements within the adjacent container pairs;

thereupon generate a global domain model for the documents of the collection of the documents by merging the produced domain sub-models, based on the relationships between the concepts;

the generating of a global model comprising aggregating identified relationship links and corresponding concepts of each of the domain sub-models across the produced domain sub-models, wherein the relationship links and corresponding concepts selected for aggregation are based upon a strategy identified based upon a level of manual review;

thereupon output the global model as a graphical representation comprising the aggregated concepts and relationship links between concepts;

ascertain one or more changes to the collection of documents; and

generate a new global model based on the one or more changes to the collection of documents by reclassifying the collection of documents and generating a new global model using the new domain sub-models generated during reclassification of the collection of documents.