SYSTEM AND METHOD TO EXTRACT MODELS FROM SEMI-STRUCTURED DOCUMENTS

• US 20120078969A1
• Filed: 09/24/2010
• Published: 03/29/2012
• Est. Priority Date: 09/24/2010
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Ananthanarayanan, Rema, Bhamidipaty, Anuradha, Kummamuru, Krishna, Mukherjee, Debdoot, Padmanabhan, Deepak S., Sinha, Vibha Singhal, Srivastava, Biplav
• Granted Patent: US 10,089,390 B2
• Time in Patent Office: Days
• US Class Current: 707/798
• CPC Class Codes: G06F 16/355 Class or cluster creation or modification

Abstract

Systems and associated methods for automated and semi-automated building of domain models for documents are described. Embodiments provide an approach to discover an information model by mining documentation about a particular domain captured in the documents. Embodiments classify the documents into one or more types corresponding to concepts using indicative words, identify candidate model elements (concepts) for document types, identify relationships both within and across document types, and consolidate and learn a global model for the domain.

Claims

1. A method for producing a global model describing a collection of documents comprising:

accessing a collection of documents, the collection of documents comprising labeled documents and unlabeled documents;

receiving input identifying indicative words for classifications;

generating a classification model;

classifying documents of the collection of documents to produce classified documents of one or more types;

extracting concepts from the classified documents;

generating a global model from the concepts; and

outputting the global model.

2. The method according to claim 1, further comprising providing one or more indicative words as suggestions to a user based on the classified documents.

3. The method according to claim 2, further comprising:

accepting one or more indicative words chosen by the user as input; and

re-classifying the documents using the one or more indicative words chosen by the user.

4. The method according to claim 1, wherein extracting concepts from the classified documents further comprises identifying concepts and links for each document type.

5. The method according to claim 4, wherein generating a global model from the concepts further comprises producing a sub-model per document type.

6. The method according to claim 5, wherein generating a global model from the concepts further comprises aggregating concepts and links across sub-models.

7. The method according to claim 1, further comprising accepting user input indicating one or more of threshold input and validation input.

8. The method according to claim 1, further comprising:

ascertaining one or more changes to the collection of documents; and

generating a new global model based on the one or more changes to the collection of documents.

9. The method according to claim 1, wherein the global model is output as a graph, the graph comprising a tree structure wherein nodes represent concepts and edges represent relationships between concepts.

10. A computer program product for producing a global model describing a collection of documents comprising:

a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising;

computer readable program code configured to access a collection of documents, the collection of documents comprising labeled documents and unlabeled documents;

computer readable program code configured to ascertain input identifying indicative words for classifications;

computer readable program code configured to generate a classification model;

computer readable program code configured to classify documents of the collection of documents to produce classified documents of one or more types;

computer readable program code configured to extract concepts from the classified documents;

computer readable program code configured to generate a global model from the concepts; and

computer readable program code configured to output the global model.

11. The computer program product according to claim 10, further comprising computer readable program code configured to provide one or more indicative words as suggestions to a user based on the classified documents.

12. The computer program product according to claim 11, further comprising:

computer readable program code configured to ascertain one or more indicative words chosen by the user as input; and

computer readable program code configured to re-classifying the documents using the one or more indicative words chosen by the user.

13. The computer product according to claim 10, wherein to extract concepts from the classified documents further comprises identifying concepts and links for each document type.

14. The computer program product according to claim 13, wherein to generate a global model from the concepts further comprises producing a sub-model per document type.

15. The computer program product according to claim 14, wherein generating a global model from the concepts further comprises aggregating concepts and links across sub-models.

16. The computer program product according to claim 10, further comprising computer readable program code configured to ascertain user input indicating one or more of threshold input and validation input.

17. The computer program product according to claim 10, further comprising:

computer readable program code configured to ascertain one or more changes to the collection of documents; and

computer readable program code configured to generate a new global model based on the one or more changes to the collection of documents.

18. The computer program product according to claim 10, wherein the global model is output as a graph, the graph comprising a tree structure wherein nodes represent concepts and edges represent relationships between concepts.

19. A system for producing a global model describing a collection of documents comprising:

one or more processors; and

a memory operatively connected to the one or more processors;

wherein, responsive to execution of computer readable program code accessible to the one or more processors, the one or more processors are configured to;

access a collection of documents, the collection of documents comprising labeled documents and unlabeled documents;

receive input identifying indicative words for classifications;

generate a classification model;

classify documents of the collection of documents to produce classified documents of one or more types;

extract concepts from the classified documents;

generate a global model from the concepts; and

output the global model.

20. The system according to claim 19, further comprising:

providing one or more indicative words as suggestions to a user based on the classified documents;

accepting one or more indicative words chosen by the user as input; and

re-classifying the documents using the one or more indicative words chosen by the user.