Semantic exploration and discovery

• US 20080010274A1
• Filed: 06/20/2007
• Published: 01/10/2008
• Est. Priority Date: 06/21/2006
• Status: Active Grant

• Current Assignee: Information Extraction Systems, Inc.
• Original Assignee: Information Extraction Systems, Inc.
• Inventors: DePlonty, Thomas, Carus, Alwin
• Granted Patent: US 7,558,778 B2
• Time in Patent Office: Days
• US Class Current: 707/005
• CPC Class Codes: G06F 16/36 Creation of semantic tools, e.g. ontology or thesauri
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access

Abstract

A semantic discovery and exploration system is disclosed where an environment enabling a developer or user to uncover, navigate, and organize semantic patterns and structures in a document collection with or without the aid of structured knowledge. The semantic discovery and exploration system provides techniques for searching document collections, categorizing documents, inducing lists of related concepts, and identifying clusters of related terms and documents. This system operates both without and with infusions of structured knowledge such as gazetteers, thesauruses, taxonomies and ontologies. System performance improves when structured knowledge is incorporated. The semantic discovery and exploration system may be used as a first step in developing an information extraction system such as to categorize or cluster documents in a particular domain or to develop gazetteers and as a part of a deployed run-time information extraction system. It may also be used as standalone utility for searching, navigating, and organizing document collections and structured knowledge bases such as dictionaries or domain-specific reference works.

Claims

1. A method for exploring and organizing a first electronic corpus of documents stored in a computer storage medium, the method comprising the steps of:

performing at least one of reviewing the text of the documents from the first electronic corpus of documents in a concordance form, collecting terms from the first electronic corpus of documents in order to build semantically related terms, or collecting documents from the first electronic corpus of documents in order to build semantically related documents clusters;

creating a first set, the first set having at least one category applying to at least one of the words and phrases in gazetteers, or at least one document in the semantically related document clusters;

creating a second set, the second set having at least one of a candidate document cluster or a candidate words and phrases list;

evaluating the second set based upon a set of predetermined factors in order to create a third set, where the third set includes at least one document semantically related to the candidate clusters or at least one semantically related word and phrase related to the candidate words and phrases that meet at least one of the predetermined factors; and

selectively substituting the third set for the first set in a subsequent iteration of the method for exploring.

2. The method according to claim 1 where the predetermined factors include a stopping condition.

3. The method according to claim 1 further comprising the step of using preexisting lists of terminology.

4. The method according to claim 1 further comprising the step of annotating words and phrases for membership in a predetermined set of categories in a predetermined number of training documents.

5. The method according to claim 1 further comprising the step of clustering documents for membership in a predetermined set of categories in a predetermined number of training documents.

6. The method according to claim 1 where the performing step is performed in at least one of an unsupervised mode or a semi-supervised mode.

7. The method according to claim 6 further comprising the step of identifying a first set of words and phrases in a second corpus of documents.

8. The method according to claim 6 further comprising the step of identifying a first set of document clusters in a second corpus of documents.

9. The method according to claim 6 further comprising the step of identifying contexts in which the first set of words and phrases occur.

10. The method according to claim 9 further comprising the step of formatting the contexts so that substantially similar candidate contexts are identified in the second corpus.

11. The method according to claim 10 further comprising the step of using the formatted contexts and proposing additional candidate terms with their respective proposed category assignments, based on associations between known contexts, the candidate contexts generated relative thereto, and the terms from the first corpus.

12. The method according to claim 10 further comprising the step of identifying contexts in which the first set of words and phrases occur.

13. The method according to claim 10 further comprising the step of identifying attributes of the first set of document clusters.

14. The method according to claim 13 further comprising the step of formatting the contexts so that substantially similar candidate contexts are identified in the second corpus.

15. The method according to claim 14 further comprising the identification of document attributes so that substantially similar candidate document attributes are identified in the second corpus.

16. The method according to claim 15 further comprising the step of using the formatted contexts and proposing additional candidate documents with their respective proposed category assignments, based on associations between known contexts, the candidate contexts generated relative thereto, and the terms from the first corpus.

17. The method according to claim 15 further comprising the step of using the document attributes and proposed additional candidate documents and proposed respective document cluster assignments, based on associations between known document attributes, the candidate attributes generated relative thereto, and the documents from the first corpus.

18. The method according to claims 1 further comprising the step organizing the concepts into at least one of a semantic thesaurus and an ontology.

19. The method according to claim 1 further comprising the step of using words, phrases, or annotations of the documents to provide examples for review in a concordance graphical user interface.

20. The method according to claim 19 further comprising the step of using a structured knowledge source.

21. The method according to claim 20 further comprising the step using a query graphical user interface.

22. A computer system for exploring and organizing an electronic corpus of documents stored in a computer storage medium, the computer system comprising:

an exploration resource module for managing the semantic exploration process;

an exploration engine in communication with the exploration resource module

a document management module in communication with the exploration resource module, the document management module for feeding document sets into the exploration resource module;

an ontology management module in communication with the exploration resource module, the ontology management module for feeding and receiving sets of ontologies to and from the exploration resource module;

a resource definitions database in communication with the exploration resource module, the resource definitions database for feeding and receiving sets of definitions into and from the exploration resource module;

a query manager module in communication with the exploration engine, the query manager module having access to a query definitions database and a query results database, the query manager module foe feeding and receiving query information to and from the exploration engine, and

an exploration interface in communication with the exploration engine, the exploration interface for displaying information related to the semantic exploration process.

24. The computer system according to claim 22 further comprising a set of predetermined ontologies, where the set of ontologies are directed into the exploration resource module via the ontology management module.

25. The computer system according to claim 22 further comprising at least one annotation interface, the annotation interface in communication with the exploration resource module.