Techniques for knowledge discovery by constructing knowledge correlations using terms

• US 20060253431A1
• Filed: 11/14/2005
• Published: 11/09/2006
• Est. Priority Date: 11/12/2004
• Status: Active Grant

• Current Assignee: Make Sence Incorporated
• Original Assignee: Sense, Inc. (Cloudera Incorporated)
• Inventors: Bobick, Mark, Wimmer, Carl
• Granted Patent: US 8,108,389 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/20 of structured data, e.g. relational data
• CPC Class Codes: G06F 16/24522 Translation of natural language queries to structured queries
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9532 Query formulation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 40/247 Thesauruses; Synonyms
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition

Abstract

Techniques for identifying knowledge use an graphical user interface for inputting one or more terms to be explored for additional knowledge. Then a search is conducted across one or more sources of information to identify resources containing information about or information associated with said terms. The resources are decomposed into elemental units of information and stored in a data structures called nodes. A group of nodes are stored in a node pool and, from the node pool, correlations of nodes are constructed that represent knowledge.

Claims

1. A method for identifying knowledge, comprising the steps of:

a. inputting one or more terms to be explored for additional knowledge;

b. searching one or more sources of information to identify resources containing information about or information associated with said terms;

c. decomposing resources identified during searching into nodes;

d. storing nodes in a node pool; and

e. from the node pool, construct correlations of nodes representing knowledge.

2. The method of claim 1 in which the step of inputting comprises one of the steps of:

a. typing one or more terms into a command line;

or

b. inputing one or more terms into a graphical user interface;

or

c. inputting a natural language description of a concept.

3. The method of claim 2 in which the step of inputting a natural language description of a concept comprises the additional step of parsing the description into tokens to be explored for additional knowledge.

4. The method of claim 1 in which the step of searching comprises at least one of the following steps:

a. searching files on a personal computer;

b. searching files on one or more computers on a network;

or

c. searching files on a network server.

5. The method of claim 4 in which the files are translated into files containing text.

6. The method of claim 4 in which the step of searching comprises using one or more spiders to explore files.

7. The method of claim 6 in which a spider clones other spiders to facilitate searching.

8. The method of claim 6 in which at least one of said one or more spiders captures information about relevancy of a resource to the terms.

9. The method of claim 6 in which at least one of said one or more spiders comprises a meta-crawler.

10. The method of claim 1 in which said one or more sources of information comprises at least one of:

a. a file system;

b. the world wide web;

c. an email repository;

d. attachments to email in an email repository;

e. a relational data base management system;

f. a data warehouse;

f. a taxonomy;

g. an ontology;

h a semantic net;

i a neural net;

j. a search engine; and

k. an index engine.

11. The method of claim 1 in which the step of searching one or more sources of information to identify resources containing information about said terms comprises searching at least one of:

a. file name;

b. title of a document;

c. abstract of a document;

d. full text of a document; and

e. a pool of nodes.

11A. A. The method of claim 1 in which said resources are one or more of:

a. a portable document format (PDF) file;

b. an rich text format (RTF) file;

c. a word processing file;

d. a Microsoft PowerPoint (PPT) file;

e. a Hyper Text Markup Language (HTML) page;

f. a file containing an email; and

g. a file containing an email attachment.

12. The method of claim 11 in which identifying resources containing information about said terms comprises at least one of:

a. finding an exact match between content of a source of information and the terms to be explored;

b. finding a singular or plural version of the terms to be explored in the content of a source;

c. finding a synonym of the terms to be explored in the content of a source;

d. finding terms contained in a dictionary definition of a terms to be explored, in the content of a source;

e. finding terms contained in a discussion in an authoritative source of a terms to be explored, in the content of a source;

f. finding terms contained in a an entry in an encyclopedia discussing the terms to be explored, in the content of a source;

g. finding terms from a taxonomy, from an ontology or from a semantic net closely adjacent to a terms to be explored, in the content of a source;

13. The method of claim 1 in which a node comprises a data structure comprising a first relatum, a bond and a second relatum.

14. The method of claim 13 in which the node additionally comprises a source or sequence entry.

15. The method of claim 13 in which the step of decomposing resources containing information about said terms into nodes comprises parsing text into linguistic units.

15A. A. The method of claim 15 in which the linguistic units are XML statements.

15B. B. The method of claim 15A in which XML headers are discarded.

16. The method of claim 15 in which the linguistic units are sentences.

17. The method of claim 15 in which the step of decomposing resources comprises analyzing a linguistic unit using a natural language parser.

18. The method of claim 17 in which the natural language parser separates linguistic units into words or tokens of the language and assigns a category to each word or token.

19. The method of claim 18 in which the natural language parser discards any characters that are not permissible to be used in constructing words or tokens in the language.

20. The method of claim 18 in which words or tokens are placed into the fields of one or more node data structures based on their category.

21. The method of claim 20 in which node data structures are stored in a node pool.

22. The method of claim 21 in which the node pool permits selection and retrieval of individual nodes based on the contents of one or more parts of a node.

23. The method of claim 22 in which the node pool permits selection and retrieval using a hash map.

24. The method of claim 22 in which the node pool is a database.

25. The method of claim 1 in which correlations of nodes representing knowledge about said terms are constructed by linking nodes from the node pool.

26. The method of claim 25 in which said nodes are linked into one or more chains of nodes.

27. The method of claim 26 in which a chain of nodes begins with an origin node explicitly containing a term of interest.

28. The method of claim 27 in which a candidate node is added to said origin node, or to a terminus node at the end of a chain of nodes originating from said origin node when a second relatum of the origin node or terminus node is associated with a relatum of a candidate node.

29. The method of claim 28 in which the test for association is one or more of:

a. finding an exact match between a relatum of the candidate node and the second relatum of the origin node or terminus node;

b. finding a singular or plural version of a relatum of the candidate node and a second relatum of the origin node or terminus node;

c. finding a match between a synonym of a relatum of the candidate node and a second relatum of the origin node or terminus node;

d. finding a match between significant terms contained in a dictionary definition of a relatum of the candidate node and a second relatum of the origin node or terminus node;

e. finding a match between significant terms contained in a discussion in an authoritative source between a relatum of the candidate node and the second relatum of the origin node or terminus node;

f. finding a match between significant terms contained in an entry in an encyclopedia discussing a relatum of the candidate node and the second relatum of the origin node or terminus node;

g. finding terms from a taxonomy, from an ontology or from a semantic net closely adjacent to a relatum of the candidate node that match the second relatum of the origin node or terminus node;

30. The method of claim 28 in which candidate nodes are added to form a chain until one of:

a. a user halts correlation;

b. a set time expires;

c. said chain comprises a number of nodes greater than a specified number;

d. no further nodes in the pool can be associated with the origin node or the terminus node of a chain;

e. a preselected term from a target node is added to the correlation; and

f. a preselected target node is added to the correlation.

31. A computer program product comprising:

a. a memory medium; and

b. programming statements stored on said memory medium for controlling a computer to perform the functions of;

b1. inputting one or more terms to be explored for additional knowledge;

b2. searching one or more sources of information to identify resources containing information about or information associated with said terms;

b3. decomposing resources identified during searching into nodes;

b4. storing nodes in a node pool; and

b5. from the node pool, construct correlations of nodes representing knowledge.

32. Apparatus for identifying knowledge, comprising:

a. an input mechanism inputting one or more terms to be explored for additional knowledge;

b. a search mechanism for searching one or more sources of information to identify resources containing information about or information associated with said terms;

c. a analysis mechanism for decomposing resources identified during searching into nodes;

d. a storing mechanism for storing nodes in a node pool; and

e. a correlation mechanism for constructing correlations of nodes from the node pool representing knowledge.

33. A system comprising:

a. a network;

b. one or more computers connected to said network;

c. at least one computer connected to said network comprising

c1. an input mechanism inputting one or more terms to be explored for additional knowledge;

c2. a search mechanism for searching one or more sources of information to identify resources containing information about or information associated with said terms;

c3. a analysis mechanism for decomposing resources identified during searching into nodes;

c4. a storing mechanism for storing nodes in a node pool; and

c5. a correlation mechanism for constructing correlations of nodes from the node pool representing knowledge.