Method and system to analyze data

• US 7,493,252 B1
• Filed: 07/07/2000
• Issued: 02/17/2009
• Est. Priority Date: 07/07/1999
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Nagano, Tohru, Nasukawa, Tetsuya
• Primary Examiner(s): Han; Qi
• Application Number:
  US09/612,136
• Time in Patent Office: 3,147 Days
• Field of Search: 707/5, 707/102, 707/100, 707/101, 707/2, 707/1, 395/600, 704/9, 704/10, 704/1, 706/12
• US Class Current: 704/9
• CPC Class Codes: G06F 16/30 of unstructured textual data document management systems G06F16/93
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access

Abstract

Useful knowledge is acquired from a large amount of data by extracting concepts of a unique characteristic. The present invention comprises a concept extractor and a unique concept extractor. The concept extractor extracts categorized concepts from the data. The unique concept extractor is a device for extracting unique concepts from those extracted concepts, and extracts in the categorized concepts, of the concepts belonging to the same category, a concept whose statistical characteristic is distinguished beyond a threshold with respect to the set in which it belongs.

Claims

1. A method of mining a collection of data, comprising:

receiving the collection of data, the collection of data comprising key words, wherein a key word comprises a coherent character string;

converting the collection of data into labeled data by grouping various types of data into a same format and assigning a label indicating a category of item contents, such that the labeled data is in analyzable condition for concept extraction, and wherein the labeled data comprises the label and a clause comprising the item contents;

assigning a category to the key words, wherein the category references a concept so that the key words can be handled as concepts with a meaning;

separating the clauses into pairs comprising an independent word and an attached word;

assigning categories to the separated clauses using syntactic patterns and a category dictionary;

generating, by syntactic analysis, a syntactic tree of a sentence comprising the separated clauses;

receiving a syntactically analyzed sentence as input, identifying mutually dependent relationships between or among the categorized key words, according to at least one rule defining mutually dependent relationships between or among categorized key words;

grouping the identified mutually dependent relationships into groups of related mutually dependent relationships; and

extracting the key words with mutually dependent relationships in the same sentence as labeled data with concepts, wherein the step of extracting key words comprises using a mutually dependent relationship extraction rule comprising a string of categories of arbitrary length to be extracted;

searching for unique concepts, a unique concept being a concept whose statistical characteristic is distinguished beyond a threshold with the set to which it belongs;

creating and keeping statistical information;

visually displaying the statistical information; and

presenting a distribution of differences of the unique concepts.

2. The method of claim 1 wherein the categorized key words identified as being in a mutually dependent relationship exist within separate clauses.

3. The method of claim 2 wherein the separate clauses exist within separate sentences.

4. The method of claim 1 wherein the converting step comprises conversion of the received collection of data into the same format and wherein the received collection of data comprises various data formats.

5. The method of claim 1 wherein the assigning of the category to each key word comprises searching the category dictionary to identify the category which matches the key word.

6. The method of claim 5 wherein the category dictionary comprises combinations of original expressions, parts of speech, concepts and categories, wherein the original expressions are equivalent to the key words, the parts of speech are a classification of the key words, the concepts are replacement expressions for the key words and the categories represent a larger group having the nature of the key word.

7. The method of claim 1 wherein the mutually dependent relationships are determined according to rules applied to key words and their associated categories within a clause.

8. The method of claim 1 wherein the mutually dependent relationship extraction rule are provided manually by a user for each of the identified groups of mutually dependent relationships in response to the user being presented a display of all the groups of mutually dependent relationships, at a GUI.

9. The method of claim 1 further comprising presenting the identified groups in which the mutually dependent relationship extraction rule applies to a user at a GUI.

10. The method of claim 1 further comprising comparing the identified groups in which the mutually dependent relationship extraction rule applies against a set of rules to determine a course of action, wherein the course of action comprises automatically notifying a user that the mutually dependent relationship extraction rule applies.

11. The method of claim 1 wherein the mutually dependent relationship extraction rule is predefined.

12. The method of claim 1 wherein the mutually dependent relationship extraction rule is entered manually by a user during a request by the user for specific relationships between categories, whereby mutually dependent relationships by categories are defined by the user according to the user'"'"'s needs.

18. The method of claim 1 wherein the data is inquiry data provided by customers.

13. An article of manufacture, embodying logic to perform a method of mining a collection of data, comprising:

a user interface for;

receiving a collection of data, the collection of data comprising key words, wherein the key words comprise coherent character strings;

visually displaying statistical information; and

presenting a distribution of differences of unique concepts; and

a processor embodying logic for;

converting the collection of data into labeled data by grouping various types of data into a same format and assigning a label indicating a category of item contents, such that the labeled data is in analyzable condition for concept extraction, and wherein the labeled data comprises the label and a clause comprising the item contents;

assigning a category to the key words, wherein the category references a concept so that the key words can be handled as concepts with a meaning;

separating the clauses into pairs comprising an independent word and an attached word;

assigning categories to the separated clauses using syntactic patterns and a category dictionary;

generating, by syntactic analysis, a syntactic tree of a sentence comprising the separated clauses;

receiving a syntactically analyzed sentence as input, identifying mutually dependent relationships between or among the categorized words, within each of the clauses;

grouping the identified mutually dependent relationships into groups of related mutually dependent relationships;

extracting the key words with mutually dependent relationships in the same sentence as labeled data with concepts, wherein the step of extracting key words comprises using a mutually dependent relationship extraction rule comprising a string of categories of arbitrary length to be extracted; and

searching for the unique concepts, the unique concept being a concept whose statistical characteristic is distinguished beyond a threshold with the set to which it belongs;

creating and keeping the statistical information.

14. The article of manufacture of claim 13 wherein converting comprises conversion of the received collection of data into the same format, wherein the received collection of data is made up of various data formats.

15. The article of manufacture of claim 13 wherein the associating of a category with each key word comprises searching a category dictionary to identify a category which matches the key word.

16. The article of manufacture of claim 15 wherein the category dictionary comprises combinations of original expressions, parts of speech, concepts and categories, wherein the original expressions are equivalent to the key words, the parts of speech are a classification of the key words, the concepts are replacement expressions for the key words and the categories represent a larger group having the nature of the key word.

17. The article of manufacture of claim 13 wherein the mutually dependent relationships are determined according to rules applied to key words and their associated categories within a clause.

19. The article of manufacture of claim 13 wherein the data is inquiry data provided by customers.