INFORMATION FILTERING SYSTEM, INFORMATION FILTERING METHOD AND INFORMATION FILTERING PROGRAM

• US 20100280981A1
• Filed: 01/08/2008
• Published: 11/04/2010
• Est. Priority Date: 01/08/2008
• Status: Active Grant

• Current Assignee: Mitsubishi Electric Corporation
• Original Assignee: Mitsubishi Electric Corporation
• Inventors: Kato, Mamoru, Kori, Mitsunori
• Granted Patent: US 8,442,926 B2
• Time in Patent Office: Days
• US Class Current: 706/20
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: G06F 16/90339 by using parallel associative memories or content-addressable memories

Abstract

A string matching unit 110 specifies a category of an input document 801 by string matching of the input document 801 and a classifying keyword shown by matching condition information 109. Learning data 209 shows statistic information of each category. A classifying unit 220 specifies the category of the input document 801 based on a correspondence ratio of the input document 801 and the statistic information shown by the learning data 209. A correct answer comparing unit 120 compares the category specified by the string matching unit 110 and a category of correct answer information 803. A learning unit 210 compares the category specified by the classifying unit 220 and the category of the correct answer information 803. An error rate calculating unit 310 calculates a classifying error rate of a string matching filter unit 100 and a learning filter unit 200 based on the comparison result of the correct answer comparing unit 120 and the comparison result of the learning unit 210. A result outputting unit 320 outputs the category specified by the filter having a smaller classifying error rate as a classified result 301 of a classifying target document 804.

Claims

1. An information filtering system comprising:

a first filter unit inputting document data belonging to at least any kind among a plurality of kinds from an input equipment, carrying out a predetermined classifying process to specify a kind to which the document data inputted belongs using a CPU (Central Processing Unit), and specifying the kind to which the document data inputted belongs as first classified information;

a second filter unit inputting the document data from an input equipment, carrying out a predetermined classifying process being different from the classifying process of the first filter unit using a CPU, specifying a kind to which the document data inputted belongs as second classified information;

a first correct answer comparing unit comparing the first classified information of a plurality of pieces of learning document data specified by the first filter unit with treating each of the plurality of pieces of learning document data which belongs to a kind previously specified as the document data and correct answer information of the plurality of pieces of learning document data which belongs to the kind previously specified using a CPU, generating first learning result information of the plurality of pieces of learning document data showing whether the first classified information matches the correct answer information or not based on comparison result, and storing the first learning result information generated of the plurality of pieces of learning document data in a memory equipment;

a second correct answer comparing unit comparing the second classified information of the plurality of pieces of learning document data specified by the second filter unit with treating each of the plurality of pieces of learning document data as the document data and the correct answer information of the plurality of pieces of learning document data using a CPU, generating second learning result information of the plurality of pieces of learning document data showing whether the second classified information matches the correct answer information based on comparison result, and storing the second learning result information generated of the plurality of pieces of learning document data in a memory equipment;

an error rate calculating unit calculating a first error rate showing a rate that the first classified information does not match the correct answer information based on the first learning result information of the plurality of pieces of learning document data generated by the first correct answer comparing unit using a CPU, and as well calculating a second error rate showing a rate that the second classified information does not match the correct answer information based on the second learning result information of the plurality of pieces of learning document data generated by the second correct answer comparing unit using a CPU; and

a result outputting unit specifying a kind to which the classifying target document data belongs using a CPU based on the first classified information specified by the first filter unit with treating classifying target document data which is a target to be classified to a specific kind as the document data, the second classified information specified by the second filter unit with treating the classifying target document data as the document data, the first error rate calculated by the error rate calculating unit, and the second error rate calculated by the error rate calculating unit, and outputting the kind specified to an output equipment as a classified result.

2. The information filtering system of claim 1,

wherein the first filter unit is a learning filter unit including:

a learning unit inputting a plurality of sets of the learning document data and the correct answer information from an input equipment, carrying out a statistic process to generate statistic information being common to each learning document data belonging to each kind as learning data based on the plurality of sets using a CPU, and storing the learning data generated by the statistic process in a memory equipment; and

a learning classifying unit inputting the classifying target document data from an input equipment, calculating a ratio of correspondence of the classifying target document data to the statistic information shown by the learning data generated by the learning unit for each kind as classification probability using a CPU, and specifying a kind to which the classifying target document data belongs based on the classification probability for each kind.

3. The information filtering system of claim 2,

wherein the learning unit carries out the statistic process using either one of a Bayesian theory, an SVM (Support Vector Machine), and a neural network model.

4. The information filtering system of claim 3,

wherein the second filter unit is a string matching filter unit inputting the document data from an input equipment, inputting matching condition information from a memory equipment which previously stores a classifying keyword belonging to each kind as the matching condition information, carrying out a string matching process using a CPU to discriminate whether the classifying keyword of each kind shown by the matching condition information is included in the document data or not, and specifying the kind, to which the classifying keyword which is discriminated to be included in the document data by the string matching process belongs, as the kind to which the document data belongs.

5. The information filtering system of claim 4,

wherein the string matching filter unit carries out the string matching process using a deterministic finite automaton.

6. The information filtering system of claim 5,

wherein the string matching filter unit carries out the string matching process using a state transition table of the deterministic finite automaton showing the classifying keyword as the matching condition information.

7. The information filtering system of claim 4,

wherein the string matching filter unit carries out the string matching process using a nondeterministic finite automaton.

8. The information filtering system of claim 4,

wherein the result outputting unit, when the first error rate is smaller than the second error rate, outputs the first classified information as the classified result, and when the second error rate is smaller than the first error rate, outputs the second classified information as the classified result.

9. The information filtering system of claim 4,

wherein the result outputting unit expresses the first classified information and the second classified information with predetermined numeral values, weights the first classified information according to the first error rate, as well weights the second classified information according to the second error rate, calculates an average value of the first classified information weighted and the second classified information weighted as a weighted average value, compares the weighted average value calculated and a predetermined threshold value, and outputs the classified result based on comparison result.

10. The information filtering system of claim 4,

wherein the matching condition information shows a specific classifying keyword among the plurality of classifying keywords as a specific keyword, and

wherein the result outputting unit outputs a kind to which the specific keyword belongs as the classified result of the classifying target document data which is discriminated to include the specific keyword by the string matching filter unit.

11. The information filtering system of claim 4,

wherein the error rate calculating unit calculates the first error rate based on the first learning result information generated within a predetermined period, and as well calculates the second error rate based on the second learning result information generated within the predetermined period.

12. The information filtering system of claim 2,

wherein the learning filter unit includes a first learning filter unit and a second learning filter unit,

wherein the first learning filter unit and the second learning filter unit respectively include the learning unit and the learning classifying unit,

wherein the learning unit of the first learning filter unit and the learning unit of the second learning filter unit respectively carry out different statistic processes, and

wherein the first classified information shows first learning classified information showing the kind specified by the learning classifying unit of the first learning filter unit and second learning classified information showing the kind specified by the learning classifying unit of the second learning filter unit.

13. The information filtering system of claim 12,

wherein the result outputting unit, when the first classified information is outputted as the classified result, obtains a logical sum of the kind shown by the first learning classified information and the kind shown by the second learning classified information, and outputs the kind shown by the logical sum obtained as the classified result.

14. The information filtering system of claim 12,

wherein the result outputting unit, when the first classified information is outputted as the classified result, outputs the classified result based on the classification probability of the kind shown by the first learning classified information and the classification probability of the kind shown by the second learning classified information.

15. The information filtering system of claim 12,

wherein the result outputting unit, when the first classified information is outputted as the classified result, weights the first learning classified information according to the first error rate of the first learning classified information, and as well weights the first learning classified information according to the first error rate of the first learning classified information, calculates an average value of the first learning classified information weighted and the second learning classified information weighted as a weighted average value, compares the weighted average value calculated and a predetermined threshold value, and outputs the classified result based on comparison result.

16. The information filtering system of claim 2,

wherein the learning filter unit includes a plurality of sets of the learning unit and the learning classifying unit,

wherein a plurality of the learning units respectively carry out the statistic processes,

wherein the first classified information shows the kind specified by the learning classifying unit of the respective set, and

wherein the result outputting unit, when the first classified information is outputted as the classified result, outputs the kind specified by the largest number of sets as the classified result.

17. An information filtering method comprising:

by a first filter unit, performing a first filter process of inputting document data belonging to at least any kind among a plurality of kinds from an input equipment, carrying out a predetermined classifying process to specify a kind to which the document data inputted belongs using a CPU (Central Processing Unit), and specifying the kind to which the document data inputted belongs as first classified information;

by a second filter unit, performing a second filtering process of inputting the document data from the input equipment, carrying out a predetermined classifying process being different from the classifying process of the first filter unit using a CPU, specifying the kind to which the document data inputted belongs as second classified information;

by a first correct answer comparing unit, performing a first correct answer comparing process of treating each of a plurality of pieces of learning document data which belongs to a kind previously specified as the document data, comparing the first classified information of the plurality of pieces of learning document data specified by the first filter unit and correct answer information of the plurality of pieces of learning document data which belongs to the kind previously specified using a CPU, generating first learning result information of the plurality of pieces of learning document data showing whether the first classified information matches the correct answer information or not based on comparison result, and storing the first learning result information generated of the plurality of pieces of learning document data in a memory equipment;

by a second correct answer comparing unit, performing a second correct answer comparing process of treating each of the plurality of pieces of learning document data as the document data, comparing the second classified information of the plurality of pieces of learning document data specified by the second filter unit and the correct answer information of the plurality of pieces of learning document data using a CPU, generating second learning result information of the plurality of pieces of learning document data showing whether the second classified information matches the correct answer information based on the comparison result, and storing the second learning result information generated of the plurality of pieces of learning document data in the memory equipment;

by an error rate calculating unit, performing an error rate calculating process of calculating a first error rate showing a rate that the first classified information does not match the correct answer information based on the first learning result information of the plurality of pieces of learning document data generated by the first correct answer comparing unit using a CPU, and as well calculating a second error rate showing a rate that the second classified information does not match the correct answer information based on the second learning result information of the plurality of pieces of learning document data generated by the second correct answer comparing unit using a CPU; and

by a result outputting unit, performing a result outputting process of treating classifying target document data which is a target to be classified to a specific kind as the document data, specifying the kind to which the classifying target document data belongs using a CPU based on the first classified information specified by the first filter unit, the second classified information specified by the second filter unit, the first error rate calculated by the error rate calculating unit, and the second error rate calculated by the error rate calculating unit, and outputting the kind specified to an output equipment as a classified result.

18. An information filtering program having a computer execute the information filtering method of claim 17.