Contents filter based on the comparison between similarity of content character and correlation of subject matter

• US 20040243537A1
• Filed: 03/05/2004
• Published: 12/02/2004
• Est. Priority Date: 09/07/2001
• Status: Active Grant

• Current Assignee: Legend Ltd.
• Original Assignee: Legend Ltd.
• Inventors: Zhu, Guang, Wang, Nan, Wang, Jiang, Gao, Jianzhong, Xiao, Hang
• Granted Patent: US 7,617,090 B2
• Time in Patent Office: Days
• US Class Current: 707/1
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation

Abstract

A contents filter based on similarity of content character and correlation of subject matter includes a filtering system and a disciplining system, and the contents filter isn'"'"'t a filtering system used for a special subject matter but a general subject matter, the filtering contents can be obtained by leaning of the disciplining system, the filtering system and the disciplining system are installed of physical separation, the filtering system communicates with the disciplining system through the data interface, the filtering system can be installed in the input device of network information. To achieve the different filtering effect the different filtering character obtained by the disciplining system are set to the filtering system located in the different input device of network information. The present invention implements filtering through analyzing and determining text contents, and offers an intelligent and effective service of contents safety for user. The use of the filter is in great agility. Furthermore the filter can identify the contents character to be filtered according to the character with disciplined class provided by user. The processing speed is fast and the filter can be conveniently installed.

Claims

1. A contents filter based on similarity of content character and correlation of subject matter, which is characterized that the contents filter includes a filtering system and a disciplining system at least;

the disciplining system learns with the appointed information to obtain the filtering character of the information;

the filtering system filters the information, and the disciplining system communicates with the filtering system.

2. The contents filter based on similarity of content character and correlation of subject matter according to claim 1, is characterized that the said contents filter includes one disciplining system and one or more filtering systems at least.

3. The contents filter based on similarity of content character and correlation of subject matter according to claim 1, is characterized that the said contents filter includes one filtering system and one or more disciplining systems at least.

4. The contents filter based on similarity of content character and correlation of subject matter according to claim 1, is characterized that the said contents filter includes more filtering systems and more disciplining systems.

5. The contents filter based on similarity of content character and correlation of subject matter according to claim 1, is characterized that the filtering system and the disciplining system are installed separately, and the filtering system communicates with the disciplining system through the data interface.

6. The contents filter based on similarity of content character and correlation of subject matter according to claim 5, is characterized that the said separate installation means separate installation in physics.

7. The contents filter based on similarity of content character and correlation of subject matter according to claim 6, is characterized that the filtering system can be installed in an input device of network information.

8. The contents filter based on similarity of content character and correlation of subject matter according to claim 7, is characterized that the different filtering characters obtained by the disciplining system are configured to the filtering systems located in the different input devices of network information.

9. The contents filter based on similarity of content character and correlation of subject matter according to claim 8, is characterized that the said configuration is to distribute the filtering character of the filtering system according to the burden capacity, the location and the purpose of the input device of network information in network.

10. The contents filter based on similarity of content character and correlation of subject matter according to claim 7, is characterized that the said input device of network information is firewall.

11. The contents filter based on similarity of content character and correlation of subject matter according to claim 8, is characterized that the said input device of network information is mail server.

12. The contents filter based on similarity of content character and correlation of subject matter according to claim 8, is characterized that the said input device of network information is proxy server.

13. The contents filter based on similarity of content character and correlation of subject matter according to claim 8, is characterized that the said input device of network information is personal computer.

14. The contents filter based on similarity of content character and correlation of subject matter according to claim 7, is characterized that the said input device of network information is one input device of network information or more input devices of network information or the combination of any type of input device of network information.

15. The contents filter based on similarity of content character and correlation of subject matter according to claim 1, is characterized that the said disciplining system includes a module of classifying character vocabulary for contents filtering, which is used to construct a classifying character vocabulary learned from the special information, and conduct the supplement or update of the filtering system.

16. The contents filter based on similarity of content character and correlation of subject matter according to claim 1, is characterized that the said disciplining system includes an anti-interference extracting module of text character for contents filtering;

which is used to examine and obtain the interferential text in the checked information, with which instructing the text filtering of the filtering system.

17. The contents filter based on similarity of content character and correlation of subject matter according to claim 1, is characterized that the said disciplining system still includes an anti-interference extracting module of text subject matter.

18. The contents filter based on similarity of content character and correlation of subject matter according to claim 17, is characterized that the procedure of the said anti-interference extracting module of text subject matter extracting the anti-interference subject matter words includes the steps as follows:

Step 1;

The anti-interference extracting module of text subject matter examines the specified character in the checked text, to determine whether the sequence of the specified character accords with the sequence of the character in the preset subject matter words, i.e. finding the specified character string;

Step 2;

The anti-interference extracting module of text subject matter determines the interferential distance, if the interferential distance is less than the preset threshold distance, the string is considered as the interferential subject matter words to be selected;

Step 3;

While the anti-interference extracting module of text subject matter concludes that the frequency of appearance of the mentioned subject matter words is beyond the preset threshold value, the subject matter words to be selected are set to the key words of the filter.

19. The contents filter based on similarity of content character and correlation of subject matter according to claim 18, is characterized that the said anti-interference extracting module of text subject matter examining the specified character still includes finding whether the specified characters contain Chinese punctuation among them, if the specified characters don'"'"'t contain Chinese punctuations, the character string is considered as the interferential subject matter words, and the anti-interference extracting module of text subject matter sets the character string as the key words of the filter.

20. The contents filter based on similarity of content character and correlation of subject matter according to claim 18, is characterized that the said step 1 can be directly implemented as follows:

the anti-interference extracting module of text subject matter examines the specified character string between two adjacent punctuations.

21. The contents filter based on similarity of content character and correlation of subject matter according to claim 18, is characterized that the said frequency of appearance of the interferential subject matter word to be selected can be summation of the interferential subject matter words of more different types.

22. The contents filter based on similarity of content character and correlation of subject matter according to claim 18, is characterized that the said anti-interference extracting module of text subject matter is used to extract the information relating to text subject matter, and rectify the extracted information, then rectify the similarity of text based on the Vector Space Model with the rectified result of the subject matter information.

23. The contents filter based on similarity of content character and correlation of subject matter according to claim 22, is characterized that rectifying the similarity of text based on the vector space according to the rectified result of the subject matter information includes the steps as follows:

Step1;

The anti-interference extracting module of text subject matter extracts the information relating to subject matter of text;

Setp2;

The anti-interference extracting module of text subject matter rectifies the result of similarity of text based on the Vector Space Model.

24. The contents filter based on similarity of content character and correlation of subject matter according to claim 23, is characterized that before the said step2, the steps also include:

the extracted information relating to subject matter is rectified, then the similarity of text based on the Vector Space Model is rectified.

25. The contents filter based on similarity of content character and correlation of subject matter according to claim 23, is characterized that the information relating to text subject matter extracted by the said anti-interference extracting module of text subject matter in step1 is the frequency of word, the concentration of frequency, the information of word length, the words, and the total words amount;

and the information relating to subject matter is the information relating to subject matter with top weight after weighting.

26. The contents filter based on similarity of content character and correlation of subject matter according to claim 25, is characterized that the extracting of the information relating to subject matter by the anti-interference extracting module of text subject matter is performed with the formula as follows:

27. The contents filter based on similarity of content character and correlation of subject matter according to claim 25, is characterized that the anti-interference extracting module of text subject matter rectifying the extracted information relating to subject matter is that the similarity of contents is determined by the degree of overlapping of subject matter information.

28. The contents filter based on similarity of content character and correlation of subject matter according to claim 24, is characterized that the rectification of the similarity of text based on the Vector Space Model is as follows:

if the degree of overlapping is more than the threshold value, the value of eigenvector similarity will be strengthened, and if the degree of overlapping is less than the threshold value, the value of eigenvector similarity will be weakened.

29. The contents filter based on similarity of content character and correlation of subject matter according to claim 24, is characterized that the rectification of the information relating to subject matter is performed with the following formula:

30. The contents filter based on similarity of content character and correlation of subject matter according to claim 28, is characterized that the rectification of the similarity of text based on the Vector Space Model is:

Sim(W_i,V_j)×

R_is, in which, Sim(W_i,V_j) is the similarity of text based on the Vector Space Model, R_is is a correlation coefficient of the subject matter word.

31. The contents filter based on similarity of content character and correlation of subject matter according to claim 23, is characterized that the said information relating to subject matter is the subject matter words or the character words.

32. The contents filter based on similarity of content character and correlation of subject matter according to claim 1, is characterized that the said disciplining system still includes an evaluation and instruction module of disciplining effect.

33. The contents filter based on similarity of content character and correlation of subject matter according to claim 32, is characterized that the said evaluation and instruction module of disciplining effect is used to obtain the coefficients of the evaluation of the character words amount, the evaluation of the rate of repeat and the evaluation of the degree of subject matter centralization, then according to these coefficients, the result of disciplining effect is educed to give an objective and quantitative instruction to disciplining.

34. The contents filter based on similarity of content character and correlation of subject matter according to claim 32, is characterized that the evaluation of the character words amount is executed according to the following formula:

35. The contents filter based on similarity of content character and correlation of subject matter according to claim 33, is characterized that the evaluation of the rate of repeat is executed according to the following formula:

36. The contents filter based on similarity of content character and correlation of subject matter according to claim 33, is characterized that the evaluation of the degree of subject matter centralization is executed according to the following formula:

37. The contents filter based on similarity of content character and correlation of subject matter according to claim 34, is characterized that the evaluation of disciplining effect is executed according to the following formula:

Q=Q₁*Q₂*Q₃ or Q=Q₁*Q₂ or Q=Q₁*Q₃ or Q=Q₁ or Q=Q₂ or Q=Q₃,

then according to the value of Q, the grade of disciplining effect is determined.

38. The contents filter based on similarity of content character and correlation of subject matter according to claim 1, is characterized that the said filtering system includes a module of classifying character vocabulary for contents filtering, an anti-interference exacting module of text character, and a module of calculating similarity between text contents to be filtered and defined filtering contents.

39. The contents filter based on similarity of content character and correlation of subject matter according to claim 1, is characterized that the said filtering system includes a module of rectifying local similarity and short text similarity with precision.

40. The contents filter based on similarity of content character and correlation of subject matter according to claim 39, is characterized that the said module of rectifying local similarity and short text similarity with precision is used to obtain precision of relegation of standard class which text to be analyzed belongs to according to the standard vector of text to be analyzed, and use the said precision to rectify the result of the similarity of text based on the Vector Space Model.

41. The contents filter based on similarity of content character and correlation of subject matter according to claims 40, is characterized that the said rectification can be Sim(W_i,V_j)×

P_i, in which P_i stands for the rectifying coefficient of precision, Sim(W_i,V_j) is the similarity of text based on the Vector Space Model.

42. The contents filter based on similarity of content character and correlation of subject matter according to claim 41, is characterized that the rectifying coefficient of precision can be obtained through the following formula:

43. The contents filter based on similarity of content character and correlation of subject matter according to claim 1, is characterized that the said filtering system includes a filtering module according to multi-step rectified degree of similarity.

44. The contents filter based on similarity of content character and correlation of subject matter according to claim 43, is characterized that the said filtering module according to multi-step rectified degree of similarity is used to gather the coefficients of precision obtained by each module, with the preset filtering threshold value U_w to determine whether the text to be filtered should be filtered.