Method for calculating relevance between words based on document set and system for executing the method

• US 20080140648A1
• Filed: 12/10/2007
• Published: 06/12/2008
• Est. Priority Date: 12/12/2006
• Status: Active Grant

• Current Assignee: Naver Corporation
• Original Assignee: Naver Business Platform Corporation (Naver Corporation)
• Inventors: Kim, Byoung Hak, Song, Ki Ho, Kim, Min Uk, Kwak, Tae Yeong
• Granted Patent: US 8,407,233 B2
• Time in Patent Office: Days
• US Class Current: 707/5
• CPC Class Codes: G06F 16/319 Inverted lists
• CPC Class Codes: G06F 16/3346 using probabilistic model

Abstract

A method and system for calculating a relevance between words using a document set is provided. The method of calculating the relevance between words based on a document set, includes: obtaining statistical information about the words based on at least one of the words, documents, a word classification of the words, and a document classification of the documents, wherein the words and the documents are included in the document set; standardizing the statistical information; and calculating the relevance between the words based on the standardized statistical information.

Claims

1. A method of calculating a relevance between words based on a document set, the method comprising:

obtaining statistical information about the words based on at least one of the words, documents, a word classification of the words, and a document classification of the documents, wherein the words and the documents are included in the document set;

standardizing the statistical information; and

calculating the relevance between the words based on the standardized statistical information.

2. The method of claim 1, wherein the obtaining comprises:

measuring an appearance frequency for each word, wherein the appearance frequency is a number of times that the word appears;

measuring a crossing frequency for each word, wherein the crossing frequency is a number of documents that include both the word and another word;

generating frequency information based on at least one of the appearance frequency and the crossing frequency; and

obtaining as the statistical information at least one of the appearance frequency, the crossing frequency, and the frequency information.

3. The method of claim 2, wherein the generating of the frequency information generates as the frequency information at least one of a word-document classification appearance frequency, a document frequency, a document classification frequency, a word-word classification crossing frequency, and a word-word combination frequency, based on the appearance frequency or the crossing frequency.

4. The method of claim 3, wherein the word-document classification appearance frequency for each word is a number of times that the word is included in a document included in the document classification.

5. The method of claim 3, wherein the document frequency for each word is a number of documents that include the word, and the document classification frequency for each word is a number of word classifications that include the word.

6. The method of claim 3, wherein the word-word classification crossing frequency for each word is a number of documents that are used together with a word included in the word classification.

7. The method of claim 3, wherein the word-word combination frequency for each word pair is a number of documents that include a pair of words, a number of documents that do not include the pair of words, and a number of documents where the pair of words appear separately.

8. The method of claim 1, wherein the standardizing comprises:

setting, as a column for a multi-dimensional vector set, at least one of the words, the documents, the word classification, and the document classification, wherein each vector includes each independent dimension in the multi-dimensional space; and

setting, as a row of the multi-dimensional vector set, the word corresponding to a vector of the multi-dimensional space.

9. The method of claim 1, wherein the standardizing comprises:

setting, as a column of a real-number-typed random variable set, at least one of the words, the documents, the word classification, and the document classification; and

setting, as a row of a real-number-typed random variable set, the word corresponding to a random variable wherein the random variable has a real number value, and the random variable has, as a random variable value, a frequency value corresponding to the column and the row.

10. The method of claim 1, wherein the standardizing generates a combination probability distribution of a random variable corresponding to a word pair and standardizes the statistical information based on a word-word combination frequency, wherein the word-word combination frequency for each word pair is a number of documents that include a pair of words, a number of documents that do not include the pair of words, and a number of documents where the pair of words appear separately, and

the random variable is defined in a point space of columns and rows that include only appearance or non-appearance points of the word.

11. The method of claim 1, wherein the standardizing comprises:

setting the word as a column of a discrete random variable set to indicate each independent point in a point space where a random variable is defined; and

setting the word as a row of the discrete random variable set to indicate a discrete random variable,

wherein the discrete random variable is generated based on a crossing frequency that is a number of documents that include both the word and another word.

12. The method of claim 1, wherein the calculating calculates the relevance between the words based on a distance or an angle between vectors that are included in a multi-dimensional vector set, wherein the multi-dimensional vector set is generated based on the statistical information.

13. The method of claim 1, wherein the calculating calculates the relevance between the words based on a statistical correlation between random variables, wherein the random variables are included in a real-number-typed random variable set that is generated based on the statistical information.

14. The method of claim 1, wherein the calculating calculates the relevance between the words based on information about two random variables of a combination probability distribution that is generated based on the statistical information.

15. The method of claim 1, wherein the calculating calculates the relevance between the words based on a document frequency and a crossing frequency wherein the document frequency is a number of documents that include the word, and the crossing frequency is a number of documents that includes both the word and another word.

16. A method of calculating a relevance between words based on a document set, the method comprising:

obtaining statistical information about the words based on at least one of the words, documents, a word classification of the words, and a document classification of the documents, wherein the words and the documents are included in the document set;

standardizing the statistical information as a multi-dimensional vector set; and

calculating the relevance between the words based on the multi-dimensional vector set.

17. The method of claim 16, wherein the calculating calculates, as the relevance between the words, at least one of an inverse L_P, a cosine coefficient, an extended Dice coefficient, an extended Jaccard'"'"'s coefficient, and a correlation with respect to two vectors a and b of the multi-dimensional vector set.

18. A method of calculating a relevance between words based on a document set, the method comprising:

obtaining statistical information about the words based on at least one of the words, documents, a word classification of the words, and a document classification of the documents, wherein the words and the documents are included in the document set;

standardizing the statistical information as a real-number-typed random variable set; and

calculating the relevance between the words based on the real-number-typed random variable set.

19. The method of claim 18, wherein the calculating calculates, as the relevance between the words, at least one of a Pearson'"'"'s correlation coefficient and a coherence with respect to two random variables X and Y of the real-number-typed random variable set.

20. A system for calculating a relevance between words based on a document set, the system comprising:

a statistical information obtainment unit operable to obtain statistical information about the words based on at least one of the words, documents, a word classification of the words, and a document classification of the documents, wherein the words and the documents are included in the document set;

a standardization unit operable to standardize the statistical information; and

a relevance calculation unit operable to calculate the relevance between the words based on the standardized statistical information.

21. The system of claim 20, wherein the statistical information obtainment unit comprises:

an appearance frequency measurement unit operable to measure an appearance frequency for each word, wherein the frequency is a number of times that the word appears;

a crossing frequency measurement unit operable to measure a crossing frequency for each word, wherein the crossing frequency is a number of documents that include both the word and another word;

a frequency information generation unit operable to generate frequency information based on at least one of the appearance frequency and the crossing frequency; and

an obtainment unit operable to obtain as the statistical information at least one of the appearance frequency, the crossing frequency, and the frequency information.

22. The system of claim 20, wherein the standardization unit standardizes the statistical information as at least one of a multi-dimensional vector set, a real-number-typed random variable set, a combination probability distribution of random variables, and a discrete random variable set.

23. The system of claim 20, wherein the relevance calculation unit calculates, as the relevance between the words, at least one of an inverse L_P, a cosine coefficient, an extended Dice coefficient, an extended Jaccard'"'"'s coefficient, and a correlation with respect to two vectors a and b of the multi-dimensional vector set.

24. The system of claim 20, wherein the relevance calculation unit calculates, as the relevance between the words, at least one of a Pearson'"'"'s correlation coefficient and a coherence with respect to two random variables of the real-number-typed random variable set.

25. The system of claim 20, wherein the relevance calculation unit calculates, as the relevance between the words, at least one of a Jaccard'"'"'s coefficient, a Dice coefficient, an odds-ratio, mutual information, and point-wise mutual information with respect to two random variables A and B by using the combination probability distribution.