KEYWORD CLASSIFICATION AND DETERMINATION IN LANGUAGE MODELLING

• US 20100325109A1
• Filed: 02/09/2007
• Published: 12/23/2010
• Est. Priority Date: 02/09/2007
• Status: Abandoned Application

• Current Assignee: Agency For Science Technology and Research
• Original Assignee: Agency For Science Technology and Research
• Inventors: Li, Haizhou, Bai, Shuanhu
• Application Number:
  US12/526,500
• Publication Number:
  US 20100325109A1
• Time in Patent Office: Days
• US Class Current: 707/737
• CPC Class Codes: G06F 16/3347 using vector based model
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A computer-implemented method and apparatus defines a keyword class vector. A set of seed keywords is determined from a set of keywords and first and second most similar keywords from the set of seed keywords are then determined. A class vector is determined from first and second keyword vectors associated with the first and second most similar keywords. The method and apparatus also classifies a keyword in a keyword class. A similarity for a keyword vector associated with the keyword is determined with reference to a plurality of class vectors, each class vector having an associated class and determines a most similar class vector of the plurality of class vectors from the similarity determination. The keyword is then classified in a most similar class associated with the most similar class vector.

Claims

1. A computer-implemented method for defining a keyword class vector, comprising:

determining a set of seed keywords from a set of keywords;

determining first and second most similar keywords from the set of seed keywords; and

determining a class vector from first and second keyword vectors associated with the first and second most similar keywords.

2. The method of claim 1, wherein determining the class vector comprises merging the first and second keyword vectors.

3. The method of claim 1, wherein the method comprises determining first and second most similar keywords by determining, for each of the set of seed keywords, a measure of similarity for a keyword vector associated with a seed keyword with keyword vectors associated with the other keywords of the set of seed keywords, and determining first and second keyword vectors which are most similar to one another.

4. The method of claim 1, wherein the method comprises determining the set of seed keywords from a word count of each of the set of keywords in a set of reference documents and adding a keyword to the set of seed keywords when the word count for that keyword satisfies a threshold criterion.

5. The method of claim 4, wherein the method comprises determining the word count from a count of an element in a keyword vector associated with the keyword, the element representing a number of occurrences of the keyword in a reference document.

6. The method of claim 4, further comprising allowing a user to refine the set of seed keywords.

7. The method of claim 6, wherein allowing a user to refine the set of seed keywords comprises allowing the user to remove selected keywords from the set of seed keywords.

8. The method of claim 4, wherein the method comprises calculating a threshold value as an average of keyword word counts, the threshold criterion being that the word count for that keyword is greater than the threshold value.

9. The method of claim 1, further comprising allowing a user to define a number of classes and/or class vectors for the classification.

10. The method of claim 1, the method being further for classifying a keyword in a keyword class and comprising:

determining a similarity for a keyword vector associated with the keyword with reference to a plurality of class vectors, each class vector having an associated class;

determining a most similar class vector of the plurality of class vectors from the similarity determination; and

classifying the keyword in a most similar class associated with the most similar class vector.

11. A computer-implemented method for classifying a keyword in a keyword class, the method comprising:

determining a similarity for a keyword vector associated with the keyword with reference to a plurality of class vectors, each class vector having an associated class;

determining a most similar class vector of the plurality of class vectors from the similarity determination; and

classifying the keyword in a most similar class associated with the most similar class vector.

12. The method of claim 11, wherein the method comprises performing the similarity determination by calculating similarity scores for the keyword vector with reference to the plurality of class vectors.

13. The method of claim 11, wherein the keyword vector comprises an element identifying a number of occurrences of the keyword in a reference document, the method further comprising assigning the reference document to a most similar class document corpus when the number of occurrences is non-zero.

14. The method of claim 11, wherein the method comprises classifying the keyword in the most similar class from a merger of the keyword vector with the most similar class vector.

15. The method of claim 11, wherein the method comprises determining the similarity scores from a measure of an angular separation in vector space of elements of the keyword vector and the class vectors.

16. A computer-implemented method for determining a keyword in a set of words, the method comprising:

assigning a distance parameter for a first word in the word set, the distance parameter designating a first word distance from the word set;

parsing a document for an occurrence of the first word in the document;

upon identification of an occurrence of the first word in the document, modifying the distance parameter; and

upon determination the modified distance parameter satisfies a threshold criterion, designating the word as a keyword.

17. The method of claim 16, further comprising, upon identification of an occurrence of the first word in the document, modifying the distance parameter in dependence of a position of the first word in the document.

18. The method of claim 16, further comprising, upon identification of an occurrence of the first word in the document, extracting a text string from the document in which the first word occurs and modifying the distance parameter in dependence of a position of the first word in the document comprises modifying the distance in dependence of a position of the word in the text string.

19. The method of claim 16, the method being executed for a plurality of words and comprising determining a plurality of modified distance parameters for the plurality of words and designating a subset of the plurality of words satisfying the threshold criterion as keywords.

20. The method of claim 19, wherein the threshold criterion to be determined comprises a determination of a plurality of keywords with modified distance parameters designating the least distance from the word set.

21. Apparatus for defining a keyword class vector, the apparatus being configured to:

determine a set of seed keywords from a set of keywords;

determine first and second most similar keywords from the set of seed keywords; and

determine a class vector from first and second keyword vectors associated with the first and second most similar keywords.

22. Apparatus for classifying a keyword in a keyword class, the apparatus being configured to:

determine a similarity for a keyword vector associated with the keyword with reference to a plurality of class vectors, each class vector having an associated class;

determine a most similar class vector of the plurality of class vectors from the similarity determination; and

classifying the keyword in a most similar class associated with the most similar class vector.

23. Apparatus for determining a keyword in a set of words, the apparatus being configured to:

assign a distance parameter for a first word in the word set, the distance parameter designating a first word distance from the word set;

parse a document for an occurrence of the first word in the document;

upon identification of an occurrence of the first word in the document, modify the distance parameter; and

upon determination the modified distance parameter satisfies a threshold criterion, designate the word as a keyword.

24. (canceled)

25. A computer program product having computer code stored thereon for defining a keyword class, the computer code being configured to:

determine a set of seed keywords from a set of keywords;

determine first and second most similar keywords from the set of seed keywords; and

determine a class vector from first and second keyword vectors associated with the first and second most similar keywords.

26. A computer program product having computer code stored thereon for classifying a keyword in a keyword class, the computer code being configured to:

determine a similarity for a keyword vector associated with the keyword with reference to a plurality of class vectors, each class vector having an associated class;

determine a most similar class vector of the plurality of class vectors from the similarity determination; and

classifying the keyword in a most similar class associated with the most similar class vector.

27. A computer program product having computer code stored thereon for classifying a keyword in a keyword class, the computer code being configured to:

assign a distance parameter for a first word in the word set, the distance parameter designating a first word distance from the word set;

parse a document for an occurrence of the first word in the document;

upon identification of an occurrence of the first word in the document, modify the distance parameter; and

upon determination the modified distance parameter satisfies a threshold criterion, designate the word as a keyword.

28. (canceled)