Method for extracting multi-word technical terms from text

• US 5,423,032 A
• Filed: 01/03/1992
• Issued: 06/06/1995
• Est. Priority Date: 10/31/1991
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Byrd, Roy J., Justeson, John S., Katz, Slava M.
• Primary Examiner(s): Kriess, Kevin A.
• Assistant Examiner(s): BUTLER, DENNIS
• Application Number:
  US07/816,908
• Time in Patent Office: 1,250 Days
• Field of Search: 395/600, 395/700, 395/934, 395/51, 395/144, 364/943.41, 364/974, 364/282.1, 364/943.42
• US Class Current: 704/1
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: Y10S 706/934 Information retrieval or Information management
• CPC Class Codes: Y10S 707/917 Text
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access

Abstract

A method and apparatus for extracting multi-word technical terms from a text file in a computer system. Word strings are selected from the text that have at least two words, that have at most a specified maximum number of words, that include none of a special set of selected tokens, and that only include selected characters. Word string which occur less than a specified minimum number of times in the text file are deleted. The remaining strings form a set of word strings very likely to be multi-word technical terms. Improvements on the quality of the set of word strings can be accomplished by deleting word strings which do not satisfy certain grammatical constraints.

Claims

1. Programmed computer apparatus for extracting a list of candidate multi-word technical terms from an input text file, a multi-word technical term being a string of at least two words having a particular meaning in some technical field, said apparatus comprising:

means for storing a stoplist of tokens which are assumed to not occur in multi-word technical terms, a token being a word, character or string of characters delimited by blanks and/or punctuation;

means for storing a maximum length parameter specifying a maximum number of tokens in any candidate multi-word technical term to be extracted;

means responsive to the stored stoplist for extracting text fragments from an input text file by identifying delimiting tokens in the input text file, including means for identifying as a delimiting token each token in the input text file which is the same as a token in the stored stoplist, the identified delimiting tokens defining text fragments therebetween;

means for deriving from the extracted text fragments all possible subsequences of tokens having a length of at least two tokens and not more than a maximum number of tokens specified by the stored maximum length parameter;

means for testing each of the derived subsequences against at least one filtering condition; and

means for creating a sublist of the derived subsequences which pass the at least one filtering condition, the created sublist being the list of candidate multi-word technical terms.

2. Programmed computer apparatus as defined in claim 1 and further comprising means for storing a list of allowable characters for candidate multi-word technical terms to be extracted, said means for extracting text fragments including means for identifying as a delimiting token each token in the input text file which contains at least one character not in the stored list of allowable characters.

3. Programmed computer apparatus as defined in claim 2 wherein said means for testing comprises:

means for storing a frequency parameter specifying a minimum frequency of occurrence for a candidate multi-word technical term to be extracted; and

means for using as a filtering condition that a derived subsequences being tested occurs in the input text file with a frequency not less than specified by the stored frequency parameter.

4. Programmed computer apparatus as defined in claim 3 wherein a filtering condition against which each of the derived subsequences is tested by said testing means is a grammatical constraint.

5. Programmed computer apparatus as defined in claim 4 and further comprising a dictionary of words for identifying potential parts of speech for each word in the dictionary, any word not in said dictionary being considered a potential full noun, and wherein said grammatical constraint is that words at particular positions in a derived subsequence being tested potentially are specified parts of speech.

6. Programmed computer apparatus as defined in claim 5 wherein said grammatical constraint is that a derived subsequence being tested satisfies the following conditions:

the last word is a potential full noun;

the first word is a potential full noun or potential full adjective;

any intermediate word in a potential full noun, a potential full adjective or a potential preposition; and

any intermediate word that is a potential preposition and not also either a potential full noun or a potential full adjective is preceded by a word that is a potential full noun.

7. Programmed computer apparatus as defined in claim 1 wherein said means for testing comprises:

a dictionary of words for identifying potential parts of speech for each word in the dictionary, any word not in said dictionary being considered a potential noun; and

means responsive to said dictionary for using as a filtering condition whether a derived subsequence being tested includes a potential noun as a last token.

8. Programmed computer apparatus as defined in claim 1 wherein a filtering condition against which each of the derived subsequences is tested by said testing means is a grammatical constraint.

9. Programmed computer apparatus as defined in claim 8 and further comprising a dictionary of words for identifying potential parts of speech for each word in the dictionary, any word not in said dictionary being considered a potential full noun, and wherein said grammatical constraint is that words at particular positions in a derived subsequence being tested potentially are specified parts of speech.

10. Programmed computer apparatus as defined in claim 9 wherein said grammatical constraint is that a derived subsequence being tested satisfies the following conditions:

the last word is a potential full noun;

the first word is a potential full noun or potential full adjective;

an intermediate word is a potential full noun, a potential full adjective or a potential preposition; and

any intermediate word that is a potential preposition and not also either a potential full noun or a potential full adjective is preceded by a word that is a potential full noun.

11. Programmed computer apparatus for extracting a list of candidate multi-word technical terms from an input text file, a multi-word technical term being a string of at least two words having a particular meaning in some technical field, said apparatus comprising:

means for storing a stoplist of tokens which are assumed to not occur in multi-word technical terms, a token being a word, character or string of characters delimited by blanks and/or punctuation;

means for storing a frequency parameter specifying a minimum frequency of occurrence for a candidate multi-word technical term to be extracted;

means responsive to the stored stoplist for extracting text fragments from an input text file by identifying delimiting tokens in the input text file, including means for identifying as a delimiting token each token in the input text file which is the same as a token in the stored stoplist, the identified delimiting tokens defining text fragments therebetween;

means for deriving from the extracted text fragments each possible subsequence of tokens having a length of at least two tokens and which occurs in the input text file with a frequency not less than specified by the stored frequency parameter;

means for testing each of the derived subsequences against at least one filtering condition; and

means for creating a sublist of the derived subsequences which pass the at least one filtering condition, the created sublist being the list of candidate multi-word technical terms.

12. Programmed computer apparatus as defined in claim 11 and further comprising means for storing a list of allowable characters for candidate multi-word technical terms to be extracted, said means for extracting text fragments including means for identifying as a delimiting token each token in the input text file which contains at least one character not in the stored list of allowable characters.

13. Programmed computer apparatus as defined in claim 11 wherein said means for testing comprises:

means for storing a maximum length parameter specifying a maximum number of token in any candidate multi-word technical term to be extracted; and

means for using as a filtering condition whether a derived subsequence being tested has a length which does not exceed the length specified by the stored maximum length parameter.

14. Programmed computer apparatus as defined in claim 11 wherein a filtering condition against which each of the derived subsequences is tested by said testing means is a grammatical constraint.

15. Programmed computer apparatus as defined in claim 14 and further comprising a dictionary of words for identifying potential parts of speech for each word in the dictionary, any word not in said dictionary being considered a potential full noun, and wherein said grammatical constraint is that words at particular positions in a derived subsequence being tested potentially are specified parts of speech.

16. Programmed computer apparatus as defined in claim 15 wherein said grammatical constraints is that a derived subsequence being tested satisfies the following conditions:

the last word is a potential full noun;

the first word is a potential full noun or potential full adjective;

any intermediate word is a potential full noun, a potential full adjective or a potential preposition; and

any intermediate word that is a potential preposition and not also either a potential full noun or a potential full adjective is preceded by a word that is a potential full noun.

17. A computer implemented method of extracting a list of candidate multi-word technical terms from an input text file, a multi-word technical term being a string of at least two words having a particular meaning in some technical field, said method comprising the computer implemented steps of:

storing a stoplist of tokens which are assumed to not occur in multi-word technical terms, a token being a word, character or string of characters delimited by blanks and/or punctuation;

storing a maximum length parameter specifying a maximum number of token in any candidate multi-word technical term to be extracted;

extracting text fragments from an input text file by identifying delimiting tokens in the input text file at least in part by identifying as a delimiting token each token in the input text file which is the same as a token in the stored stoplist, the identified delimiting tokens defining text fragments therebetween;

deriving from the extracted text fragments all possible subsequences of tokens having a length of at least two tokens and no more than a maximum number of tokens specified by the stored maximum length parameter;

testing each of the derived subsequences against at least one filtering condition; and

creating a sublist of the derived subsequences which pass the at least one filtering condition, the created sublist being the list of candidate multi-word technical terms.

18. A computer implemented method as defined in claim 17 and further comprising the steps of storing a list of allowable characters for candidate multi-word technical terms to be extracted, and in the step of extracting text fragments, identifying as a delimiting token each token in the input text file which contains at least one character not in the stored list of allowable characters.

19. A computer implemented method as defined in claim 18 wherein the testing step further comprises the steps of:

storing a frequency parameter specifying a minimum frequency of occurrence for a candidate multi-word technical term to be extracted; and

using as a filtering condition in the testing step whether a derived subsequence being tested occurs in the input text file with a frequency not less than specified by the stored frequency parameter.

20. A computer implemented method as defined in claim 17 wherein the testing step further comprises the steps of:

storing a frequency parameter specifying a minimum frequency of occurrence for a candidate multi-word technical term to be extracted; and

using as a filtering condition in the testing step whether a derived subsequence being tested occurs in the input text file with a frequency not less than specified by the stored frequency parameter.

21. A computer implemented method as defined in claim 17 wherein the testing step includes the step of testing each of the derived subsequences against a grammatical constraint filtering condition.

22. A computer implemented method as defined in claim 21 and further comprising the step of storing a dictionary of words identifying potential parts of speech for each word in the dictionary, any word not in said dictionary being considered a potential full noun, and wherein the step of testing each derived subsequence against a grammatical constraint includes the step of determining whether words at particular positions in a derived subsequence being tested potentially are specified parts of speech.

23. Apparatus as defined in claim 22 wherein said step of testing each derived subsequence against a grammatical constraint includes the step of determining whether a derived subsequence being tested satisfies the following conditions:

the last word is a potential full noun;

the first word is a potential full noun or potential full adjective;

any intermediate word is a potential full noun, a potential full adjective or a potential preposition; and

any intermediate word that is a potential preposition and not also either a potential full noun or a potential full adjective is preceded by a word that is a potential full noun.

24. A computer implemented method as defined in claim 17 wherein the testing step further comprises the steps of:

storing a dictionary identifying potential parts of speech for each word in the dictionary, any word not in said dictionary being considered a potential noun; and

using as a filtering condition whether a derived subsequence being tested includes a potential noun as a last token by finding potential parts of speech for the last token in the stored dictionary.

25. A computer implemented method of extracting a list of candidate multi-word technical terms from an input text file, a multi-word technical term being a string of at least two words having a particular meaning in some technical field, said method comprising the computer implemented steps of:

storing a stoplist of tokens which are assumed to not occur in multi-word technical terms, a token being a word, character of string of characters delimited by blanks and/or punctuation;

storing a frequency parameter specifying a minimum frequency of occurrence for a candidate multi-word technical term to be extracted;

extracting test fragments from an input text file by identifying delimiting tokens in the input text file at least in part by identifying as a delimiting token each token in the input text file which is the same as a token in the stored stoplist, the identified delimiting tokens defining text fragments therebetween;

deriving from the extracted text fragments all possible subsequences of tokens having a length of at least two tokens and which occur in the input text file with a frequency not less than specified by the stored frequency parameter;

testing each of the derived subsequences against at least one filtering condition; and

creating a sublist of the derived subsequences which pass the at least one filtering condition, the created sublist being the list of candidate multi-word technical terms.

26. A computer implemented method as defined in claim 25 and further comprising the steps of storing a list of allowable characters for candidate multi-word technical terms to be extracted, and in the step of extracting text fragments, identifying as a delimiting token each token in the input text file which contains at least one character not in the stored list of allowable characters.

27. A computer implemented method as defined in claim 26 wherein the testing step includes the step of testing each of the derived subsequences against a grammatical constraint filtering condition.

28. A computer implemented method as defined in claim 25 wherein the testing step includes the step of testing each of the derived subsequences against a grammatical constraint filtering condition.