CREATING A TERMS DICTIONARY WITH NAMED ENTITIES OR TERMINOLOGIES INCLUDED IN TEXT DATA

• US 20100174528A1
• Filed: 01/04/2010
• Published: 07/08/2010
• Est. Priority Date: 01/05/2009
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: TOYOSHIMA, HIROBUMI, TAKUMA, DAISUKE, OYA, HIROKI
• Granted Patent: US 8,538,745 B2
• Time in Patent Office: Days
• US Class Current: 704/10
• CPC Class Codes: G06F 40/242 Dictionaries
• CPC Class Codes: G06F 40/268 Morphological analysis
• CPC Class Codes: G06F 40/295 Named entity recognition

Abstract

A computer system of an embodiment of the disclosure can be used to automatically create or populate a terms dictionary using a set of computing units. A morphological analysis unit can acquire token sequence data by performing morphological analysis for the text data. A category distinguishing unit can distinguish tokens of the token sequence data by using a category dictionary to extract uncategorized words. An uncategorized-word comparing unit can compare each of the extracted uncategorized words with an uncategorized-word comparison rule to extract an uncategorized word matching the uncategorized-word comparison rule as a registration candidate word. A token-sequence comparing unit can compare a token sequence of the token sequence data with a token-sequence comparison rule to extract a token sequence matching the token-sequence comparison rule as registration candidate words. A permission unit can permit a user to select whether to register the registration candidate words in the category dictionary.

Claims

1. A method of creating a terms dictionary with named entities or terminologies included in text data, comprising:

acquiring token sequence data by performing morphological analysis for the text data;

distinguishing tokens of the token sequence data by using a category dictionary to extract uncategorized words;

comparing each of the extracted uncategorized words with an uncategorized-word comparison rule to extract an uncategorized word matching the uncategorized-word comparison rule as a registration candidate word, wherein the uncategorized-word comparison rule includes a token composed of a first character string and a first regular expression for use in extracting the matching uncategorized word;

comparing a token sequence of the token sequence data with a token-sequence comparison rule to extract a token sequence matching the token-sequence comparison rule as registration candidate words, wherein the token-sequence comparison rule includes a token sequence including a second character string and a second regular expression for use in extracting the matching token sequence; and

permitting a user to select whether to register the registration candidate words in the category dictionary.

2. The method according to claim 1, further comprising:

calculating a confidence factor of each of the registration candidate words, wherein the confidence factor represents a degree of probability with which the registration candidate word is registered in the category dictionary or probability with which the registration candidate word is added to the token-sequence comparison rule.

3. The method of claim 2, wherein the confidence factor calculates the confidence factor by using information on a number of characters or types of characters.

4. The method of claim 2, wherein the calculating of the confidence factor uses information in a plurality of external dictionaries, which include a technical terminology dictionary and a general terms dictionary, and wherein the calculating of the confidence factor calculation adds more points to the score of the confidence factor in cases where a character string in a token which matches the first or second regular expression exists in a vocabulary of the technical terminology dictionary than in cases where the character string exists in a vocabulary of the general terms dictionary.

5. The method according to claim 2, wherein permitting the user to sect whether to register the registration candidate words comprises:

sorting the registration candidate words according to the confidence factors calculated for each of the registration candidate words; and

displaying the sorted registration candidate words on a display device;

6. The method according to claim 1, further comprising:

registering an uncategorized word, which is extracted as the registration candidate word and selected by the user, in the category dictionary

7. The method according to claim 1, wherein the category dictionary includes a classification category dictionary and a nontarget category dictionary.

8. The method of claim 4, wherein the distinguishing of the tokens of the token sequence data comprises:

distinguishing the tokens of the token sequence data by using the classification category dictionary and the nontarget category dictionary, wherein the classification category dictionary includes tokens registered in the terms dictionary and arbitrarily includes the word class information of the tokens and the nontarget category dictionary includes tokens not registered in the terms dictionary and arbitrarily includes the word class information of the tokens.

9. The method of claim 8, wherein the permitting of the user to select whether to register the registration candidate words comprises at least one of:

registering an uncategorized word, which is extracted as the registration candidate word and selected by the user, in the classification category dictionary;

registering an uncategorized word, which is extracted as the registration candidate word and not selected by the user, in the nontarget category dictionary; and

adding the token sequence, which is extracted as the registration candidate words and not selected by the user, in the token-sequence list.

10. The method of claim 8, wherein the permitting of the user to select whether to register the registration candidate words comprises:

registering an uncategorized word, which is extracted as the registration candidate word and selected by the user, in the classification category dictionary.

11. The method of claim 8, wherein the permitting of the user to select whether to register the registration candidate words comprises:

registering an uncategorized word, which is extracted as the registration candidate word and not selected by the user, in the nontarget category dictionary

12. The method of claim 8, wherein the permitting of the user to select whether to register the registration candidate words comprises:

adding the token sequence, which is extracted as the registration candidate words and not selected by the user, in the token-sequence list.

13. A computer program for creating a terms dictionary with named entities or terminologies included in text data, wherein the computer program is stored in a tangible storage medium and when executed causes a computer system to:

acquire token sequence data by performing morphological analysis for the text data;

distinguish tokens of the token sequence data by using a category dictionary to extract uncategorized words;

compare each of the extracted uncategorized words with an uncategorized-word comparison rule to extract an uncategorized word matching the uncategorized-word comparison rule as a registration candidate word, wherein the uncategorized-word comparison rule includes a token composed of a first character string and a first regular expression for use in extracting the matching uncategorized word;

compare a token sequence of the token sequence data with a token-sequence comparison rule to extract a token sequence matching the token-sequence comparison rule as registration candidate words, wherein the token-sequence comparison rule includes a token sequence including a second character string and a second regular expression for use in extracting the matching token sequence; and

permit a user to select whether to register the registration candidate words in the category dictionary.

14. A computer system for creating a terms dictionary with named entities or terminologies included in text data, the computer system comprising:

a morphological analysis unit for acquiring token sequence data by performing morphological analysis for the text data;

a category distinguishing unit for distinguishing tokens of the token sequence data by using a category dictionary to extract uncategorized words;

an uncategorized-word comparing unit for comparing each of the extracted uncategorized words with an uncategorized-word comparison rule to extract an uncategorized word matching the uncategorized-word comparison rule as a registration candidate word, wherein the uncategorized-word comparison rule includes a token composed of a first character string and a first regular expression for use in extracting the matching uncategorized word;

a token-sequence comparing unit for comparing a token sequence of the token sequence data with a token-sequence comparison rule to extract a token sequence matching the token-sequence comparison rule as registration candidate words, wherein the token-sequence comparison rule includes a token sequence including a second character string and a second regular expression for use in extracting the matching token sequence; and

a permission unit for permitting a user to select whether to register the registration candidate words in the category dictionary.

15. The computer system according to claim 14, further comprising:

a confidence factor calculation unit for calculating a confidence factor of each of the registration candidate words, wherein the confidence factor represents a degree of probability with which the registration candidate word is registered in the category dictionary or probability with which the registration candidate word is added to the token-sequence comparison rule.

16. The computer system according to claim 14, further comprising:

a registration unit for registering an uncategorized word, which is extracted as the registration candidate word and selected by the user, in the category dictionary.

17. The computer system according to claim 14, wherein:

the morphological analysis unit when executed acquires word class information corresponding to the respective tokens of the token sequence data; and

the category distinguishing unit when executed does not extract the tokens of the token sequence data in cases where the tokens of the token sequence data and the word class information corresponding to the tokens match the tokens registered in the category dictionary and the word class information corresponding to the tokens, respectively.

18. The computer system according to claim 14, wherein:

the morphological analysis unit when executed acquires the word class information corresponding to the respective tokens of the token sequence data; and

the token-sequence comparing unit when executed does not extract the token sequence of the token sequence data in cases where the token sequence of the token sequence data and the word class information corresponding to the tokens of the token sequence match the token sequence registered in the token-sequence list and the word class information corresponding to the tokens of the token sequence, respectively.

19. The computer system according to claim 14, wherein:

the morphological analysis unit when executed acquires the word class information corresponding to the tokens of the token sequence data;

the category dictionary includes a classification category dictionary and a nontarget category dictionary;

the classification category dictionary includes tokens registered in the terms dictionary and arbitrarily includes the word class information of the tokens; and

the nontarget category dictionary includes tokens not registered in the terms dictionary and arbitrarily includes the word class information of the tokens.

20. The computer system according to claim 19, wherein:

the category distinguishing unit when executed distinguishes the tokens of the token sequence data by using the classification category dictionary;

the permission unit when executed permits the user to select whether to register the registration candidate words in the classification category dictionary; and

the registration unit when executed registers an uncategorized word, which is extracted as the registration candidate word and selected by the user, in the category dictionary.