EXTRACTING TERMS FROM DOCUMENT DATA INCLUDING TEXT SEGMENT

• US 20130253916A1
• Filed: 05/21/2013
• Published: 09/26/2013
• Est. Priority Date: 10/02/2008
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Ikawa, Yohei, Takeuchi, Hironori, Negishi, Shiho
• Granted Patent: US 9,043,339 B2
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 16/345 Summarisation for human users
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30

Abstract

A computer system, method, and article of manufacture for extracting a term from electronic document data that includes a text segment. The system includes: a first extraction unit that uses a first text processing information to extract a noun word from the document data; a second extraction unit that uses a second text processing information to extract a term candidate in relation to the noun word or a corpus that includes text data described in the same language used in the document data; a weight assignment unit that uses a third text processing information to select which type to assign a weight from the plurality of types and assigns the weight to the selected type for each noun word and term candidate; a determination unit that determines the type to which the noun word and term candidate belong; and an output unit to output the noun word and term candidate.

Claims

1. A computer-implemented system including a memory and a processor communicatively coupled to the memory for extracting terms from electronic document data that includes a text segment, the computer system comprising:

a first extraction unit that uses a first text processing information to extract a noun word from the document data;

a second extraction unit that uses a second text processing information to extract a term candidate in relation to the extracted noun word from the document data or from a corpus that includes text data described in the same language used in the document data;

a weight assignment unit that, in order to determine which one of a plurality of noun word types the extracted noun word and the extracted term candidate each belong to, uses a third text processing information to select which type to assign a weight from the plurality of types and assigns the weight to the selected type for each of the extracted noun word and the extracted term candidate;

a determination unit that determines the type to which the extracted noun word and the extracted term candidate each belong, based on the assigned weight; and

an output unit which follows the determination to output the extracted noun word and the extracted term candidate each in association with the determined type.

2. The computer-implemented system according to claim 1, wherein:

the first text processing information is selected from the group consisting of;

a document structure information, a language superficial information, a language part-of-speech information, and a morphological analysis information; and

the second text processing information, the third text processing information, and the fourth text processing information are word modification information.

3. The computer-implemented system according to claim 1, wherein:

the first extraction unit uses the first text processing information to extract the noun word from the document data by performing a morphological analysis on the document data to extract noun words (Ki (i=1, 2, . . . , n)); and

the first extract unit further assigns, to each of the extracted noun words Ki, a weight corresponding to at least one of a position and a proportion of the noun word Ki in the document data.

4. The computer-implemented system according to claim 3, wherein:

the weight assignment corresponding to the position of each noun word Ki in the document data is to assign a weight corresponding to whether the noun word Ki is in a sentence, in the text segment, or in a parenthesis; and

the weight assignment corresponding to the proportion of each noun word Ki in the document data is to assign a weight corresponding to whether the noun word Ki occupies a predetermined proportion of the text segment or of a character string in the parenthesis.

5. The computer-implemented system according to claim 3, wherein:

when the position of the noun word Ki is not in a sentence, a determination is made as to whether or not the noun word Ki takes up the entire text segment;

when the noun word Ki takes up the entire text segment, a score W is assigned to the noun word Ki;

when the noun word Ki does not take up the entire text segment, a score Y is assigned to the noun word Ki;

when the position of the noun word Ki is in a sentence, a determination is made as to whether or not the noun word Ki is in a parenthesis in the sentence and takes up the entire character string in the parenthesis;

when the noun word Ki is in the parenthesis and takes up the entire character string in the parenthesis, a score X is assigned to the noun word Ki; and

when the noun word Ki is not in a parenthesis or does not take up the entire character string, a score Z is assigned to the noun word Ki, wherein score W is greater than score X, score X is greater than score Y, and score Y is greater than score Z.

6. The computer-implemented system according to claim 3, wherein the first extraction unit further:

collects the same noun words from the noun words Ki and sets the same noun words as a certain one of noun words Si (i=1, 2, . . . , k) (n≧

k);

assigns a weight to each of the noun words Si based on the weights assigned to the corresponding noun words Ki;

and

extracts the noun word Si assigned a weight equal to or greater than a predetermined threshold value.

7. The computer-implemented system according to claim 6, wherein:

the first extraction unit determines whether or not the weight of each noun word Si is in a predetermined threshold value range; and

when the weight of the noun word Si is in the predetermined threshold value range, the first extraction unit determines whether or not a character type of the noun word Si is adequate as a term;

when determining the character type as adequate, the first extraction unit sets the noun word Si as a noun word to be extracted;

when determining the character type as inadequate, the first extraction unit sets the weight of the noun word Si to zero; and

when the weight of the noun word Si is not in the predetermined threshold value range, the first extraction unit sets the weight of the noun word Si to zero.

8. The computer-implemented system according to claim 7, wherein:

the first extraction unit determines the character type of the noun word Si as inadequate when the character type is any one of a number, symbol, or hiragana or a combination of only the number and the symbol.

9. The computer-implemented system according to claim 1, wherein:

the second extraction unit extracts a term candidate satisfying a predetermined grammar requirement from the document data or from the corpus including the text data described in the same language used in the document data.

10. The computer-implemented system according to claim 9, wherein:

the extraction of the term candidate satisfying the predetermined grammar requirement includes identifying a verb word for which the extracted noun word is a direct object and extracting a noun word which is an object of the same verb word as the identified verb word, in the document data or in the corpus including the text data described in the same language used in the document data.

11. The computer-implemented system according to claim 1, wherein the weight assignment unit uses the third text processing information to select which type to assign the weight from the plurality of types and assigns the weight to the selected type for each of the extracted noun word and the extracted term candidate by:

obtaining a number of times a genitive case word modifies the extracted noun word and a number of times a genitive case word modifies the extracted term candidate, in the document data or in the corpus including the text data described in the same language used in the document data; and

selecting the type to be assigned a weight according to whether or not the obtained number of times is in a predetermined threshold value range.

12. The computer-implemented system according to claim 1, wherein:

the plurality of types are a component type (Vc), a parameter type (Vp), and a schema type (Vc);

the component type Vc represents a level of a term belonging to a component;

the parameter type Vp represents a level of a term representing a parameter; and

the schema type Vs represents a level of a term representing a schema.

13. The computer-implemented system according to claim 12, wherein the weight assignment unit further:

assigning a score A to the component type Vc for the extracted noun word when the number of times obtained for the extracted noun word is lower than the predetermined threshold value; and

assigning the score A to the parameter type Vp and the schema type Vs for the extracted noun word when the number of times obtained for the extracted noun word is the predetermined threshold value or greater.

14. The computer-implemented system according to claim 13, wherein the weight assignment unit further:

assigns a score B to the component type Vc for the extracted term candidate when the number of times obtained for the extracted term candidate is lower than the predetermined threshold value; and

assigns the score B to the parameter type Vp and the schema type Vs for the extracted term candidate when the number of times obtained for the extracted term candidate is the predetermined threshold value or greater, wherein score A is greater than score B.

15. The computer-implemented system according to claim 14, wherein

the second extraction unit uses the second text processing information to extract the term candidate in relation to the extracted noun word from the document data or from the corpus that includes text data described in the same language used in the document data by identifying a verb word for which the extracted noun word is a direct object and extracts a noun word NPi which is an object of the same verb word as the identified verb word, from the document data or from the corpus including the text data described in the same language used in the document data;

and

the weight assignment unit further assigns a score C to the parameter type Vp for the extracted noun word when the noun word NPi includes a number, wherein score B is greater than score C.

16. The computer-implemented system according to claim 15, wherein the weight assignment unit further:

obtains a number of times a genitive case word modifies a noun word ti which, out of the extracted noun words, appears in a first row of the text segment of the document data, in the document data or the corpus including the text data described in the same language used in the document data; and

selects the type to be assigned a weight according to whether the obtained number of times is in a predetermined threshold value range.

17. The computer-implemented system according to claim 16, wherein the weight assignment unit further:

assigns a score D to the schema type Vs for the extracted noun word corresponding to the noun word ti when the number of times obtained for the noun word ti is lower than the predetermined threshold value; and

assigns the score A to the component type Vc for the extracted noun word corresponding to the noun word ti when the number of times obtained for the noun word ti is the predetermined threshold value or greater, wherein

score A is greater than score B, score B is greater than score C, score C is greater than score D.

18. The computer-implemented system according to claim 17, wherein the weight assignment unit further:

obtains, in response to a list input of a term (s-seed) as a possible schema, frequencies of a noun word following the s-seed and a noun word modified by the s-seed as a genitive case word, in the document data or in the corpus including the text data described in the same language used in the document data, and

stores in a list the noun word the frequency of which is one or more, out of the noun word following the s-seed and the noun word modified by the s-seed as a genitive case word.

19. The computer-implemented system according to claim 18, wherein the weight assignment unit further assigns the score D to the component type Vc for the extracted noun word corresponding to the noun word ti when the noun word ti which, out of the extracted noun words, appears in the first row of the text segment of the text data coincides with the noun word stored in the list.

20. A computer-implemented method for extracting terms from electronic document data that includes a text segment, the method comprising the steps of:

using a first text processing information to extract a noun word from the document data and storing the extracted noun word in a storage unit;

using a second text processing information to extract a term candidate in relation to the extracted noun word from the document data or from a corpus that includes text data described in the same language used in the document data and storing the extracted term candidate in the storage unit;

in order to determine which noun word type out of a plurality of types the extracted noun word and the extracted term candidate each belong to, using a third text processing information to select which type to assign a weight from the plurality of types, assigning the weight to the selected type for each of the extracted noun word and the extracted term candidate, and storing the assigned weight in the storage unit;

determining the type to which the extracted noun word and the extracted term candidate each belong, based on the assigned weight; and

following the determination to output the extracted noun word and the extracted term candidate each in association with the determined type onto a display device.