Method and system for automatically extracting new word

• US 7,478,036 B2
• Filed: 08/30/2001
• Issued: 01/13/2009
• Est. Priority Date: 08/30/2000
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Shi, Qin, Shen, Liqin, Chai, Haixin
• Primary Examiner(s): Hudspeth; David R.
• Assistant Examiner(s): Han; Qi
• Application Number:
  US09/944,332
• Publication Number:
  US 20020077816A1
• Time in Patent Office: 2,693 Days
• Field of Search: 704/9, 704/10, 704/1, 704/8, 341/28, 715/533, 717/147
• US Class Current: 704/9
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models

Abstract

A method of and system for automatically extracting new words are provided. The method and system are highly efficient for automatically extracting new words from a mass amount of cleaned corpus.

Claims

1. A method of extracting new words automatically, said method comprising the steps of:

segmenting a cleaned corpus in a domain to form a segmented corpus;

splitting the segmented corpus to form sub strings, and counting the occurrences of each sub string appearing in the corpus; and

filtering out false candidates to output new words, wherein the new words are words not contained in a base vocabulary;

wherein the segmenting and the splitting is not dependent upon word boundaries;

wherein new words are determined based upon the domain of the cleaned corpus;

wherein the step of splitting and counting is implemented using a GAST (general atom suffix tree) contained in a reduced memory space;

wherein the GAST is implemented by limiting length of character sub strings.

2. The method of extracting new words automatically according to claim 1, wherein the step of segmenting comprises using punctuations, Arabic digits and alphabetic strings, or new word patterns to split the cleaned corpus.

3. The method of extracting new words automatically according to claim 1, wherein the step of segmenting comprises using common vocabulary to segment the cleaned corpus.

4. The method of extracting new words automatically according to claim 3, wherein the step of segmenting cleaned corpus comprises using pre-recognized functional words as segment boundary patterns.

5. The method of extracting new words automatically according to claim 3, wherein the step of filtering out false words comprises:

filtering out functional words;

filtering out those sub strings which almost always appear along with a longer sub string; and

filtering out those sub strings for which the occurrence is less than a predetermined threshold.

6. The method of extracting new words automatically according to claim 1, wherein the step of filtering out false candidates comprises:

filtering out functional words;

filtering out those sub strings which almost always appear along with a longer sub string; and

filtering out those sub strings for which the occurrence is less than a predetermined threshold.

7. The method of extracting new words automatically according to claim 1, wherein the step of segmenting the cleaned corpus comprises using pre-recognized functional words as segment boundary patterns.

8. An automatic new word extraction system, comprising:

a segmentor which segments a cleaned corpus in a domain to form a segmented corpus;

a splitter which splits the segmented corpus to form sub strings, and which counts the number of the sub strings appearing in the corpus; and

a filter which filters out false candidates to output new words, wherein the new words are words not contained in a base vocabulary;

wherein the segmenting and the splitting is not dependent upon word boundaries;

wherein new words are determined based upon the domain of the cleaned corpus;

wherein the splitter builds a GAST (general atom suffix tree) contained in a reduced memory space;

wherein the GAST limits the length of character sub strings.

9. The automatic word extraction system according to claim 8, wherein the segmentor uses punctuations, Arabic digits and alphabetic strings, or new word patterns to segment the cleaned corpus.

10. The automatic word extraction system according to claim 8, wherein the segmentor uses common vocabulary to segment the cleaned corpus.

11. The automatic word extraction system according to claim 10, wherein the segmentor uses pre-recognized functional words as segment boundary patterns.

12. The automatic word extraction system according to claim 10, wherein the filter filters out functional words;

those sub strings which almost always appear along with a longer sub string; and

those sub swings for which the occurrence is less than a predetermined threshold.

13. The automatic word extraction system according to claim 8, wherein the filter filters out functional words;

those sub strings which almost always appear along with longer sub strings; and

those sub strings for which the occurrence is less than a predetermined threshold.

14. The automatic word extraction system according to claim 8, wherein the segmentor uses pre-recognized functional words as segment boundary patterns.

15. A program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for extracting new words automatically, said method comprising the steps of:

segmenting a cleaned corpus in a domain to form a segmented corpus;

splitting the segmented corpus to form sub strings, and counting the occurrences of each sub string appearing in the corpus; and

filtering out false candidates to output new words, wherein the new words are words not contained in a base vocabulary;

wherein the segmenting and the splitting is not dependent upon word boundaries;

wherein new words are determined based upon the domain of the cleaned corpus;

wherein the step of splitting and counting is implemented using a GAST (general atom suffix tree) contained in a reduced memory space;

wherein the GAST is implemented by limiting length of character sub strings.