System for chinese tokenization and named entity recognition

• US 6,311,152 B1
• Filed: 02/17/2000
• Issued: 10/30/2001
• Est. Priority Date: 04/08/1999
• Status: Expired due to Fees

• Current Assignee: Kent Ridge Digital Labs
• Original Assignee: Kent Ridge Digital Labs
• Inventors: Loudon, Gareth, Li, Haizhou, Bai, Shuanhu, Wu, Horng Jyh Paul
• Primary Examiner(s): Thomas, Joseph
• Application Number:
  US09/403,069
• Time in Patent Office: 621 Days
• Field of Search: 704/9, 704/10, 704/1, 704/251, 704/255, 704/256, 704/257, 704/242, 704/243, 707/530, 707/531, 707/532, 707/536, 707/535
• US Class Current: 704/9
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/289 Phrasal analysis, e.g. finite state techniques or chunking
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06F 40/53 Processing of non-Latin text kana-to-kanji conversion G06F40/129; vowelisation G06F40/232

Abstract

A system (100, 200) for tokenization and named entity recognition of ideographic language is disclosed. In the system, a word lattice is generated for a string of ideographic characters using finite state grammars (150) and a system lexicon (240). Segmented text is generated by determining word boundaries in the string of ideographic characters using the word lattice dependent upon a contextual language model (152A) and one or more entity language models (152B). One or more named entities is recognized in the string of ideographic characters using the word lattice dependent upon the contextual language model (152A) and the one or more entity language models (152B). The contextual language model (152A) and the one or more entity language models (152B) are each class-based language models. The lexicon (240) includes single ideographic characters, words, and predetermined features of the characters and words.

Claims

1. A method of tokenization and named entity recognition of ideographic language, said method including the steps of:

generating a word lattice for a string of ideographic characters using finite state grammars and a system lexicon, said finite state grammars are a dynamic and complementary extension of said lexicon for creating named entity hypotheses and said lexicon includes single ideographic characters, words, and predetermined features of said characters and words;

generating segmented text by determining word boundaries in said string of ideographic characters using said word lattice dependent upon a contextual language model and one or more entity language models; and

recognizing one or more named entities in said string of ideographic characters using said word lattice dependent upon said contextual language model and said one or more entity language models.

2. The method according to claim 1. further including the step of combining said contextual language model and said one or more entity language models.

3. The method according to claim 1, wherein said contextual language model and said one or more entity language models are each class-based language models.

4. The method according to claim 1, wherein said contextual language model and said one or more entity language models incorporate local and contextual linguistic information, respectively, for producing prioritized word and corresponding category sequences.

5. The method according to claim 4, wherein said contextual language model and said one or more entity language models are dependent upon an n-gram paradigm.

6. The method according to claim 1, wherein said lattice generating step includes the step of generating one or more elements of said lattice using said lexicon.

7. The method according to claim 1, wherein said finite state grammars run on said predetermined features contained in said lexicon to suggest possible entities, entity boundaries and entity categories.

8. An apparatus for tokenization and named entity recognition of ideographic language, said apparatus including:

means for generating a word lattice for a string of ideographic characters using finite state grammars and a system lexicon, said finite state grammars are a dynamic and complementary extension of said lexicon for creating named entity hypotheses and said lexicon includes single ideographic characters, words, and predetermined features of said characters and words;

means for generating segmented text by determining word boundaries in said string of ideographic characters using said word lattice dependent upon a contextual language model and one or more entity language models; and

means for recognizing one or more named entities in said string of ideographic characters using said word lattice dependent upon said contextual language model and said one or more entity language models.

9. The apparatus according to claim 8, further including means for combining said contextual language model and said one or more entity language models.

10. The apparatus according to claim 8, wherein said contextual language model and said one or more entity language models are each class-based language models.

11. The apparatus according to claim 8, wherein said contextual language model and said one or more entity language models incorporate local and contextual linguistic information, respectively, for producing prioritized word and corresponding category sequences.

12. The apparatus according to claim 11 wherein said contextual language model and said one or more entity language models are dependent upon an n-gram paradigm.

13. The apparatus according to claim 8, wherein said lattice-generating means includes means for generating one or more elements of said lattice using said lexicon.

14. The apparatus according to claim 8, wherein said finite state grammars run on said predetermined features contained in said lexicon to suggest possible entities, entity boundaries and entity categories.

15. A computer program product having a computer readable medium having a computer program recorded therein for tokenization and named entity recognition of ideographic language, said computer program product including:

computer program means for generating a word lattice for a string of ideographic characters using finite state grammars and a system lexicon, said finite state grammars are a dynamic and complementary extension of said lexicon for creating named entity hypotheses and said lexicon includes single ideographic characters, words, and predetermined features of said characters and words;

computer program means for generating segmented text by determining word boundaries in said string of ideographic characters using said word lattice dependent upon a contextual language model and one or more entity language models; and

computer program means for recognizing one or more named entities in said string of ideographic characters using said word lattice dependent upon said contextual language model and said one or more entity language models.

16. The computer program product according to claim 15, further including computer program means for combining said contextual language model and said one or more entity language models.

17. The computer program product according to claim 15, wherein said contextual language model and said one more entity language models are each class-based language models.

18. The compute program product according to claim 15, wherein said cotextual language model and said one or more entity language models incorporate local and contextual linguistic information, respectively, for producing prioritized word and corresponding category sequences.

19. The computer program product according to claim 18, wherein said contextual language model and said one or more entity language models are dependent upon an n-gram paradigm.

20. The computer program product according to claim 15, wherein said computer program means for lattice-generating includes computer program code means for generating one or more elements of said lattice using said lexicon.

21. The computer program product according to claim 15, wherein said finite state grammars run on said predetermined features contained in said lexicon to suggest possible entities, entity boundaries and entity categories.