Method for segmenting a text into words

• US 4,750,122 A
• Filed: 07/31/1985
• Issued: 06/07/1988
• Est. Priority Date: 07/31/1984
• Status: Expired due to Fees

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Shimizu, Fumon, Isatsu, Atsuko, Kaji, Hiroyuki, Nitta, Yoshihiko
• Primary Examiner(s): Ruggiero, Joseph F.
• Assistant Examiner(s): Meyer, Charles B.
• Application Number:
  US06/760,918
• Time in Patent Office: 1,042 Days
• Field of Search: 364/419, 364/900
• US Class Current: 704/1
• CPC Class Codes: G06F 40/242 Dictionaries
• CPC Class Codes: G06F 40/253 Grammatical analysis; Style critique
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/53 Processing of non-Latin text kana-to-kanji conversion G06F40/129; vowelisation G06F40/232

Abstract

A method of segmenting a text into words in which a dictionary search is made while using a character string in the text as a search key, and it is checked whether a word retrieved from the dictionary can be grammatically connected to another word adjacent thereto or not. Segmentation processing is carried out using only words registered in a word dictionary, processing for identifying an unknown word is carried out when the segmentation processing comes to a deadlock, and then the segmentation processing is continued for that portion of the text which follows the identified unknown word.

Claims

1. A method of segmenting a text into words for which a dictionary search is carried out using a character string in the text as a search key, and in which it is determined whether a word retrieved from a word dictionary can be grammatically connected to a preceding word, said method comprising the steps of:

segmenting a character string of the text into words using only words registered in the word dictionary;

identifying a word in the text undergoing segmentation processing as a possible unknown word, when said segmentation processing reaches a deadlock;

verifying that the possible unknown word is the unknown word based on character kind information in the text; and

continuing said segmentation processing for a portion of said text which follows the identified unknown word.

2. A method of segmenting a text into words according to claim 1, wherein said step of verifying an unknown word includes a step of preserving the result of segmentation which is obtained during a backtracking processing and is likely to be interrupted by the presence of an unknown word, and a step of considering a succeeding word of said result of segmentation which is obtained during the backtracking processing, as an unknown word immediately after it has been determined that other segmenting operations are now allowed.

3. A method of segmenting a text into words according to claim 1, wherein said step of verifying an unknown word includes a step of carrying out a backtracking processing within a predetermined range so that said backtracking processing is inhibited from being carried out beyond a position where the boundary between different kinds of characters coincides with the boundary between words.

4. A method of segmenting a text into words according to claim 2, wherein said step of verifying an unknown word includes a step of carrying out a backtracking processing within a predetermined range so that said backtracking processing is inhibited from being carried out beyond a position where the boundary between different kinds of characters coincides with the boundary between words.

5. A method of segmenting a text into words according to claim 1, wherein said step of verifying an unknown word includes a step of using one of a Chinese character string, a Japanese kana character string and an alphabetic character string each sandwiched between characters different in kind from said character string, as a possible unknown word, when said segmentation processing which uses only words registered in the word dictionary reaches a deadlock.

6. A method of segmenting text into words according to claim 1, wherein said text includes sentences lacking discrete boundaries between the words thereof.

7. A method of segmenting a text into words comprising:

a first step of carrying out a dictionary search using a character string in said text as a search key;

a second step of checking whether a word of said text retrieved from a dictionary can be grammatically connected to a preceding word of said text;

a third step of preserving a result obtained during the segmenting processing when a backtracking processing is carried out for a deadlock encountered in the processing in at least one of said first and second steps;

a fourth step of determining the presence of an unknown word in the text, based on character kind information of the text; and

a fifth step of resuming the segmenting processing utilizing the preserved result when the presence of the unknown word is determined.

8. A method of segmenting a text into words according to claim 7, wherein said text includes sentences lacking discrete boundaries between the words thereof.