Device for generating aligned corpus based on unsupervised-learning alignment, method thereof, device for analyzing destructive expression morpheme using aligned corpus, and method for analyzing morpheme thereof

• US 10,282,413 B2
• Filed: 08/27/2014
• Issued: 05/07/2019
• Est. Priority Date: 10/02/2013
• Status: Active Grant

• Current Assignee: Systran International Co., Ltd.
• Original Assignee: Systran International Co., Ltd.
• Inventors: Ji, Chang Jin
• Primary Examiner(s): Le, Thuykhanh
• Application Number:
  US15/026,275
• Publication Number:
  US 20160217122A1
• Time in Patent Office: 1,714 Days
• Field of Search: None
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 40/232 Orthographic correction, e.g. spell checking or vowelisation
• CPC Class Codes: G06F 40/237 Lexical tools
• CPC Class Codes: G06F 40/242 Dictionaries
• CPC Class Codes: G06F 40/268 Morphological analysis
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

Disclosed is a device for generating an aligned corpus based on unsupervised-learning alignment, and a method thereof, a device for analyzing a destructive expression morpheme using an aligned corpus, and a method for analyzing a morpheme thereof.

The morpheme analyzing device includes a knowledge database and an analyzer. The knowledge database includes an aligned corpus for storing a plurality of knowledge information sets used for a per-language morpheme analysis, and stores a morpheme dictionary for storing morpheme information corresponding to a normal expression and normal expression information corresponding to a destructive expression (here, the destructive expression represents an expression that is erroneous in orthography or is not normalized and standardized). The analyzer performs a morpheme analysis on an input separate word by use of the knowledge database and outputs an analysis result, and when a morpheme on the input separate word is not provided in the morpheme dictionary, finds a normal expression corresponding to the destructive expression by use of the aligned corpus regarding the destructive expression included in the input separate word, and performs a morpheme analysis.

Claims

1. A device for analyzing a morpheme in natural language information processing, comprising:

a knowledge database including an aligned corpus for storing a plurality of knowledge information sets used for a per-language morpheme analysis, and storing a morpheme dictionary for storing morpheme information corresponding to a normal expression and normal expression information corresponding to a destructive expression, wherein the destructive expression represents an expression that is erroneous in orthography or is not normalized and standardized, and includes an orthographic error; and

an analyzer for, by a processor, performing a morpheme analysis on an input separate word by use of the knowledge database and outputting an analysis result, and when a morpheme on the input separate word is not provided in the morpheme dictionary, finding the normal expression corresponding to the destructive expression by use of the aligned corpus regarding the destructive expression included in the input separate word and performing a morpheme analysis,

wherein the aligned corpus is generated by performing an unsupervised-learning-based alignment on a parallel corpus storing pairs of a destructive sentence including the destructive expression and a normal sentence corresponding to the destructive sentence, and

wherein the parallel corpus is built by collecting a plurality of destructive sentences through a network, performing retrieval through the network with the destructive expression included in the collected destructive sentence as a query to determine universality on the corresponding destructive sentence, generating the normal sentence corresponding to the destructive sentence when the collected destructive sentence is determined to have universality, and forming the generated normal sentence and the corresponding destructive sentence into one pair to build the parallel corpus,

wherein the knowledge database further includes an analyzed dictionary for storing per-morpheme access information, and

the analyzer includes;

a morpheme divider for dividing the morphemes forming the input separate word by use of the morpheme dictionary, and when the morpheme forming the input separate word is not provided in the morpheme dictionary, performing a morpheme division by using the corresponding normal expression by use of the aligned corpus;

an access information checker for extracting morphemes that are capable of being combined with the morphemes divided by the morpheme divider by use of the analyzed dictionary; and

an original form restoring unit for performing an original form restoration on the morphemes extracted by the access information checker and outputting it as a morpheme analysis result.

2. The device of claim 1, wherein

the morpheme divider includes:

a candidate group generator for dividing the input separate word into morphemes to generate morpheme candidates;

a normal expression checking unit for checking whether a corresponding morpheme candidate exists regarding the morpheme candidate generated by the candidate group generator by use of the morpheme dictionary;

a destructive expression checking unit for checking whether the destructive expression of the input separate word exists regarding the corresponding morpheme candidate by use of the aligned corpus when the morpheme candidate is determined to not exist in the morpheme dictionary by the normal expression checking unit;

a dividing processor for morpheme-dividing the corresponding morpheme candidate from the input separate word when it is determined by the normal expression checking unit that the morpheme candidate exists in the morpheme dictionary; and

a controller for controlling the candidate group generator, the normal expression checking unit, the destructive expression checking unit, and the dividing processor to perform a morpheme division on the input separate word, and when the morpheme candidate generated by the candidate group generator is determined to not exist in the morpheme dictionary by the normal expression checking unit, finding the corresponding normal expression through the destructive expression checking unit and re-inputting it to the candidate group generator.

3. The device of claim 1, wherein

the aligned corpus includes a correlation probability that is an alignment probability value between the destructive expression and the normal expression corresponding to the destructive expression.

4. The device of claim 3, wherein

in the case of a morpheme analysis, when it is determined whether the morpheme exists through the morpheme dictionary, each corresponding string is digitized and expressed as a probability that the same is a morpheme by using the correlation probability.

5. The device of claim 1, wherein the aligned corpus is generated based on an expectation-maximization (EM) algorithm.

6. The device of claim 1, further comprising an unsupervised-learning aligner for, by the processor, generating the aligned corpus by learning an optimal alignment probability value among a single syllable, multiple syllables, or words of the destructive expression and the normal expression by use of an unsupervised-learning method to the parallel corpus.

7. The device of claim 6, wherein the unsupervised-learning aligner includes:

an initializer for setting a sub-string pair in a random manner or according to an initialization methodology from a given string of the parallel corpus and initializing a relevance probability;

an E-step processor for performing a match between the destructive expression and the normal expressions so that a gap between the destructive sentence and the normal sentence is optimized according to matching of strings and a corresponding match probability value;

an M-step processor for recalculating a probability between respective sub-string pairs according to a new alignment; and

a controller for inputting a string of the destructive sentence and the normal sentence built in the parallel corpus to the initializer, controlling repetition of the E-step processor and the M-step processor until a stop condition of unsupervised-learning is satisfied after a calculation of probability by the M-step processor, and when the stop condition is satisfied, generating the aligned corpus storing the destructive expression, the normal expression, and a correlation probability of a final output.

8. A method for analyzing a morpheme in natural language information processing by a processor by using a database storing a morpheme dictionary and an aligned corpus, comprising:

dividing an input separate word into a morpheme and generating a morpheme candidate;

determining whether the morpheme candidate exists in the morpheme dictionary, wherein the morpheme dictionary represents a dictionary for storing morpheme information corresponding to a normal expression;

when the morpheme candidate exists in the morpheme dictionary, performing a corresponding morpheme analysis;

when the morpheme candidate does not exist in the morpheme dictionary, determining whether a destructive expression included in the input separate word exists in the aligned corpus, wherein the destructive expression represents an expression that is erroneous in orthography or is not normalized and standardized, and includes an orthographic error, and wherein the aligned corpus represents a corpus for storing normal expression information corresponding to the destructive expression; and

when the destructive expression exists in the aligned corpus, finding the normal expression corresponding to the destructive expression and performing a morpheme analysis,

wherein the generating of the aligned corpus includes

building a parallel corpus storing pairs of a destructive sentence including the destructive expression and a normal sentence corresponding to the destructive sentence, and performing an unsupervised-learning-based alignment on the parallel corpus to generate the aligned corpus aligned with the destructive expression and the normal expression corresponding to the destructive expression, and

wherein the building of the parallel corpus includes collecting a plurality of destructive sentences through a network, performing retrieval through the network with the destructive expression included in the collected destructive sentence as a query to determine universality on the corresponding destructive sentence, generating the normal sentence corresponding to the destructive sentence when the collected destructive sentence is determined to have universality, and forming the generated normal sentence and the corresponding destructive sentence into one pair to build the parallel corpus,

wherein

the performing of a morpheme analysis includes;

performing a morpheme division by using the corresponding normal expression by use of the aligned corpus when the morpheme candidate is not provided in the morpheme dictionary;

extracting a morpheme that is capable of being combined with the morpheme by use of an analyzed dictionary for storing per-morpheme access information; and

performing an original form restoration on the extracted morpheme to output it as a morpheme analysis result.

9. The method of claim 8, further comprising

before the determining whether the morpheme candidate exists in the morpheme dictionary, generating the aligned corpus.

10. The method of claim 8, wherein

in the determining of universality,

the corresponding destructive expression is determined to have universality when it exceeds a reference amount with respect to the amount of a result of the retrieval according to the query, and

the determining of universality is automatically performed through web retrieval through a plurality of portal sites.

11. The method of claim 8, wherein

the generating of an aligned corpus further includes:

setting a sub-string pair in a random manner or according to an initialization methodology from a given string of the parallel corpus and initializing a relevance probability;

performing a match between the destructive expression and the normal expressions so that a gap between the destructive sentence and the normal sentence is optimized according to matching of strings and a corresponding match probability value;

recalculating a probability between respective sub-string pairs according to a new alignment; and

repeating the performing of a match and the calculating until a stop condition of an unsupervised-learning is satisfied, and when the stop condition is satisfied, storing the destructive expression, a normal expression, and a correlation probability of a final output in the aligned corpus, wherein the correlation probability is an alignment probability value of the destructive expression and the normal expression.

12. The method of claim 8, wherein

an expectation-maximization (EM) algorithm is performed so as to find the unsupervised-learning-based alignment performed to generate the aligned corpus, and set a probability value of the alignment.