METHOD AND APPARATUS FOR BILINGUAL WORD ALIGNMENT, METHOD AND APPARATUS FOR TRAINING BILINGUAL WORD ALIGNMENT MODEL

• US 20070203689A1
• Filed: 02/23/2007
• Published: 08/30/2007
• Est. Priority Date: 02/28/2006
• Status: Active Grant

• Current Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Original Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Inventors: Wang, Haifeng, Liu, Zhanyi, WU, Hua
• Granted Patent: US 7,827,027 B2
• Time in Patent Office: Days
• US Class Current: 704/2
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars

Abstract

The present invention provides method and apparatus for bilingual word alignment, method and apparatus for training bilingual word alignment model. The method for bilingual word alignment, comprising: training a bilingual word alignment model using a word-aligned labeled bilingual corpus; word-aligning a plurality of bilingual sentence pairs in a unlabeled bilingual corpus using said bilingual word alignment model; determining whether the word alignment of each of said plurality of bilingual sentence pairs is correct, and if it is correct, adding the bilingual sentence pair into the labeled bilingual corpus and removing the bilingual sentence pair from the unlabeled bilingual corpus; retraining the bilingual word alignment model using the expanded labeled bilingual corpus; and re-word-aligning the remaining bilingual sentence pairs in the unlabeled bilingual corpus using the retrained bilingual word alignment model.

Claims

1. A method for bilingual word alignment, comprising:

training a bilingual word alignment model using a word-aligned labeled bilingual corpus;

word-aligning a plurality of bilingual sentence pairs in a unlabeled bilingual corpus using said bilingual word alignment model;

determining whether the word alignment of each of said plurality of bilingual sentence pairs is correct, and if it is correct, adding the bilingual sentence pair into the labeled bilingual corpus and removing the bilingual sentence pair from the unlabeled bilingual corpus;

retraining the bilingual word alignment model using the expanded labeled bilingual corpus; and

re-word-aligning the remaining bilingual sentence pairs in the unlabeled bilingual corpus using the retrained bilingual word alignment model.

2. The method for bilingual word alignment according to claim 1, further comprising:

repeating the steps of determining, retraining and re-word-aligning, until no new correct word alignment is generated.

3. The method for bilingual word alignment according to claim 1, wherein said bilingual word alignment model comprises at least word translation probability, position distortion probability and word fertility probability.

4. The method for bilingual word alignment according to any one of claims 1˜

3, wherein said step of training a bilingual word alignment model comprises;

training a forward bilingual word alignment model using the word-aligned labeled bilingual corpus; and

training a backward bilingual word alignment model using the word-aligned labeled bilingual corpus;

wherein said step of word-aligning a plurality of bilingual sentence pairs in a unlabeled bilingual corpus comprises;

forward-word-aligning each of said plurality of bilingual sentence pairs using said forward bilingual word alignment model; and

backward-word-aligning each of said plurality of bilingual sentence pairs using said backward bilingual word alignment model;

wherein said step of determining whether the word alignment of each of said plurality of bilingual sentence pairs is correct comprises;

calculating the intersection set between the forward-word-aligning result and the backward-word-aligning result of the bilingual sentence pair;

calculating the union set between the forward-word-aligning result and the backward-word-aligning result of the bilingual sentence pair; and

if the ratio of the element number of said intersection set to the element number of said union set is greater than a predetermined threshold, determining the word alignment of said bilingual sentence pair is correct.

5. The method for bilingual word alignment according to claim 4, wherein said step of retraining the bilingual word alignment model comprises:

retraining the forward bilingual word alignment model using said expended labeled bilingual corpus; and

retraining the backward bilingual word alignment model using said expended labeled bilingual corpus.

6. The method for bilingual word alignment according to claim 4, wherein said step of adding the bilingual sentence pair into the labeled bilingual corpus comprises:

if the forward-word-aligning result and the backward-word-aligning result are not identical, calculating word translation probabilities of the different portions in the forward-word-aligning result and the backward-word-aligning result, and adding the word-aligning result having higher word translation probability into the labeled bilingual corpus.

7. A method for training bilingual word alignment model, comprising:

training an initial bilingual word alignment model using a word-aligned labeled bilingual corpus;

word-aligning a plurality of bilingual sentence pairs in a unlabeled bilingual corpus using said initial bilingual word alignment model;

determining whether the word alignment of each of said plurality of bilingual sentence pairs is correct, and if it is correct, adding the bilingual sentence pair into the labeled bilingual corpus and removing the bilingual sentence pair from the unlabeled bilingual corpus;

training a bilingual word alignment model using the expanded labeled bilingual corpus.

8. The method for training bilingual word alignment model according to claim 7, further comprising:

re-word-aligning the remaining bilingual sentence pairs in the unlabeled bilingual corpus using the newly trained bilingual word alignment model; and

repeating the steps of determining, training and re-word-aligning, until no new correct word alignment is generated.

9. The method for training bilingual word alignment model according to claim 7, wherein said bilingual word alignment model comprises at least word translation probability, position distortion probability and word fertility probability.

10. The method for training bilingual word alignment model according to any one of claims 7˜

9, wherein said step of training a initial bilingual word alignment model comprises;

training a forward initial bilingual word alignment model using the word-aligned labeled bilingual corpus; and

training a backward initial bilingual word alignment model using the word-aligned labeled bilingual corpus;

wherein said step of word-aligning a plurality of bilingual sentence pairs in a unlabeled bilingual corpus comprises;

forward-word-aligning each of said plurality of bilingual sentence pairs using said forward initial bilingual word alignment model; and

backward-word-aligning each of said plurality of bilingual sentence pairs using said backward initial bilingual word alignment model;

wherein said step of determining whether the word alignment of each of said plurality of bilingual sentence pairs is correct comprises;

calculating the intersection set between the forward-word-aligning result and the backward-word-aligning result of the bilingual sentence pair;

calculating the union set between the forward-word-aligning result and the backward-word-aligning result of the bilingual sentence pair; and

if the ratio of the element number of said intersection set to the element number of said union set is greater than a predetermined threshold, determining the word alignment of said bilingual sentence pair is correct.

11. The method for training bilingual word alignment model according to claim 10, wherein said step of training the bilingual word alignment model comprises:

training a forward bilingual word alignment model using said expended labeled bilingual corpus; and

training a backward bilingual word alignment model using said expended labeled bilingual corpus.

12. The method for training bilingual word alignment model according to claim 10, wherein said step of adding the bilingual sentence pair into the labeled bilingual corpus comprises:

if the forward-word-aligning result and the backward-word-aligning result are not identical, calculating word translation probabilities of the different portions in the forward-word-aligning result and the backward-word-aligning result, and adding the word-aligning result having higher word translation probability into the labeled bilingual corpus.

13. An apparatus for bilingual word alignment, comprising:

a model training unit configured to train a bilingual word alignment model using a word-aligned labeled bilingual corpus;

a word-aligning unit configured to word-align a plurality of bilingual sentence pairs in a unlabeled bilingual corpus using said bilingual word alignment model;

a determining unit configured to determine whether the word alignment of each of said plurality of bilingual sentence pairs is correct, and if it is correct, to add the bilingual sentence pair into the labeled bilingual corpus and to remove the bilingual sentence pair from the unlabeled bilingual corpus;

a model retraining unit configured to retrain the bilingual word alignment model using the labeled bilingual corpus expanded by said determining unit; and

a re-word-aligning unit configured to re-word-align the remaining bilingual sentence pairs in the unlabeled bilingual corpus using the retrained bilingual word alignment model.

14. The apparatus for bilingual word alignment according to claim 13, wherein said bilingual word alignment model comprises at least word translation probability, position distortion probability and word fertility probability.

15. The apparatus for bilingual word alignment according to claim 13 or 14, wherein said model training unit trains a forward bilingual word alignment model using the word-aligned labeled bilingual corpus, and trains a backward bilingual word alignment model using the word-aligned labeled bilingual corpus;

wherein said word-aligning unit forward-word-aligns each of said plurality of bilingual sentence pairs using said forward bilingual word alignment model, and backward-word-aligns each of said plurality of bilingual sentence pairs using said backward bilingual word alignment model;

wherein said determining unit calculates the intersection set between the forward-word-aligning result and the backward-word-aligning result of the bilingual sentence pair, calculates the union set between the forward-word-aligning result and the backward-word-aligning result of the bilingual sentence pair, and if the ratio of the element number of said intersection set to the element number of said union set is greater than a predetermined threshold, and determines the word alignment of said bilingual sentence pair is correct.

16. The apparatus for bilingual word alignment according to claim 15, wherein said model retraining unit retrains the forward bilingual word alignment model using said expended labeled bilingual corpus, and retrains the backward bilingual word alignment model using said expended labeled bilingual corpus.

17. The apparatus for bilingual word alignment according to claim 15, wherein said determining unit further calculates word translation probabilities of the different portions in the forward-word-aligning result and the backward-word-aligning result, if the forward-word-aligning result and the backward-word-aligning result are not identical, and adds the word-aligning result having higher word translation probability into the labeled bilingual corpus.

18. An apparatus for training bilingual word alignment model, comprising:

an initial model training unit configured to train an initial bilingual word alignment model using a word-aligned labeled bilingual corpus;

a word-aligning unit configured to word-align a plurality of bilingual sentence pairs in a unlabeled bilingual corpus using said initial bilingual word alignment model;

a determining unit configured to determine whether the word alignment of each of said plurality of bilingual sentence pairs is correct, and if it is correct, to add the bilingual sentence pair into the labeled bilingual corpus and removing the bilingual sentence pair from the unlabeled bilingual corpus;

a model training unit configured to train a bilingual word alignment model using the labeled bilingual corpus expanded by said determining unit.

19. The apparatus for training bilingual word alignment model according to claim 18, wherein said word-aligning unit further re-word-aligns the remaining bilingual sentence pairs in the unlabeled bilingual corpus using the bilingual word alignment model newly trained by said model training unit.

20. The apparatus for training bilingual word alignment model according to claim 18, wherein said bilingual word alignment model comprises at least word translation probability, position distortion probability and word fertility probability.

21. The apparatus for training bilingual word alignment model according to any one of claims 18˜

20, wherein said initial model training unit trains a forward initial bilingual word alignment model using the word-aligned labeled bilingual corpus, and trains a backward initial bilingual word alignment model using the word-aligned labeled bilingual corpus;

wherein said word-aligning unit forward-word-aligns each of said plurality of bilingual sentence pairs using said forward initial bilingual word alignment model, and backward-word-aligns each of said plurality of bilingual sentence pairs using said backward initial bilingual word alignment model;

wherein said determining unit calculates the intersection set between the forward-word-aligning result and the backward-word-aligning result of the bilingual sentence pair, calculates the union set between the forward-word-aligning result and the backward-word-aligning result of the bilingual sentence pair, and determines the word alignment of said bilingual sentence pair is correct if the ratio of the element number of said intersection set to the element number of said union set is greater than a predetermined threshold.

22. The apparatus for training bilingual word alignment model according to claim 21, wherein said model training unit trains a forward bilingual word alignment model using said labeled bilingual corpus expended by said determining unit, and trains a backward bilingual word alignment model using said labeled bilingual corpus expended by said determining unit.

23. The apparatus for training bilingual word alignment model according to claim 21, wherein said determining unit further calculates word translation probabilities of the different portions in the forward-word-aligning result and the backward-word-aligning result, if the forward-word-aligning result and the backward-word-aligning result are not identical, and adds the word-aligning result having higher word translation probability into the labeled bilingual corpus.