Bilingual corpus update method, bilingual corpus update apparatus, and recording medium storing bilingual corpus update program

• US 10,354,646 B2
• Filed: 08/29/2017
• Issued: 07/16/2019
• Est. Priority Date: 09/21/2016
• Status: Active Grant

• Current Assignee: Panasonic Intellectual Property Management Co., Ltd. (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Intellectual Property Management Co., Ltd. (Panasonic Holdings Corporation)
• Inventors: Fujiwara, Nanami, Yamauchi, Masaki, Imade, Masahiro
• Primary Examiner(s): Sarpong, Akwasi M
• Application Number:
  US15/688,934
• Publication Number:
  US 20180082681A1
• Time in Patent Office: 686 Days
• Field of Search: 704243, 704236, 704 2, 704 9
• CPC Class Codes: G06F 40/44 Statistical methods, e.g. probability models
• CPC Class Codes: G06F 40/45 Example-based machine translation; Alignment
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/197 Probabilistic grammars, e.g. word n-grams
• CPC Class Codes: G10L 2015/0636 Threshold criteria for the updating

Abstract

A third sentence obtained by replacing a first phrase of a first sentence with a second phrase is input, and it is judged whether a third phrase is included in a first database including at least a phrase used in written text. If the third phrase is not included, a first evaluation value in the first database is calculated for a seventh phrase obtained by replacing the second phrase of the third phrase with a sixth phrase. It is judged whether the third phrase is included in a second database including at least a phrase used in spoken text and whether a second evaluation value calculated from the first evaluation value satisfies a predetermined condition. If the third phrase is included, and the second evaluation value satisfies the predetermined condition, the third sentence and the second sentence as a pair are added to a bilingual corpus.

Claims

1. A method for updating a bilingual corpus, the bilingual corpus including a plurality of pairs, each composed of a sentence described in a first language and a translated sentence described in a second language, the bilingual corpus including a first sentence described in the first language and a second sentence described in the second language as a pair, the second sentence being a translated sentence corresponding to the first sentence, the method comprising:

inputting a third sentence obtained by replacing a first phrase among a plurality of phrases constituting the first sentence with a second phrase;

judging whether a third phrase is included in a first database, the third phrase including at least the second phrase and a fourth phrase immediately anterior to the second phrase in the third sentence or the second phrase and a fifth phrase immediately posterior to the second phrase in the third sentence, the first database including at least a phrase used in written text;

calculating, on the basis of the first database, a first evaluation value in the first database for a seventh phrase obtained by replacing the second phrase of the third phrase with a sixth phrase if it is judged that the third phrase is not included in the first database, the sixth phrase being different from the second phrase;

judging whether the third phrase is included in a second database and judging whether a second evaluation value calculated on the basis of the first evaluation value satisfies a predetermined condition, the second database including at least a phrase used in spoken text, the phrase used in the spoken text being associated with an occurrence frequency in the second database of the phrase used in the spoken text; and

adding the third sentence and the second sentence as a pair to the bilingual corpus if it is judged that the third phrase is included in the second database and that the second evaluation value satisfies the predetermined condition.

2. The method according to claim 1, wherein

the third sentence is generated by replacing the first phrase with the second phrase included in a third database, and the third database associates a phrase with a phrase identical in meaning to and different in expression from the phrase.

3. The method according to claim 1, wherein

the second database is generated on the basis of a phrase used in a social networking service.

4. The method according to claim 1, further comprising:

adding the third sentence and the second sentence as the pair to the bilingual corpus if it is judged that the third phrase is included in the first database.

5. The method according to claim 1, further comprising:

excluding the sixth phrase of the seventh phrase from judgment and judging whether the seventh phrase is present in the first database if it is judged that the third phrase is not included in the first database, and not adding the third sentence to the bilingual corpus if the seventh phrase is not present in the first database.

6. The method according to claim 5, further comprising:

using an n-word n-gram including the second phrase as the third phrase and a database for an n-gram language model as the first database, judging whether the n-gram is present in the database for the n-gram language model, and adding the third sentence and the second sentence as the pair to the bilingual corpus if the n-gram is present in the database for the n-gram language model.

7. The method according to claim 5, further comprising:

using an n-word n-gram including the second phrase as the third phrase and a database for an n-gram language model as the first database, obtaining an occurrence probability or an occurrence frequency of the n-gram from the database for the n-gram language model, and adding the third sentence and the second sentence as the pair to the bilingual corpus if a third evaluation value calculated from the occurrence probability or the occurrence frequency of the n-gram is not less than a predetermined threshold.

8. The method according to claim 6, further comprising:

judging whether the n-gram, the second phrase of which is excluded from judgment, is present in the database for the n-gram language model if it is judged that the third phrase is not included in the first database, and not adding the third sentence to the bilingual corpus if the n-gram, the second phrase of which is excluded from judgment, is not present in the database for the n-gram language model.

9. The method according to claim 6, further comprising:

obtaining an occurrence probability or an occurrence frequency of the n-gram, the second phrase of which is excluded from judgment, from the database for the n-gram language model if it is judged that the third phrase is not included in the first database, and not adding the third sentence to the bilingual corpus if a fourth evaluation value calculated from the occurrence probability or the occurrence frequency of the n-gram, the second phrase of which is excluded from judgment, is less than a predetermined threshold.

10. The method according to claim 6, further comprising:

judging whether a surface expression anterior and posterior portion composed of the second phrase, the fourth phrase, and the fifth phrase of the n-gram is present in the second database if the seventh phrase is present in the first database, and adding the third sentence and the second sentence as the pair to the bilingual corpus if the surface expression anterior and posterior portion is present in the second database, and a surface expression anterior- and posterior-side evaluation value calculated from an occurrence probability or an occurrence frequency of the n-gram, the second phrase of which is excluded from judgment, is not less than a predetermined threshold.

11. The method according to claim 10, further comprising:

judging whether a surface expression anterior word portion composed of the second phrase and the fourth phrase of the n-gram or a surface expression posterior word portion composed of the second phrase and the fifth phrase is present in the second database if the seventh phrase is present in the first database, and adding the third sentence and the second sentence as the pair to the bilingual corpus if the surface expression anterior word portion or the surface expression posterior word portion is present in the second database, and a surface expression one-side evaluation value calculated from the occurrence probability or the occurrence frequency of the n-gram, the second phase of which is excluded from judgment, is not less than a predetermined threshold.

12. The method according to claim 11, wherein

the surface expression anterior- and posterior-side evaluation value is a value obtained by multiplying the first evaluation value obtained from the occurrence probability or the occurrence frequency of the n-gram, the second phrase of which is excluded from judgment, by a predetermined first weighting factor, and

the surface expression one-side evaluation value is a value obtained by multiplying the first evaluation value by a second weighting factor less than the first weighting factor.

13. The method according to claim 11, further comprising:

judging whether a part-of-speech expression anterior and posterior portion composed of the second phrase of the n-gram, an anterior part-of-speech portion obtained by replacing the fourth phrase with a part of speech of the fourth phrase, and a posterior part-of-speech portion obtained by replacing the fifth phrase with a part of speech of the fifth phrase is present in the second database if the surface expression anterior and posterior portion is not present in the second database, if the surface expression anterior- and posterior-side evaluation value is less than the predetermined threshold, if neither the surface expression anterior word portion nor the surface expression posterior word portion is present in the second database, or if the surface expression one-side evaluation value is less than the predetermined threshold, and adding the third sentence and the second sentence as the pair to the bilingual corpus if the part-of-speech expression anterior and posterior portion is present in the second database, and a part-of-speech expression anterior- and posterior-side evaluation value calculated from the occurrence probability or the occurrence frequency of the n-gram, the second phrase of which is excluded from judgment, is not less than a predetermined threshold.

14. The method according to claim 13, further comprising:

judging whether a part-of-speech expression anterior word portion composed of the second phrase of the n-gram and the anterior part-of-speech portion obtained by replacing the fourth phrase with the part of speech of the fourth phrase or a part-of-speech expression posterior word portion composed of the second phrase and the posterior part-of-speech portion obtained by replacing the fifth phrase with the part of speech of the fifth phrase is present in the second database if the surface expression anterior and posterior portion is not present in the second database, if the surface expression anterior- and posterior-side evaluation value is less than the predetermined threshold, if neither the surface expression anterior word portion nor the surface expression posterior word portion is present in the second database, or if the surface expression one-side evaluation value is less than the predetermined threshold, and adding the third sentence and the second sentence as the pair to the bilingual corpus if the part-of-speech expression anterior word portion or the part-of-speech expression posterior word portion is present in the second database, and a part-of-speech expression one-side evaluation value calculated from the occurrence probability or the occurrence frequency of the n-gram, the replacing portion of which is excluded from judgment, is not less than a predetermined threshold.

15. The method according to claim 14, wherein

the surface expression anterior- and posterior-side evaluation value is a value obtained by multiplying the first evaluation value obtained from the occurrence probability or the occurrence frequency of the n-gram, the second phrase of which is excluded from judgment, by a predetermined first weighting factor,

the surface expression one-side evaluation value is a value obtained by multiplying the first evaluation value by a second weighting factor less than the first weighting factor,

the part-of-speech expression anterior- and posterior-side evaluation value is a value obtained by multiplying the first evaluation value by a third weighting factor less than the second weighting factor, and

the part-of-speech expression one-side evaluation value is a value obtained by multiplying the first evaluation value by a fourth weighting factor less than the third weighting factor.

16. The method according to claim 14, further comprising:

judging whether the second phrase is present in the second database if the part-of-speech expression anterior and posterior portion is not present in the second database, if the part-of-speech expression anterior- and posterior-side evaluation value is less than the predetermined threshold, if neither the part-of-speech expression anterior word portion nor the part-of-speech expression posterior word portion is present in the second database, or if the part-of-speech expression one-side evaluation value is less than the predetermined threshold, and adding the third sentence and the second sentence as the pair to the bilingual corpus if the second phrase is present in the second database, and a replacing portion evaluation value calculated from the occurrence probability or the occurrence frequency of the n-gram, the second phrase of which is excluded from judgment, is not less than a predetermined threshold.

17. The method according to claim 16, wherein

the surface expression anterior- and posterior-side evaluation value is a value obtained by multiplying the first evaluation value obtained from the occurrence probability or the occurrence frequency of the n-gram, the second phrase of which is excluded from judgment, by a predetermined first weighting factor,

the surface expression one-side evaluation value is a value obtained by multiplying the first evaluation value by a second weighting factor less than the first weighting factor,

the part-of-speech expression anterior- and posterior-side evaluation value is a value obtained by multiplying the first evaluation value by a third weighting factor less than the second weighting factor,

the part-of-speech expression one-side evaluation value is a value obtained by multiplying the first evaluation value by a fourth weighting factor less than the third weighting factor, and

the replacing portion evaluation value is a value obtained by multiplying the first evaluation value by a fifth weighting factor less than the fourth weighting factor.

18. The method according to claim 10, wherein

the second database is a database which includes more colloquial expressions than the database for the n-gram language model.

19. An apparatus for updating a bilingual corpus, the bilingual corpus including a plurality of pairs, each composed of a sentence described in a first language and a translated sentence described in a second language, the bilingual corpus including a first sentence described in the first language and a second sentence described in the second language as a pair, the second sentence being a translated sentence corresponding to the first sentence, the apparatus comprising:

an inputter which inputs a third sentence obtained by replacing a first phrase among a plurality of phrases constituting the first sentence with a second phrase;

a first database judge which judges whether a third phrase is included in a first database, the third phrase including at least the second phrase and a fourth phrase immediately anterior to the second phrase in the third sentence or the second phrase and a fifth phrase immediately posterior to the second phrase in the third sentence, the first database including at least a phrase used in written text;

a calculator which calculates, on the basis of the first database, a first evaluation value in the first database for a seventh phrase obtained by replacing the second phrase of the third phrase with a sixth phrase if it is judged that the third phrase is not included in the first database, the sixth phrase being different from the second phrase;

a second database judge which judges whether the third phrase is included in a second database and judges whether a second evaluation value calculated on the basis of the first evaluation value satisfies a predetermined condition, the second database including at least a phrase used in spoken text, the phrase used in the spoken text being associated with an occurrence frequency in the second database of the phrase used in the spoken text; and

an outputter which adds the third sentence and the second sentence as a pair to the bilingual corpus if it is judged that the third phrase is included in the second database and that the second evaluation value satisfies the predetermined condition.

20. A Non-transitory recording medium storing a program for causing a computer to function as an apparatus for updating a bilingual corpus,

the bilingual corpus including a plurality of pairs, each composed of a sentence described in a first language and a translated sentence described in a second language, the bilingual corpus including a first sentence described in the first language and a second sentence described in the second language as a pair,

the second sentence being a translated sentence corresponding to the first sentence, the program causing the computer to execute:

inputting a third sentence obtained by replacing a first phrase among a plurality of phrases constituting the first sentence with a second phrase;

judging whether a third phrase is included in a first database, the third phrase including at least the second phrase and a fourth phrase immediately anterior to the second phrase in the third sentence or the second phrase and a fifth phrase immediately posterior to the second phrase in the third sentence, the first database including at least a phrase used in written text;

calculating, on the basis of the first database, a first evaluation value in the first database for a seventh phrase obtained by replacing the second phrase of the third phrase with a sixth phrase if it is judged that the third phrase is not included in the first database, the sixth phrase being different from the second phrase;

judging whether the third phrase is included in a second database and judging whether a second evaluation value calculated on the basis of the first evaluation value satisfies a predetermined condition,

the second database including at least a phrase used in spoken text, the phrase used in the spoken text being associated with an occurrence frequency in the second database of the phrase used in the spoken text;

and adding the third sentence and the second sentence as a pair to the bilingual corpus if it is judged that the third phrase is included in the second database and that the second evaluation value satisfies the predetermined condition.