WORD BREAKER FROM CROSS-LINGUAL PHRASE TABLE

• US 20140309986A1
• Filed: 04/11/2013
• Published: 10/16/2014
• Est. Priority Date: 04/11/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: El-Sharqwi, Mohamed Ahmed, Chalabi, Achraf Abdel-Moneim Tawfik Mahmoud
• Granted Patent: US 9,330,087 B2
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 40/268 Morphological analysis
• CPC Class Codes: G06F 40/44 Statistical methods, e.g. probability models
• CPC Class Codes: G06F 40/45 Example-based machine translation; Alignment

Abstract

Automatically creating word breakers which segment words into morphemes is described, for example, to improve information retrieval, machine translation or speech systems. In embodiments a cross-lingual phrase table, comprising source language (such as Turkish) phrases and potential translations in a target language (such as English) with associated probabilities, is available. In various examples, blocks of source language phrases from the phrase table are created which have similar target language translations. In various examples, inference using the target language translations in a block enables stem and affix combinations to be found for source language words without the need for input from human-judges or prior knowledge of source language linguistic rules or a source language lexicon.

Claims

1. A method of automatically building a word breaker for segmenting words of a source language into morphemes comprising:

accessing, at a processor, a cross-lingual phrase table comprising a plurality of source language phrases, each source language phrase having at least one target language translation;

using the cross-lingual phrase table to infer and store, for source language words from the cross-lingual phrase table, morphemes comprising stems and affixes of the words.

2. A method as claimed in claim 1 comprising accessing at the processor, probability values associated with the target language translations and using the probability values to calculate a probability associated with a stem and an affix of the source language.

3. A method as claimed in claim 1 comprising searching the cross-lingual phrase table to find blocks of two or more source language phrases which have similar target language translations.

4. A method as claimed in claim 3 comprising finding similar target language translations by finding target language translations comprising at least one identical single word.

5. A method as claimed in claim 3 comprising finding similar target language translations by finding target language translations comprising the same target language word in any of a plurality of inflected forms.

6. A method as claimed in claim 3 comprising finding similar target language translations by finding target language translations comprising two or more shared target language words.

7. A method as claimed in claim 3 comprising searching source language phrases of a block to find a stem, comprising a sequence of characters occurring in each source language phrase of the block.

8. A method as claimed in claim 7 comprising finding a plurality of affixes being other sequences of characters in the source language phrases of the block.

9. A method as claimed in claim 7 comprising inferring whether the stem is a noun or a verb by using the target language phrases.

10. A method as claimed in claim 8 comprising inferring applicability of the affixes by using the target language phrases.

11. A method as claimed in claim 1 comprising receiving a source language word to be segmented into morphemes and finding at least one combination of the stems and affixes which match the word.

12. A method as claimed in claim 11 comprising outputting a probability value associated with the at least one combination of the stems and affixes.

13. A method of automatically building a word breaker for segmenting words of a source language into morphemes comprising:

accessing, at a processor, a cross-lingual phrase table comprising a plurality of source language phrases, each source language phrase having at least one target language translation;

using the cross-lingual phrase table to infer and store, for source language words from the cross-lingual phrase table, morphemes comprising stems and affixes of the words, the inference using similar target language translations identified from the cross-lingual phrase table.

14. A method as claimed in claim 13 comprising searching the cross-lingual phrase table to find blocks of two or more source language phrases which have similar target language translations.

15. A method as claimed in claim 13 comprising accessing at the processor, probability values associated with the target language translations and using the probability values to calculate a probability associated with a stem and an affix.

16. A word breaker building system comprising:

a processor arranged to access a cross-lingual phrase table comprising a plurality of source language phrases, each source language phrase having at least one target language translation;

the processor arranged to use the cross-lingual phrase table to infer and store, for source language words from the cross-lingual phrase table, morphemes comprising stems and affixes of the words.

17. A system as claimed in claim 16 comprising a blocking component arranged to search the cross-lingual phrase table to find blocks of two or more source language phrases which have similar target language translations.

18. A system as claimed in claim 17 comprising a stem inference component arranged to search source language phrases of a block to find a stem, comprising a sequence of characters occurring in each source language phrase of the block.

19. A system as claimed in claim 18 comprising an affix inference component arranged to find a plurality of affixes being other sequences of characters in the source language phrases of the block.

20. A system as claimed in claim 16 being at least partially implemented using hardware logic selected from any one or more of:

a field-programmable gate array, a program-specific integrated circuit, a program-specific standard product, a system-on-a-chip, a complex programmable logic device.