Transliterating semitic languages including diacritics

• US 8,612,206 B2
• Filed: 12/08/2009
• Issued: 12/17/2013
• Est. Priority Date: 12/08/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Chalabi, Achraf, Grees, Hany, Ashour, Mostafa, Mohammed, Roaa
• Primary Examiner(s): AZAD, ABUL K
• Application Number:
  US12/633,748
• Publication Number:
  US 20110137635A1
• Time in Patent Office: 1,470 Days
• Field of Search: 704 2- 8
• US Class Current: 704/8
• CPC Class Codes: G06F 40/53 Processing of non-Latin text kana-to-kanji conversion G06F40/129; vowelisation G06F40/232

Abstract

The present disclosure describes a system and method of transliterating Semitic languages with support for diacritics. An input module receives and pre-processes Romanized character and forwards the pre-processed Romanized characters to a transliteration engine. The transliteration engine selects candidate transliteration rules, applies the rules, and scores and ranks the results for output. To optimize search for candidate transliteration rules, the transliteration engine may apply word-stemming strategies to process inflections indicated by affixes. The present disclosure further describes optimizations as pre-processing emphasis text, caching, dynamic transliteration rule pruning, and buffering/throttling input. The system and methods are suitable for multiple applications including but not limited to web applications, windows applications, client-server applications and input method editors such as those via Microsoft Text Services Framework TSF™.

Claims

1. A method of transliterating text, the method comprising:

receiving a Romanized input text of a natural language that comprises a native character set that includes diacritics;

setting a threshold probability;

selecting at least one candidate transliteration rule in response to determining that a probability that the at least one candidate transliteration rule should apply is at least equal to the threshold probability;

applying each selected candidate transliteration rule to the Romanized input text to transliterate the Romanized input text into at least one corresponding candidate diacritized text in the native character set of the natural language;

computing a confidence score for each candidate diacritized text;

ranking each candidate diacritized text based at least on the computed confidence scores; and

outputting at least one candidate diacritized text based at least on the ranking.

2. The method of claim 1, wherein:

the receiving of the Romanized text comprises pre-processing the received Romanized input text to detect emphasis text in the received Romanized input text, store the detected emphasis text, and remove the detected emphasis text from the received Romanized input text; and

the outputting of the at least one candidate diacritized text comprises restoring the stored detected emphasis text to the at least one output candidate diacritized text.

3. The method of claim 1, wherein the at least one candidate transliteration rule is selected from a transliteration model comprising at least one transliteration rule and a corresponding respective probability that the transliteration rule may apply.

4. The method of claim 3, wherein the selecting of the at least one candidate transliteration rule comprises selecting at least two candidate transliteration rules to apply to the Romanized input text, and further comprising dynamically pruning at least one selected candidate transliteration rule during run time processing of the Romanized input text.

5. The method of claim 3, wherein the natural language is Arabic, and the transliteration model is based at least on a first corpus of text comprised of a set of Romanized Arabic texts and a second corpus of text comprised of a set of Romanized Arabic texts mapped in parallel to corresponding transliterated Arabic texts.

6. The method of claim 1, wherein:

the at least one candidate transliteration rule comprises at least one constraint to limit applicability of the transliteration rule;

the receiving of the Romanized input text comprises;

identifying at least one word from the received Romanized input text; and

extracting a word stem of the identified at least one word by removing affixes from the identified at least one word;

the selecting of the at least one candidate transliteration rule is based at least on the extracted word stem; and

the applying of the selected at least one candidate transliteration rule comprises applying the at least one constraint and the selected at least one candidate transliteration rule to the extracted word stem.

7. The method of claim 6, wherein the outputting of the at least one candidate diacritized text comprises transliterating the removed affixes and reaffixing the translated removed affixes to the transliterated extracted word stem.

8. The method of claim 1, wherein the computed confidence scores are based at least on a language model comprising a word model and a character model.

9. The method of claim 8, wherein the natural language is Arabic and the language model is based at least on an Arabic language corpus.

10. The method of claim 1, wherein the receiving of the Romanized input text is from a web application and the outputting of the at least one candidate diacritized text is to the web application.

11. The method of claim 1, wherein the receiving of the Romanized input text is from a windowed application and the outputting of the at least one candidate diacritized text is to the windowed application.

12. The method of claim 1, wherein the receiving of the Romanized input text is from an input method editor and the outputting of the at least one candidate diacritized text is to the input method editor.

13. A method of inputting text into a transliteration engine, the method comprising:

receiving input text;

buffering at least some of the received input text;

detecting a first condition;

terminating the buffering based at least on the detecting of the first condition;

attempting to detect a second condition;

setting a threshold probability;

forwarding the buffered input text to the transliteration engine if the second condition is detected, wherein at least one transliteration rule is applied to the input text in response to determining that a probability that the at least one candidate transliteration rule should apply is at least equal to the threshold probability; and

recommencing buffering if the second condition is not detected.

14. The method of claim 13 wherein the detecting of the first condition comprises detecting a whitespace character in the received input text.

15. The method of claim 13 wherein the second condition comprises an indication from the transliteration engine that the transliteration engine is ready to receive input.

16. The method of claim 15 wherein the detecting of the first condition comprises receiving a notification from the transliteration engine that the transliteration engine is ready to receive input.

17. The method of claim 13, wherein the received input text is received via a tool bar.

18. A system to transliterate text, the system comprising:

a first processor,

a computer readable memory containing computer executable instructions, the instructions to perform acts that include;

receiving a Romanized input text of a natural language that comprises a native character set that includes diacritics;

selecting at least one candidate transliteration rule in response to determining that an absolute probability that the at least one candidate transliteration rule should apply is at least equal to a threshold probability;

applying each selected candidate transliteration rule to the Romanized input text to transliterate the Romanized input text into at least one corresponding candidate diacritized text in the native character set of the natural language;

computing an accuracy score for each candidate diacritized text;

ranking each candidate diacritized text based at least on the computed accuracy scores; and

outputting at least one candidate diacritized text based at least on the ranking.

19. The system of claim 18 wherein the first processor is hosted on a standalone system.

20. The system of claim 18 wherein:

the natural language is Arabic, and the transliteration model is based at least on a first corpus of text comprised of a set of Romanized Arabic texts and a second corpus of text comprised of a set of Romanized Arabic texts mapped in parallel to corresponding transliterated Arabic texts, and

the first and second corpus are centrally stored on a server.