MINING TRANSLITERATIONS FOR OUT-OF-VOCABULARY QUERY TERMS

• US 20100185670A1
• Filed: 01/09/2009
• Published: 07/22/2010
• Est. Priority Date: 01/09/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Krishnan, Saravanan, Udupa, Uppinakuduru Raghavendra
• Granted Patent: US 8,332,205 B2
• Time in Patent Office: Days
• US Class Current: 707/776
• CPC Class Codes: G06F 16/3335 Syntactic pre-processing, e.g. stopword elimination, stemming
• CPC Class Codes: G06F 16/3337 Translation of the query language, e.g. Chinese to English
• CPC Class Codes: G06F 40/129 Handling non-Latin characters, e.g. kana-to-kanji conversion

Abstract

An approach is described for using a query expressed in a source language to retrieve information expressed in a target language. The approach uses a translation dictionary to convert terms in the query from the source language to appropriate terms in the target language. The approach determines viable transliterations for out-of-vocabulary (OOV) query terms by retrieving a body of information based on an in-vocabulary component of the query, and then mining the body of information to identify the viable transliterations for the OOV query terms. The approach then adds the viable transliterations to the translation dictionary. The retrieval, mining, and adding operations can be repeated one or more or times.

Claims

1. A method for retrieving information, implemented by an information retrieval system, comprising:

receiving a query that includes an in-vocabulary component and an out-of-vocabulary (OOV) component, the in-vocabulary component comprising at least one term that is included in a translation dictionary, and the OOV component comprising at least one term that is not included in the translation dictionary;

identifying a body of information associated with the in-vocabulary component of the query, using the translation dictionary;

performing mining analysis to extract at least one viable transliteration associated with the OOV component of the query from the body of information;

updating the translation dictionary to include said at least one viable transliteration, to provide an updated translation dictionary; and

identifying another body of information associated with the in-vocabulary component of the query, using the updated translation dictionary.

2. The method of claim 1, further comprising repeating, at least once, said performing mining analysis, said updating, and said identifying of another body of information.

3. The method of claim 1, wherein the query is expressed in a source language and the body of information is expressed in a target language.

4. The method of claim 3, wherein the source language and the target language have different respective orthographies.

5. The method of claim 3, wherein the source language and the target language use different respective phonetic alphabets.

6. The method of claim 1, wherein the body of information comprises at least one document that is selected from a collection of documents based on the in-vocabulary component of the query.

7. The method of claim 1, wherein the body of information comprises a plurality of candidate words, and wherein, for each candidate word under consideration and for each OOV term under consideration, the mining analysis comprises:

determining a transliteration measure between the candidate word under consideration and the OOV term under consideration; and

determining that the candidate word under consideration is a viable transliteration of the OOV term under consideration if the transliteration measure satisfies a matching criterion.

8. The method of claim 7, wherein said determining of the transliteration measure is performed using a statistical transliteration similarity model.

9. The method of claim 1, further comprising supplementing said mining analysis by performing machine transliteration.

10. The method of claim 9, wherein the machine transliteration is applied to an OOV term under consideration if said mining analysis is unsuccessful in generating a viable transliteration for the OOV term under consideration.

11. An information retrieval system for retrieving information, comprising:

a data store providing a translation dictionary for correlating terms in a source language to corresponding terms in a target language; and

a transliteration processing module configured to convert queries in the source language to respective counterparts in the target language, the queries encompassing a type of query that includes at an in-vocabulary component and an out-of-vocabulary (OOV) component, the in-vocabulary component comprising at least one term that is included in the translation dictionary, and the OOV component comprising at least one term that is not included in the translation dictionary, the transliteration processing module comprising;

an in-vocabulary determination module configured to determine at least one in-vocabulary translation associated with the in-vocabulary component of the query;

a mining module configured to identify, within a body of information, at least one viable transliteration associated with the OOV component of the query, the body of information being identified based on said at least one in-vocabulary translation provided by the in-vocabulary determination module; and

an updating module configured to add said at least one viable transliteration to the translation dictionary to provide an updated translation dictionary.

12. The information retrieval system of claim 11, further comprising a document retrieval module, the document retrieval module being configured to retrieve the body of information.

13. The information retrieval system of claim 12, wherein the body of information comprises at least one document, and wherein the document retrieval module is configured to select said at least one document from a collection of documents based on said at least one in-vocabulary translation.

14. The information retrieval system of claim 11, wherein the information retrieval system is further configured to identify another body of information associated with the in-vocabulary component of the query, using the updated translation dictionary.

15. The document retrieval system of claim 11, wherein the source language and the target language have different respective orthographies.

16. The document retrieval system of claim 11, wherein the source language and the target language use different respective phonetic alphabets.

17. A computer-readable medium for storing computer-readable instructions, the computer-readable instructions providing a mining module when executed by one or more processing devices, the computer-readable instructions comprising:

logic configured to identify at least one viable transliteration for an out-of-vocabulary (OOV) term within a query by mining an identified body of information, wherein the identified body of information is retrieved based on an in-vocabulary component of the query,

wherein the OOV term corresponds to a term of the query that is not presently included in a translation dictionary, and the in-vocabulary component corresponds at least one term that is included in the translation dictionary.

18. The computer-readable medium of claim 17, wherein the body of information comprises at least one document that is selected from a collection of documents based on the in-vocabulary component of the query.

19. The computer-readable medium of claim 17, wherein the body of information comprises a plurality of candidate words, and wherein said logic configured to identify comprises:

measure-computing logic configured to determine a transliteration measure between a candidate word under consideration and the OOV term; and

comparison logic configured to determine that the candidate word under consideration is a viable transliteration of the OOV term if the transliteration measure satisfies a matching criterion.

20. The computer-readable medium of claim 19, wherein said measure-computing logic is configured to determine the transliteration measure using a statistical transliteration similarity model.