Adaptive pattern learning for bilingual data mining

• US 8,275,604 B2
• Filed: 03/18/2009
• Issued: 09/25/2012
• Est. Priority Date: 03/18/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Jiang, Long, Yang, Shiquan, Zhou, Ming, Liu, Xiaohua
• Primary Examiner(s): Rider, Justin
• Application Number:
  US12/406,722
• Publication Number:
  US 20100241416A1
• Time in Patent Office: 1,287 Days
• Field of Search: None
• US Class Current: 704/4
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06F 40/45 Example-based machine translation; Alignment

Abstract

Embodiments for the adaptive learning of translation layout patterns to mine bilingual data are disclosed. In accordance with at least one embodiment, the adaptive learning of patterns to mine bilingual data includes processing a bilingual web page into a Document Object Model (DOM) tree. The embodiment further includes linking the bilingual snippet pairs of each node into a plurality bilingual snippet pairs. The embodiment also includes determining one or more best fit candidate patterns based on the plurality of translation snippets via a Support Vector Machine classifier. The embodiment additionally includes mining one or more translation pairs from the bilingual web page using the one or more best fit candidate patterns. The translation pairs are further stored in a data storage. The one or more translation pairs including at least one of a term pair, a phrase pair, or a sentence pair.

Claims

1. A system, comprising:

one or more processors; and

memory that includes a plurality of computer-executable components executable by the one or more processors, the plurality of computer-executable components comprising;

a pre-processing component to process a bilingual web page into a Document Object Model (DOM) tree that includes at least one node;

a seed mining component to link bilingual snippet pairs of the at least one node into a plurality of translation snippet pairs;

a pattern learning component to determine one or more best fit candidate patterns based on the plurality of translation snippet pairs via a Support Vector Machine (SVM) classifier;

a data mining component to mine one or more translation pairs from the bilingual web page using the one or more best fit candidate patterns; and

a data storage component to store the one or more translation pairs, wherein the one or more translation pairs including at least one of a term pair, a phrase pair, or a sentence pair.

2. The system of claim 1, wherein the pre-processing component is to further remove attribute nodes and link together content nodes of the DOM tree.

3. The system of claim 1, wherein the pre-processing component is to further generate the plurality of translation snippet pairs from inner text of the at least one node based on Unicode values of the inner text.

4. The system of claim 1, wherein the pre-processing component is to further determine whether the at least one node is a collective node, whether a percentage of snippets that do not belong to bilingual snippet pairs in the collective node does not exceed a maximum percentage, and whether the collective node includes a minimum number of non-overlapping bilingual snippet pairs.

5. The system of claim 1, wherein the seed mining component is to link bilingual snippet pairs of the at least one node into the plurality of translation snippet pairs using a bilingual dictionary or a transliteration model.

6. The system of claim 1, wherein the seed mining component is to link bilingual snippet pairs of the at least one node into the plurality of translation snippet pairs via segmenting language content of each snippet in each bilingual snippet pair into words.

7. The system of claim 1, wherein the seed mining component is to link bilingual snippet pairs of the at least one node into the plurality translation snippet pairs via at least one of stemming one or more words in each bilingual snippet pair or removing one or more stop words from each bilingual snippet pair.

8. The system of claim 1, wherein the seed mining component is to link bilingual snippet pairs of the at least one node into the plurality of translation snippet pairs by:

forming the bilingual snippet pairs into a first language collection and a second language collection, and

linking the first language and the second language collection to maximize an overall translation rate between the first language collection and the second language collection.

9. The system of claim 1, wherein the pattern learning component is to determine the one or more best fit candidate patterns by:

merging each translation snippet pair into a string, and forming one or more candidate patterns based on the strings using a PAT tree, and

selecting the one or more best fit candidate patterns from the one or more candidate patterns using the SVM classifier.

10. The system of claim 1, wherein the SVM classifier is to incorporate at least a generality measure feature, an average translation score feature, a length measure feature, and a regularity measure feature when determining the one or more best fit candidate patterns from the plurality of the translation snippet pairs.

11. The system of claim 1, data mining component is to mine one or more translation pairs from the bilingual web page by:

merging each bilingual snippet pair of the one or more nodes in the bilingual web page into a string;

transforming each of the one or more best fit candidate patterns into a standard regular expression; and

matching the one or more standard regular expressions with the strings to derive translation pairs.

12. The system of claim 1, further comprising a search engine component to retrieve one or more bilingual web pages, wherein the search engine component is to retrieve the one or more bilingual web pages via a query based on at least one of a search term or one or more trigger words, or via downloading from a web site having a particular top domain.

13. A method, comprising:

processing a bilingual web page into a Document Object Model (DOM) tree that includes at least one node;

linking bilingual snippet pairs of the at least one node into a plurality of translation snippet pairs using an alignment model, the alignment model including a bilingual dictionary or a transliteration model;

determining one or more best fit candidate patterns based on the plurality of translation snippet pairs via a Support Vector Machine (SVM) classifier;

mining one or more translation pairs from the bilingual web page using the one or more best fit candidate patterns; and

storing the one or more translation pairs into the alignment model, wherein the one or more translation pairs including at least one of a term pair, a phrase pair, or a sentence pair.

14. The method of claim 13, further comprising re-linking the bilingual snippet pairs of the at least one node into the plurality of translation snippet pairs using the alignment model that includes the one or more mined translation pairs.

15. The method of claim 13, wherein the processing includes generating the plurality of translation snippet pairs from inner text of the at least one node based on Unicode values of the inner text.

16. The method of claim 13, wherein the processing includes processing a bilingual web page into a Document Object Model (DOM) tree that includes at least one collective node by determining that a percentage of snippets that do not belong to the bilingual snippet pairs in the at least one node does not exceed a maximum percentage, and the at least one node includes a minimum number of non-overlapping bilingual snippet pairs.

17. The method of claim 13, wherein the linking includes:

segmenting language content in each bilingual snippet pair into words, the language content including of first language snippet and a second language snippet;

reducing the language content of each bilingual snippet pair by stemming one or more words in each bilingual snippet pair or removing one or more stop words from each bilingual snippet pair; and

linking the first language snippet and the second language snippet of the bilingual snippet pairs based on the alignment model to generate translation snippet pairs.

18. The method of claim 13, wherein the determining includes merging each translation snippet pair into a string, and forming one or more candidate patterns based on the strings using a PAT tree, and selecting the one or more best fit candidate patterns from the one or more candidate patterns using the SVM classifier.

19. The method of claim 13, wherein the mining includes:

merging each bilingual snippet pair of the one or more nodes in the bilingual web page into a string;

transforming each of the one or more best fit candidate patterns into a standard regular expression; and

matching the one or more standard regular expressions with the strings to derive translation pairs.

20. A computer readable storage device storing computer-executable instructions that, when executed, cause one or more processors to perform acts comprising:

processing a bilingual web page into a Document Object Model (DOM) tree that includes and at least one attribute node and at least one content node that is a child of the at least one attribute node;

removing the at least one attribute node from the DOM tree and linking the at least one content node to a root node of the DOM tree;

linking bilingual snippet pairs of the at least one content node into a plurality of translation snippet pairs using an alignment model;

determining one or more best fit candidate patterns based on the plurality of translation snippet pairs via a Support Vector Machine (SVM) classifier;

mining one or more translation pairs from the bilingual web page using the one or more best fit candidate patterns;

storing the one or more translation pairs into the alignment model, wherein the one or more translation pairs including at least one of a term pair, a phrase pair, or a sentence pair; and

re-linking the bilingual snippet pairs of the at least one node into the plurality of translation snippet pairs using the alignment model that includes the one or more mined translation pairs.