Identifying parallel bilingual data over a network

• US 20080126076A1
• Filed: 08/07/2006
• Published: 05/29/2008
• Est. Priority Date: 08/07/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Ming, Zhou, Cheng, Niu
• Granted Patent: US 8,249,855 B2
• Time in Patent Office: Days
• US Class Current: 704/4
• CPC Class Codes: G06F 16/3332 Query translation
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 40/45 Example-based machine translation; Alignment

Abstract

A set of candidate documents, each of which may be part of a bilingual, parallel set of documents, are identified. The set of documents illustratively includes textual material in a source language. It is then determined whether parallel text can be identified. For each document in the set of documents, it is first determined whether the parallel text resides within the document itself. If not, the document is examined for links to other documents, and those linked documents are examined for bilingual parallelism with the selected documents. If not, named entities are extracted from the document and translated into the target language. The translations are used to query search engines to retrieve the parallel correspondent for the selected documents.

Claims

1. A method of identifying parallel, bilingual texts, comprising:

executing a query against a data store over a network to identify a candidate text in a source language;

determining whether the candidate text includes a link to a linked text in a target language;

if the candidate text includes a link to a linked text, retrieving the linked text;

determining whether the candidate text and the linked text are parallel, bilingual texts; and

if so, providing an output identifying the parallel, bilingual texts.

2. The method of claim 1 and further comprising:

if the candidate text does not include a link or if the linked text and the candidate text are not parallel, bilingual texts, determining whether the candidate text includes one or more words in the target language; and

if so, executing a query against the data store over the network wherein the query includes the words in the target language.

3. The method of claim 1 and further comprising:

prior to determining whether the candidate text includes a link, determining whether the candidate text includes a set of parallel, hi-lingual texts.

4. The method of claim 3 and further comprising:

determining whether the candidate text includes words in the target language;

if so, executing a target language query against a data store, over the network, using the words in the target language; and

determining whether texts returned in response to the target language query are parallel to the candidate text.

5. The method of claim 4 and further comprising:

identifying named entities in the candidate text in the source language;

translating the named entities into the target language;

executing a named entity query against a data store, over the network, using the translated named entities; and

determining whether texts returned in response to the named entity query are parallel to the candidate text.

6. The method of claim 1 wherein executing the query comprises:

generating the query using a set of predetermined translation trigger words identified as being words that, when used in the query, are likely to return candidate texts for which a translation into the target language exists.

12. The method of claim 4-07 wherein the hierarchical set of content-based processing steps comprise:

translating named entities in the selected candidate web page into the target language;

executing a named entity query over the wide area network based on the translated named entities; and

determining whether web pages returned based on the named entity query are parallel to the selected candidate web page.

7. A method of generating parallel, bi-lingual corpora, comprising:

executing a query against a data store over a wide area network to identify a set of candidate web pages in a first language;

executing a hierarchical set of content-based processing steps on a selected candidate web page to identify a possibly parallel web page in a second language that is possibly parallel to the selected candidate web page in the first language, wherein executing a hierarchical set of content-based processing steps to identify a possible parallel web page comprises;

determining whether the selected candidate web page, itself includes parallel bi-lingual texts, and

following hyperlinks in the selected candidate web page to linked web pages and determining whether the linked web pages are parallel to the selected candidate web page;

verifying that the candidate web page and the possibly parallel web page are sufficiently parallel; and

if so, outputting an indication that the candidate web page and the possibly parallel web page are parallel, bi-lingual web pages.

8. The method of claim 7 and further comprising:

repeating the steps of executing the hierarchical set of content-based processing steps, verifying that the candidate web page and the possibly parallel web page are sufficiently parallel, and outputting an indication that the candidate web page and the possibly parallel web page are parallel, hi-lingual web pages, for each web page in the set of candidate web pages.

11. The method of claim 7 wherein the hierarchical set of content-based processing steps comprise:

determining whether the selected candidate web page includes words in the second language;

if so, executing a target language query over the wide area network based on the words in the target language; and

determining whether web pages returned based on the target language query are parallel to the selected candidate web page.

13. The method of claim 7 wherein executing the query comprises:

forming the query of translation trigger words selected to identify web pages that have a corresponding translationally equivalent web page.

9-10. -10. (canceled)

14. A system for identifying parallel, bilingual documents stored in one or more data stores accessible over a network, the system comprising:

a search engine configured to execute a query over the network against the one or more data stores having a plurality of documents each defining a plurality of sentences, the search engine being further configured to retrieve a candidate document in a first language from the plurality of documents based on the query; and

a parallelism verification system that comprises;

a document scanning component configured to scan the candidate document to determine whether the candidate document includes words in a second language, the second language being different than the first language; and

a parallelism verification component configured to determine whether the words in the second language comprise parallel text, that is parallel to text, in the first language, in the candidate document.

15. The system of claim 14 wherein the parallelism verification system is configured to identify links in the candidate document to linked documents in the second language, and wherein the a parallelism verification component is configured to determine whether the words in the linked document are is parallel to the candidate document.

16. The system of claim 14 wherein the parallelism verification system comprises:

a query generation component configured to generate a query, executable by the search engine, based on the words in the second language identified by the document scanning component.

17. The system of claim 14 wherein the parallelism verification system comprises:

a named entity extraction component configured to extract named entities from the candidate document; and

a translation component configured to translate the named entities into the second language.

18. The system of claim 14 wherein the parallelism verification component comprises a sentence alignment component configured to verify parallelism based on alignment of sentences in the first and second languages.

19. The system of claim 14 wherein the parallelism verification system is configured to output parallel texts, identifying the parallel texts as parallel, bilingual texts, for storage or to another processing system.