Method for building parallel corpora

• US 20080262826A1
• Filed: 04/20/2007
• Published: 10/23/2008
• Est. Priority Date: 04/20/2007
• Status: Active Grant

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Pacull, Francois
• Granted Patent: US 7,949,514 B2
• Time in Patent Office: Days
• US Class Current: 704/3
• CPC Class Codes: G06F 40/44 Statistical methods, e.g. probability models
• CPC Class Codes: G06F 40/45 Example-based machine translation; Alignment

Abstract

A method for identifying documents for enriching a statistical translation tool includes retrieving a source document which is responsive to a source language query that may be specific to a selected domain. A set of text segments is extracted from the retrieved source document and translated into corresponding target language segments with a statistical translation tool to be enriched. Target language queries based on the target language segments are formulated. Sets of target documents responsive to the target language queries are retrieved. The sets of retrieved target documents are filtered, including identifying any candidate documents which meet a selection criterion that is based on co-occurrence of a document in a plurality of the sets. The candidate documents, where found, are compared with the retrieved source document for determining whether any of the candidate documents match the source document. Matching documents can then be stored and used at their turn in a training phase for enriching the translation tool.

Claims

1. A method for identifying documents for enriching a statistical:

translation tool comprising;

retrieving at least one source document which is responsive to a source language query;

for each retrieved source document;

extracting a set of text segments from the retrieved source document;

translating the extracted text segments into target language segments with a statistical translation tool to be enriched;

formulating target language queries based on the target language segments;

for each of a plurality of the target language queries, retrieving a set of target documents responsive to the target language query;

filtering the sets of retrieved target documents that are responsive to the target language queries, the filtering including identifying candidate documents which meet a selection criterion that is based on co-occurrence of a target document in a plurality of the sets; and

comparing the candidate documents with the retrieved source document for determining whether any of the candidate documents match the source document.

2. The method of claim 1, further comprising, where a candidate document is determined to match the source document, enriching the translation tool with aligned text fragments from matched source and target documents.

3. The method of claim 2, further comprising:

utilizing the enriched translation tool to translate additional extracted text segments for retrieving additional target documents responsive to target language queries.

4. The method of claim 1, wherein the source language query relates to a particular domain in which the statistical training tool is to be enriched.

5. The method of claim 1, wherein the retrieving at least one source document which is responsive to the source language query includes inputting the query to a search engine for retrieving web-based documents.

6. The method of claim 1, wherein the extracting of the set of text segments from the retrieved source document comprises automatically extracting text segments throughout the document.

7. The method of claim 1, wherein the extracting of the set of text segments from the retrieved source document comprises extracting at least a preselected minimum number of text segments from the retrieved source document.

8. The method of claim 7, wherein the extracted text segments together comprise less than the entire document.

9. The method of claim 1, wherein the extracting of the set of text segments from the retrieved source document comprises extracting text segments spaced from each other by document text.

10. The method of claim 1, wherein the extracting of the set of text segments from the retrieved source document comprises extracting text segments substantially at random from at least a portion of the retrieved source document.

11. The method of claim 1, wherein the identifying of candidate documents which meet a selection criterion comprises identifying target documents which are retrieved as being responsive to at least a preselected minimum quantity of the target language queries.

12. The method of claim 11, wherein the identifying of candidate documents which meet a selection criterion comprises identifying any target documents which are retrieved as being responsive to at least two of the target language queries.

13. The method of claim 1, wherein the retrieving target documents responsive to the target language queries includes inputting the target language queries to a search engine for retrieving web-based documents.

14. The method of claim 1, wherein the filtering of the retrieved target documents that are responsive to the target language queries further comprises selecting a subset of documents most responsive to the query based on a ranking provided by a search engine which retrieves the target documents.

15. The method of claim 1, wherein the extracting of the set of text segments from the retrieved source document, translating the extracted text segments into target language segments, formulating target language queries based on the target language segments, retrieving target documents responsive to the target language queries, and filtering the retrieved target documents that are responsive to the target language queries, are computer implemented.

16. The method of claim 1, wherein the text segments are selected from phrases, sentences, groups of at least two words, and combinations and multiples thereof.

17. A computer program product encoding instructions, which when executed on a computer causes the computer to perform the method of claim 1.

18. A system for enriching a statistical translation tool comprising:

memory which stores instructions for;

retrieving at least one source document which is responsive to a source language query;

for each retrieved source document, extracting a set of text segments from the retrieved source document;

translating the extracted text segments into target language segments with a statistical translation tool;

formulating target language queries based on the target language segments;

retrieving target documents responsive to the target language queries;

filtering the retrieved target documents that are responsive to the target language queries, the filtering including identifying candidate documents from the target documents that are responsive to a preselected minimum amount of queries;

comparing the candidate documents with the retrieved source document for identifying whether any of the candidate documents match the source document; and

enriching the translation tool with aligned text fragments from matching source and target documents; and

a processor which executes the instructions.

19. A method for enriching a statistical translation tool comprising:

for each of a plurality of source documents in a target language;

extracting a set of text segments from the source document;

translating the extracted text segments into target language segments with a statistical translation tool;

formulating target language queries, each query being based on one of the target language segments;

retrieving target documents responsive to the target language queries; and

filtering the retrieved target documents that are responsive to the target language queries, the filtering including identifying candidate documents from among the retrieved target documents which are meet a selection criterion, the selection criterion being based on a measure of the queries to which a document is retrieved as being responsive; and

comparing the candidate documents which meet the selection criterion with the retrieved source document for identifying whether any of the candidate documents match the source document; and

enriching the translation tool with aligned text fragments from the matched source and target documents.