Methods and systems for alignment of parallel text corpora

• US 9,047,275 B2
• Filed: 05/04/2012
• Issued: 06/02/2015
• Est. Priority Date: 10/10/2006
• Status: Expired due to Fees

• Current Assignee: ABBYY Production LLC (ABBYY Software)
• Original Assignee: ABBYY InfoPoisk LLC
• Inventors: Parfentieva, Tatiana, Krivenko, Anton, Zuev, Konstantin, Anisimovich, Konstantin, Selegey, Vladimir
• Primary Examiner(s): Harper, Vincent P
• Application Number:
  US13/464,447
• Publication Number:
  US 20120239378A1
• Time in Patent Office: 1,124 Days
• Field of Search: 704/4, 704/2, 704/3, 704/5, 704/7, 704/9, 704/245
• US Class Current: 1/1
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/56 Natural language generation

Abstract

Computer-implemented systems and methods align fragments of a first text with corresponding fragments of a second text, which is a translation of the first text. One preferred embodiment preliminarily divides the first and second texts into fragments; generates a hypothesis about the correspondence between the fragments of the first and second texts; performs a lexico-morphological analysis of the fragments using linguistic descriptions; performs a syntactic analysis of the fragments using linguistic descriptions and generates syntactic structures for the fragments; generates semantic structures for the fragments; and estimates the degree of correspondence between the semantic structures.

Claims

1. A computer-implemented method of aligning fragments of a first text in a first language with corresponding fragments of a second text, which is a translation of the first text into a second language, comprising:

preliminarily dividing the first and second texts into fragments;

generating a hypothesis about correspondence between at least first fragment in the first text and at least second fragment in the second text;

determining estimations reflecting correspondence between the first and the second fragments, wherein each estimation is based at least on a ratio between;

(a) a number of words in at least one of the first segment or the second segment; and

(b) a number of words in the first fragment that have a corresponding translation in the second fragment according to a normalized one-to-one dictionary;

determining a degree of correspondence between the first and the second fragments based on the estimations, including adjusting the estimations by using weight coefficients selected on the basis of heuristics or training; and

comparing the degree of correspondence to a predetermined threshold.

2. The method of claim 1, wherein the fragments are sentences.

3. The method of claim 1, wherein the hypothesis about correspondence between the fragments is based at least on one of the following:

size of the fragments, punctuation symbols, formatting, capital letters, headings, numeration of text parts or paragraphs.

4. The method of claim 1 wherein the estimations include the presence of corresponding word combinations in the first and the second fragments.

5. The method of claim 1 wherein the estimations include the presence in the first and the second fragments of at least one of the following:

dates, numbers, abbreviations, family names, or geographical names.

6. The method of claim 1 further comprising, if the degree of correspondence is sufficiently significant, then confirming the hypothesis and saving the correspondence of the first and the second fragments, otherwise, changing boundaries of at least one fragment so as to select a new hypothesis.

7. The method of claim 1 further comprising, adjusting fragment alignment using a user interface that enables a user to consolidate fragments and to modify the correspondence of fragments in the first text and in the second text.

8. A computer-implemented method of aligning fragments of a first text in a first language with corresponding fragments of a second text, which is a translation of the first text into a second language, comprising:

preliminarily dividing the first and second texts into fragments;

generating a hypothesis about correspondence between at least first fragment in the first text and at least second fragment in the second text;

performing a lexico-morphological analysis of the first and the second fragments using linguistic descriptions;

performing a syntactic analysis of the first and the second fragments using linguistic descriptions and generating a syntactic structure for the first fragment and a syntactic structure for the second fragment;

generating a semantic structure for the first fragment and a semantic structure for the second fragment, wherein the semantic structures are directional acyclic graphs with nodes that are assigned elements of semantic hierarchy;

estimating the degree of correspondence between the semantic structure for the first fragment and the semantic structure for the second fragment, wherein estimating the degree of correspondence between the semantic structures includes identifying correspondence of tree structure, deep slots, non-tree links, and semantic classes; and

if the degree of correspondence between the semantic structure for the first fragment and the semantic structure for the second fragment satisfies a predetermined threshold, saving the generated syntactic and semantic structures for the first fragment in connection with the first fragment; and

saving the generated syntactic and semantic structures for the second fragment in connection with the second fragment.

9. The method of claim 8, wherein the fragments are sentences.

10. The method of claim 8 further comprising, if the degree of correspondence is sufficiently significant, confirming the hypothesis and saving correspondence between the first and second fragments, otherwise changing boundaries of the fragment so as to select a new hypothesis.

11. The method of claim 10, wherein saving the generated syntactic and semantic structures includes saving syntactic and semantic parameters determined during the lexico-morphological analysis, the syntactic analysis and parameters of the semantic structures.

12. The method of claim 10 wherein saving correspondence between the first and second fragments further comprise saving correspondence between their syntactic structures and saving correspondence between their semantic structures.

13. The method of claim 8 further comprising indexing fragments of the first text and indexing fragments of the second text in accordance with determined correspondence of the fragments of the first and the second texts.

14. The method of claim 13 wherein indexing further comprises indexing syntactic structures and their parameters of the first text and the second text.

15. The method of claim 13 wherein indexing further comprises indexing semantic structures and their parameters of the first text and the second text.

16. The method of claim 8 wherein syntactic and semantic structures are displayed to a user with an XML-like notation.

17. The method of claim 8 further comprising, adjusting fragment alignment using a user interface that enables a user to consolidate fragments and to modify the correspondence of fragments in the first text and in the second text.

18. The method of claim 8, wherein non-tree links of the graphs reflect linguistic phenomena including, at least, ellipsis and coordination.

19. The method of claim 8, wherein estimating the correspondence between the semantic structures comprises mapping graphs to produce a set of node pairs and computing node distances for the pairs.

20. The method of claim 19, wherein estimating the correspondence between the semantic structures further comprises comparing a predetermined value to the accumulated node distances.

21. The method of claim 19, wherein the node distances are determined by the relative location of nodes in a semantic hierarchy.

22. The method of claim 8, wherein the syntactic and semantic structures, generated for the first and the second fragments, are displayed to a user with a graph-like notation.

23. A computer system for aligning fragments of a first text in a first language with corresponding fragments of a second text, which is a translation of the first text into a second language, comprising:

a processor; and

a memory coupled to the processor, the memory storing instructions, which when executed by the computer system for aligning fragments cause the system to perform a method comprising;

preliminarily dividing the first and second texts into fragments;

generating a hypothesis about correspondence between at least first fragment in the first text and at least second fragment in the second text;

performing a lexico-morphological analysis of the first and the second fragments using linguistic descriptions;

performing a syntactic analysis of the first and the second fragments using linguistic descriptions and generating a syntactic structure for the first fragment and a syntactic structure for the second fragment;

generating a semantic structure for the first fragment and a semantic structure for the second fragment, wherein the semantic structures are directional acyclic graphs with nodes that are assigned elements of semantic hierarchy;

estimating the degree of correspondence between the semantic structure for the first fragment and the semantic structure for the second fragment, wherein estimating the degree of correspondence between the semantic structures includes identifying correspondence of tree structure, deep slots, non-tree links, and semantic classes; and

if the degree of correspondence between the semantic structure for the first fragment and the semantic structure for the second fragment satisfies a predetermined threshold, saving the generated syntactic and semantic structures for the first fragment in connection with the first fragment; and

saving the generated syntactic and semantic structures for the second fragment in connection with the second fragment.

24. The system of claim 23, wherein the fragments are sentences.

25. The system of claim 23, wherein the semantic structures are directional acyclic graphs with nodes that are assigned elements of semantic hierarchy.

26. A computer-readable medium having stored thereon a sequence of instructions, which, when executed by a computer system for aligning fragments of a first text in a first language with corresponding fragments of a second text, which is a translation of the first text into a second language, causes the system to perform a method, comprising:

preliminarily dividing the first and second texts into fragments;

generating a hypothesis about correspondence between at least first fragment in the first text and at least second fragment in the second text;

performing a lexico-morphological analysis of the first and the second fragments using linguistic descriptions;

performing a syntactic analysis of the first and the second fragments using linguistic descriptions and generating a syntactic structure for the first fragment and a syntactic structure for the second fragment;

generating a semantic structure for the first fragment and a semantic structure for the second fragment, wherein the semantic structures are directional acyclic graphs with nodes that are assigned elements of semantic hierarchy;

estimating the degree of correspondence between the semantic structure for the first fragment and the semantic structure for the second fragment, wherein estimating the degree of correspondence between the semantic structures includes identifying correspondence of tree structure, deep slots, non-tree links, and semantic classes; and

if the degree of correspondence between the semantic structure for the first fragment and the semantic structure for the second fragment satisfies a predetermined threshold, saving the generated syntactic and semantic structures for the first fragment in connection with the first fragment; and

saving the generated syntactic and semantic structures for the second fragment in connection with the second fragment.

27. The computer-readable medium of claim 26, wherein the fragments are sentences.

28. The computer-readable medium of claim 26 further comprising, if the degree of correspondence is sufficiently high, confirming the hypothesis, otherwise changing boundaries of the fragment so as to select a new hypothesis.