Identifying documents which form translated pairs, within a document collection

• US 20070033001A1
• Filed: 08/03/2005
• Published: 02/08/2007
• Est. Priority Date: 08/03/2005
• Status: Active Grant

• Current Assignee: SDL PLC (RWS Holdings Plc)
• Original Assignee: Language Weaver, Inc. (RWS Holdings Plc)
• Inventors: Marcu, Daniel, Muslea, Ion, Knight, Kevin
• Granted Patent: US 7,813,918 B2
• Time in Patent Office: Days
• US Class Current: 704/3
• CPC Class Codes: G06F 40/45 Example-based machine translation; Alignment

Abstract

A training system for text to text application. The training system finds groups of documents, and identifies automatically similar documents in the groups which are similar. The automatically identified documents can then be used for training of the text to text application. The comparison uses reduced size versions of the documents in order to minimize the amount of processing.

Claims

1. A method, comprising:

obtaining a group of documents;

determining reduced size versions of said documents; and

comparing said reduced size versions, to determine documents that represent similar information; and

using said documents that represent similar information for training for a text-to-text application.

2. A method as in claim 1, further comprising using said documents that represent similar information to train a machine translation system.

3. A method as in claim 1, further comprising changing an order of information within said reduced size versions, prior to said comparing.

4. A method as in claim 3, further comprising sorting said reduced size versions.

5. A method as in claim 1, further comprising carrying out a rough translation of documents in said group which are in a second language, to form a group of translated documents, and comparing said group of translated documents to other documents, prior to said determining.

6. A method as in claim 1 wherein said determining reduced size versions comprises forming vectors indicative of said documents, and comparing said vectors.

7. A method as in claim 1, wherein said determining reduced size versions comprises comparing words in the documents to specified dictionaries of words, and defining the documents in terms of information about the words in the dictionaries.

8. A method as in claim 7, wherein said reduced versions include keys representing positions of words in the dictionaries.

9. A method as in claim 8, further comprising changing an order of said keys prior to said comparing.

10. A system comprising:

a database, including a group of documents;

a processor that determines reduced size versions of said documents and compares said reduced size versions, to determine documents within the group that represent similar information; and

a text to text application module, using said documents that represent similar information for training for a text-to-text application.

11. A system as in claim 10, wherein said text-to-text application is a machine translation system.

12. A system as in claim 10, wherein said text-to text application module further operates to change an order of information within said reduced size versions, prior to said comparing.

13. A system as in claim 12, wherein said text-to-text application sorts said reduced size versions.

14. A system as in claim 10, wherein said text-to-text application carries out a rough translation of documents in said group which are in a second language, to form a group of translated documents, and compares said group of translated documents to other documents, prior to said determining documents that represent similar information.

15. A system as in claim 10 wherein said text-to-text application forms vectors indicative of said documents, and compares said vectors.

16. A system as in claim 10, further comprising a plurality of word dictionaries, having a plurality of words therein, and wherein said text-to-text application determines reduced size versions by comparing words in the documents to said dictionaries of words, and defines the documents in terms of information about the words in the dictionaries.

17. A system as in claim 16, wherein said reduced versions include keys representing positions of words in the dictionaries.

18. A system as in claim 17, further comprising changing an order of said keys prior to said comparing.

19. A method, comprising:

obtaining a first group of documents in a first language, and a second group of documents in a second language;

carrying out a rough translation of said documents in said second language, to form a third group of translated documents, that have been translated to said first language;

determining reduced size versions of said first and third groups of documents; and

comparing said reduced size versions, to determine documents that represent similar information.

20. A method as in claim 19, further comprising using said documents that represent similar information to train a text to text application system.

21. A method as in claim 19, further comprising changing an order of information within said reduced size versions, prior to said determining.

22. A method as in claim 21, further comprising sorting said reduced size versions.

23. A method, comprising:

obtaining a group of documents that includes documents that are in at least a first language and a second language;

Determining reduced size versions of at least some of said documents; and

comparing said reduced size versions, to determine a first document in said first language that represents similar information to a second document in a second language.

24. A method as in claim 23, further comprising using said documents that represent similar information for training for a text-to-text application.

25. A method as in claim 23, further comprising changing an order of information within said reduced size versions, prior to said comparing.

26. A method as in claim 25, further comprising sorting said reduced size versions.

27. A method as in claim 23, further comprising carrying out a rough translation of documents in said group which are in said second language, to form a group of translated documents, and comparing said group of translated documents to other documents, prior to said determining.