METHODS FOR AUTOMATIC GENERATION OF PARALLEL CORPORA

• US 20150248401A1
• Filed: 12/31/2014
• Published: 09/03/2015
• Est. Priority Date: 02/28/2014
• Status: Active Grant

• Current Assignee: PayPal, Inc. (PayPal Holdings, Inc.)
• Original Assignee: Hassan Sawaf
• Inventors: Ruvini, Jean-David, Sawaf, Hassan, Barnes, Derek
• Granted Patent: US 9,881,006 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06F 40/42 Data-driven translation
• CPC Class Codes: G06F 40/44 Statistical methods, e.g. probability models
• CPC Class Codes: G06F 40/45 Example-based machine translation; Alignment
• CPC Class Codes: G06F 40/49 using very large corpora, e.g. the web
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G06Q 30/0601 Electronic shopping [e-shopping]

Abstract

A method of forming parallel corpora comprises receiving sets of items in first language and second languages, each of the sets having one or more associated descriptions and metadata. The metadata is collected from the two sets of items and are aligned using the metadata. The aligned metadata are mapped from the first language to the second language for each of the sets. The descriptions of two items are fetched and the structural similarity of the descriptions is measured to assess whether two items are likely to be translations of each other. For mapped items with structurally similar descriptions, the mapped item descriptions are formed into respective sentences in first language and in the second language. The sentences are parallel corpora which may be used to translate an item from the first language to the second language, and also to train a machine translation system.

Claims

1. A computer implemented method comprising:

receiving sets of item listings in a first language and sets of item listings in a second language, each of the item listings in the sets of item listings comprising one or more descriptions and metadata;

collecting the metadata from the sets of item listings and aligning the sets of item listings using the metadata;

mapping the aligned sets of item listings from the first language to the second language for each of the sets of item listings;

fetching the descriptions of the mapped aligned sets of item listings and measuring the structural similarity of the fetched descriptions of the mapped aligned sets of item listings to assess whether mapped aligned sets of item listings are likely to be translations of each other, and

for pairs of mapped aligned sets of item listings having structurally similar descriptions, forming the descriptions of the mapped aligned sets of item listings into respective sentences in the first language and in the second language.

2. The method of claim 1 wherein the sentences in the first language and in the second language comprise parallel corpora, the method further comprising using the parallel corpora to translate an item listing from the first language to the second language.

3. The method of claim 1, further comprising sorting the metadata based on one of a bar code, an item identifier, an item title, a Universal Product Code (UPC) aspect name, a European Article Number (EAN) aspect name, a UPC aspect value. an EAN aspect value, an International Standard Book Number (ISBN), a category identifier, a site identifier, a seller identifier, a picture of the item, or an auction end date.

4. The method of claim 1 wherein the metadata is collected by one of an API to a web-based metadata collection module or a by stand-alone metadata collection module.

5. The method of claim 1 wherein measuring the structural similarity of the mapped aligned sets of item listings comprises using metadata to align the item titles and the item descriptions.

6. The method of claim 4 wherein the structural similarity of the item descriptions is based on a hypertext markup language (HTML) code of the item descriptions.

7. The method of claim 1 wherein respective pairs of the sentences in the first language and in the second language comprise parallel corpora, the method further comprising using the parallel corpora to train a machine translation system.

8. One or more computer-readable hardware storage device having embedded therein a set of instructions which, when executed by one or more processors of a computer, causes the computer to execute operations comprising:

receiving sets of item listings in a first language and sets of item listings in a second language, each of the item listings in the sets of item listings comprising one or more descriptions and metadata;

collecting the metadata from the sets of item listings and aligning the sets of item listings using the metadata;

mapping the aligned sets of item listings from the first language to the second language for each of the sets of item listings;

fetching the descriptions of the mapped aligned sets of item listings and measuring the structural similarity of the fetched descriptions of the mapped aligned sets of item listings to assess whether mapped aligned sets of item listings are likely to be translations of each other; and

for pairs of mapped aligned sets of item listings having structurally similar descriptions, forming the descriptions of the mapped aligned sets of item listings into respective sentences in the first language and in the second language.

9. The one or more computer-readable hardware storage device of claim 8 wherein the sentences in the first language and in the second language comprise parallel corpora, the method further comprising using the parallel corpora to translate an item listing from the first language to the second language.

10. The one or more computer-readable hardware storage device of claim 8, further comprising sorting the metadata based on one of a bar code, an item identifier, an item title, a Universal Product Code (UPC) aspect name, a European Article Number (EAN) aspect name, a UPC aspect value. an EAN aspect value, an International Standard Book Number (ISBN), a category identifier, a site identifier, a seller identifier, a picture of the item, or an auction end date.

11. The one or more computer-readable hardware storage device of claim 8 wherein the metadata is collected by one of an API to a web-based metadata collection module or a by stand-alone metadata collection module.

12. The one or more computer-readable hardware storage device of claim 8 wherein measuring the structural similarity of the mapped aligned sets of item listings comprises using metadata to align the item titles and the item descriptions.

13. The one or more computer-readable hardware storage device of claim 11 wherein the structural similarity of the item descriptions is based on a hypertext markup language (HTML) code of the item descriptions.

14. The one or more computer-readable hardware storage device of claim 8 wherein respective pairs of the sentences in the first language and in the second language comprise parallel corpora, the method further comprising using the parallel corpora to train a machine translation system.

15. One or more hardware processors configured to include:

a receiver module to receive sets of item listings in a first language and sets of item listings in a second language, each of the item listings in the sets of item listings comprising one or more descriptions and metadata;

metadata collection module to collect the metadata from the sets of item listings and aligning the sets of item listings using the metadata;

a mapping module to map the aligned sets of item listings from the first language to the second language for each of the sets of item listings;

a description fetch module to fetch the descriptions of the mapped aligned sets of item listings and measuring the structural similarity of the fetched descriptions of the mapped aligned sets of item listings to assess whether mapped aligned sets of item listings are likely to be translations of each other; and

a sentence formation module to form, for pairs of mapped aligned sets of item listings having structurally similar descriptions, the descriptions of the mapped aligned sets of item listings into respective sentences in the first language and in the second language.

16. The one or more hardware processors of claim 15 wherein the sentences in the first language and in the second language comprise parallel corpora, the method further comprising using the parallel corpora to translate an item listing from the first language to the second language.

17. The one or more hardware processors of claim 15 further configured to include a sort module to sort the metadata based on one of a bar code, an item identifier, an item title, a Universal Product Code (UPC) aspect name, a European Article Number (EAN) aspect name, a UPC aspect value, an EAN aspect value, an International Standard Book Number (ISBN), a category identifier, a site identifier, a seller identifier, a picture of the item, or an auction end date.

18. The one or more hardware processors of claim 15 wherein the metadata is collected by one of an API to a web-based metadata collection module or a by stand-alone metadata collection module.

19. The one or more hardware processors of claim 15 wherein measuring the structural similarity of the mapped aligned sets of item listings comprises using metadata to align the item titles and the item descriptions.

20. The one or more hardware processors of claim 18 wherein the structural similarity of the item descriptions is based on a hypertext markup language (HTML) code of the item descriptions.