Creating taxonomies and training data in multiple languages

• US 20040122660A1
• Filed: 12/20/2002
• Published: 06/24/2004
• Est. Priority Date: 12/20/2002
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Gates, Stephen C., Cheng, Keh-Shin Fu
• Application Number:
  US10/324,919
• Publication Number:
  US 20040122660A1
• Time in Patent Office: Days
• US Class Current: 704/10
• CPC Class Codes: G06F 40/237 Lexical tools
• CPC Class Codes: G06F 40/45 Example-based machine translation; Alignment

Abstract

The problem of creating of taxonomies of objects, particularly objects that can be represented as text in various languages, and categorizing such objects is addressed by a method for taking the training documents generated in a first language, translating it to a target language, and then generating from a plurality of training documents one or more sets of features representing one or more categories in the target language. The method includes the steps of: forming a first list of items such that each item in the first list represents a particular training document having an association with one or more elements related to a particular category; developing a second list from the first list by deleting one or more candidate documents which satisfy at least one deletion criterion; translating the documents in the second list from the source language to the target language, and extracting the one or more sets of features from the translated second list using one or more feature selection criteria.

Claims

1. A method of creating a taxonomy and categorization system in a target language based on a set of training documents in a source language comprising the steps of:

selecting a source set of training documents in said source language, said set representing one or more categories;

translating said source set of training documents into a target set of target language training documents; and

extracting a set of differentiating features for each category from said target set.

2. A method according to claim 1, in which one or more members of said target set are removed from said target set, before said step of extracting a set of differentiating features, according to at least one removal criterion.

3. A method according to claim 2, in which said removal criterion is that one or more of said members of said target set are too dissimilar to other members of said target set by an amount that exceeds a removal threshold.

4. A method according to claim 1, further comprising a step of:

grouping at least one subset of said set of categories into at least one broader supercategory including said subset.

5. A method according to claim 4, further comprising a step of:

reducing overlap between categories within said broader supercategory.

6. A method according to claim 5, further comprising a step of sequentially comparing pairs of categories with the highest similarity and merging pairs of categories that satisfy a merge criterion.

7. A computer system for creating a categorization system in a target language based on a set of training data in a source language, comprising a processing unit for processing data and a storing unit for storing data, in which said processing unit contains instructions for executing a method comprising:

selecting a source set of training documents in said source language;

translating said source set of training documents into a target set of target language training documents; and

extracting a set of differentiating features, corresponding to a set of categories, from said target set.

8. A system according to claim 7, in which some members of said target set are removed from said target set, before said step of extracting a set of differentiating features, according to at least one removal criterion.

9. A system according to claim 8, in which said removal criterion is that one or more of said members of said target set are too dissimilar to other members of said target set by an amount that exceeds a removal threshold.

10. A system according to claim 7, further comprising a step of:

grouping at least one subset of said set of categories into at least one broader category including said subset.

11. A system according to claim 10, further comprising a step of:

reducing overlap between categories within said broader category.

12. A system according to claim 11, further comprising a step of sequentially comparing pairs of categories with the highest similarity and merging pairs of categories that satisfy a merge criterion.

13. A system according to claim 8, in which said step of removing some members of said target set is effected by a method further comprising:

selecting a set of potential categories;

selecting a set of training data;

eliminating some members of said set of training data; and

extracting differentiating features characteristic of an nth category that differentiate the nth category from other categories.

14. An article of manufacture in computer readable form comprising means for performing a method for operating a computer system having a program, said method comprising the steps of:

selecting a source set of training documents in said source language;

translating said source set of training documents into a target set of target language training documents; and

extracting a set of differentiating features, corresponding to a set of categories, from said target set.

15. An article of manufacture according to claim 14, in which:

some members of said target set, before said step of extracting a set of differentiating features, are removed from said target set according to at least one criterion.

16. A method according to claim 15, in which said removal criterion is that one or more of said members of said target set are too dissimilar to other members of said target set by an amount that exceeds a removal threshold.

17. A system according to claim 15, further comprising a step of:

grouping at least one subset of said set of categories into at least one broader category including said subset.

18. A system according to claim 17, further comprising a step of:

reducing overlap between categories within said broader category.

19. A system according to claim 18, further comprising a step of sequentially comparing pairs of categories with the highest similarity and merging pairs of categories that satisfy a merge criterion.

20. A system according to claim 15, in which said step of removing some members of said target set is effected by a method further comprising:

selecting a set of potential categories;

selecting a set of training data;

eliminating some members of said set of training data; and

extracting differentiating features characteristic of an nth category that differentiate the nth category from other categories.