UNSUPERVISED LEARNING USING GLOBAL FEATURES, INCLUDING FOR LOG-LINEAR MODEL WORD SEGMENTATION

• US 20110144992A1
• Filed: 12/15/2009
• Published: 06/16/2011
• Est. Priority Date: 12/15/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Cherry, Colin Andrew, Toutanova, Kristina N., Poon, Hoifung
• Granted Patent: US 8,909,514 B2
• Time in Patent Office: Days
• US Class Current: 704/243
• CPC Class Codes: G10L 15/18 using natural language modelling

Abstract

Described is a technology for performing unsupervised learning using global features extracted from unlabeled examples. The unsupervised learning process may be used to train a log-linear model, such as for use in morphological segmentation of words. For example, segmentations of the examples are sampled based upon the global features to produce a segmented corpus and log-linear model, which are then iteratively reprocessed to produce a final segmented corpus and a log-linear model.

Claims

1. In a computing environment, a method performed on at least one processor, comprising, performing unsupervised learning on examples in training data, including processing the examples to extract global features, in which the global features are based on a plurality of the examples, and learning a model from the global features.

2. The method of claim 1 wherein processing the examples comprises processing language data.

3. The method of claim 2 wherein the global features include lexicon length features for one or more lexicons of morphemes, wherein other features include morpheme features and context features extracted from words, with overlapping context and morpheme values, and wherein processing the language data comprises obtaining a morphological segmentation of the words.

4. The method of claim 2 wherein processing the language data comprises determining a neighborhood corresponding to an example, including at least one negative example.

5. The method of claim 3 wherein processing the language data comprises imposing one exponential prior to penalize segmentation of words into longer segments, and another exponential prior to penalize over-segmentation.

6. The method of claim 2 wherein processing the language data comprises providing a morphological segmentation of words in the language data.

7. The method of claim 3 wherein processing the language data comprises sampling based on a neighborhood corresponding to each example.

8. The method of claim 7 further comprising, using deterministic annealing in the sampling.

9. The method of claim 1 wherein learning the model comprises learning a log-linear model, and further comprising, outputting the log-linear model.

10. In a computing environment, a method performed on at least one processor, comprising:

(a) processing unlabeled examples of words into an interim segmented corpus and an interim log-linear model;

(b) using the interim log-linear model to reprocess the interim segmented corpus into a revised segmented corpus and a revised log-linear model;

(c) iterating until a stop criterion is met by returning to step (b) with the revised segmented corpus being used as the interim corpus and the revised log-linear model being used as the interim model; and

(d) when the stop criterion is met, outputting the log-linear model for use in morphological segmentation.

11. The method of claim 10 wherein processing the unlabeled examples of words comprises maintaining a lexicon set of morphemes, and weight information for morpheme and context features, where contexts describe characters adjacent a particular morpheme.

12. The method of claim 11 wherein maintaining the lexicon set comprises maintaining a lexicon for prefixes, maintaining a lexicon for stems and maintaining a lexicon for suffixes.

13. The method of claim 10 wherein processing the unlabeled examples of words comprises computing overlapping context and morpheme features.

14. The method of claim 13 further comprising, using the overlapping context and morpheme features and their weights to sample a neighborhood for an example.

15. The method of claim 14 further comprising, sampling at least one negative example for contrastive estimation.

16. One or more computer-readable media having computer-executable instructions, which when executed perform steps, comprising:

(a) processing unlabeled examples of words into global features, in which the global features are based on a plurality of the examples;

(b) sampling segmentations of the examples to produce an interim segmented corpus and an interim log-linear model that uses the global features;

c) using the interim log-linear model to reprocess the interim segmented corpus into a revised segmented corpus and a revised log-linear model;

d) iterating by returning to step (c) until a stop criterion is met, with the revised segmented corpus being used as the interim corpus and the revised log-linear model being used as the interim model.

17. The one or more computer-readable media of claim 16 having further computer-executable instructions, comprising, imposing one exponential prior to penalize segmentation of words into longer segments, and another exponential prior to penalize over-segmentation.

18. The one or more computer-readable media of claim 16 having further computer-executable instructions, comprising, sampling at least one negative example for contrastive estimation.

19. The one or more computer-readable media of claim 16 wherein processing the unlabeled examples of words comprises maintaining a lexicon set of morphemes, and a set of weights for morphemes and contexts that each describe characters adjacent a particular morpheme.

20. The one or more computer-readable media of claim 16 having further computer-executable instructions comprising, after step (d) using the revised log-linear model as a final log-linear model to determine the most probable segmentation of a new word.