Discriminative training for language modeling

• US 20060277033A1
• Filed: 06/01/2005
• Published: 12/07/2006
• Est. Priority Date: 06/01/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Suzuki, Hisami, Gao, Jianfeng
• Granted Patent: US 7,680,659 B2
• Time in Patent Office: Days
• US Class Current: 704/10
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/197 Probabilistic grammars, e.g. word n-grams

Abstract

A method of training language model parameters trains discriminative model parameters in the language model based on a performance measure having discrete values.

Claims

1. A method of training language model parameters, the method comprising:

training discriminative model parameters in the language model based on a performance measure having discrete values.

2. The method of claim 1 wherein training discriminative model parameters comprises selecting a feature to include in the language model.

3. The method of claim 2 wherein selecting a feature to include in the language model comprises:

determining a discriminative model parameter for each of a set of candidate features;

determining a performance measure for each feature using the respective discriminative model parameter; and

selecting a feature to include in the language model based on the performance measure of at least one feature.

4. The method of claim 3 wherein determining a performance measure for a feature comprises determining a performance measure for each of a plurality of training samples and combining the performance measures from the plurality of training samples to form a sample risk and wherein selecting a feature based on the performance measure comprises selecting a feature based on a sample risk.

5. The method of claim 4 wherein selecting a feature based on a sample risk comprises determining a difference between the sample risk and a sample risk determined using a baseline model.

6. The method of claim 4 wherein selecting a feature further comprises determining a similarity between performance measures determined for the plurality of training samples and performance measures determined for the plurality of training samples using features that were previously selected for the language model.

7. The method of claim 1 wherein training a discriminative model parameter comprises:

for each possible value of the discriminative model parameter, determining a discriminative language model score for each of a set of candidate word sequences;

selecting a candidate word sequence for each possible value of the discriminative model parameter based on the discriminative language model scores; and

using the selected candidate word sequences to select a value for the discriminative model parameter.

8. The method of claim 7 further comprising, performing the steps of determining a discriminative language model score for each of a set of candidate word sequences and selecting a candidate word sequence for each possible value of the discriminative model parameter for each of a set of training samples.

9. The method of claim 8 wherein using the selected candidate word sequences comprises determining a performance measure for each selected candidate word sequence, combining performance measures for multiple training samples to form a sample risk for each value of the discriminative model parameter and using the sample risks to select a value for the discriminative model parameter.

10. A computer-readable medium having computer-executable instructions for performing steps comprising:

selecting a feature for inclusion in a discriminative language model;

selecting a separate word sequence for each of a plurality of different possible values of a model parameter associated with the selected feature;

for each value of the model parameter, using the respective selected word sequence to generate a performance measure that is associated with the value of the model parameter; and

using the performance measures to select a value of the model parameter to store in the discriminative language model.

11. The computer-readable medium of claim 10 wherein selecting a word sequence comprises scoring each of a plurality of candidate word sequences using the value of the model parameter and the selected feature and selecting the word sequence with the best score.

12. The computer-readable medium of claim 11 wherein scoring each of the plurality of candidate word sequences further comprises using the values of model parameters and features that have previously been included in the discriminative language model.

13. The computer-readable medium of claim 10 further comprising repeating the steps of selecting a word sequence and generating a performance measure for each of a plurality of training samples.

14. The computer-readable medium of claim 13 wherein using the performance measure to select a value comprises combining performance measures of each training sample for each value of the model parameter to form a sample risk for each value and using the sample risks to select a value.

15. The computer-readable medium of claim 10 wherein selecting a feature comprises:

determining a plurality of sample risks from a plurality of candidate features, with one sample risk being associated with each candidate feature;

determining a baseline sample risk from a baseline model; and

using the plurality of sample risks and the baseline sample risk to select a feature from the plurality of candidate features.

16. The computer-readable medium of claim 15 wherein using the plurality of sample risks and the baseline sample risk comprises generating a score for each feature, each score being based on the difference between the sample risk associated with the feature and the baseline sample risk.

17. The claim 16 wherein generating a score further comprises determining a measure of the interference between the feature and features that have previously been inserted into the discriminative language model.

18. A method of selecting features for a discriminative language model, the method comprising:

for each of a set of candidate features, determining a difference between a performance measure associated with a discriminative language model that uses the feature and a performance measure associated with a discriminative language model that does not use the feature;

using each difference to score each candidate feature; and

selecting a candidate feature based on the scores.

19. The method of claim 18 wherein using each difference to score each candidate feature further comprises:

determining an interference score for the feature, the interference score indicating the similarity in the performance of a discriminative language model that uses the feature and a discriminative language model that does not use the feature; and

using the interference score and the difference to score the feature.

20. The method of claim 18 wherein determining the performance measure for the discriminative model that uses the feature comprises:

determining a plurality of performance measures using different values for a model parameter associated with the feature;

selecting a value for the model parameter based on the plurality of performance measures; and

using the selected value of the model parameter when determining the performance measure for the discriminative model that uses the feature.