Unified treatment of data-sparseness and data-overfitting in maximum entropy modeling

• US 8,700,403 B2
• Filed: 11/03/2005
• Issued: 04/15/2014
• Est. Priority Date: 11/03/2005
• Status: Active Grant

• Current Assignee: Robert Bosch Corporation (Robert Bosch GmbH), Robert Bosch GmbH
• Original Assignee: Robert Bosch GmbH
• Inventors: Weng, Fuliang, Zhao, Lin
• Primary Examiner(s): GUERRA-ERAZO, EDGAR X
• Application Number:
  US11/266,867
• Publication Number:
  US 20070100624A1
• Time in Patent Office: 3,085 Days
• Field of Search: 704/2, 704/9, 704/10, 704/224, 704/240, 704/256.2, 704/3, 704/4, 704/7, 704/276, 704/235, 704/243, 7042561-2568
• US Class Current: 704/256.2
• CPC Class Codes: G06F 40/216 using statistical methods
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/197 Probabilistic grammars, e.g. word n-grams

Abstract

A method of statistical modeling is provided which includes constructing a statistical model and incorporating Gaussian priors during feature selection and during parameter optimization for the construction of the statistical model.

Claims

1. A computer-implemented method for modeling spoken language for a conversational dialog system, comprising:

modeling, by a computer processor, dependency relations of the spoken language via a probabilistic dependency model;

incorporating, by the computer processor, Gaussian priors during feature selection and during parameter optimization;

parsing, by the computer processor, a sequence of words, the parsing including systematically searching through pairs of head words bottom-up using a chart parsing technique; and

at each step in the search, computing, by the computer processor, the probabilistic scores for each pair based on the probabilistic dependency model and keeping n best candidate pairs for each region;

wherein the dependency model is decomposed into a model for a first sub-region, a second sub-region, and a component which includes a last dependency relation that connects the first and second sub-regions, with an adjustment of mutual information between the last dependency relation and the first and second sub-regions.

2. The method of claim 1, further comprising:

incorporating a Gaussian priors during feature selection and during parameter optimization for construction of a statistical model, wherein the incorporated Gaussian priors have a plurality of parameters between which calculated means and variances vary.

3. The method of claim 2, wherein the statistical model includes a conditional maximum entropy model.

4. The method of claim 2, wherein feature selection includes a selective gain (SCG) computation method.

5. The method of claim 2, further comprising:

calculating a Gaussian distribution for each parameter of the statistical model.

6. The method of claim 2, further comprising:

calculating a Gaussian distribution for each selected feature of the statistical model.

7. The method of claim 1, wherein the probabilistic dependency model includes a conditional maximum entropy model.

8. The method of claim 1, wherein feature selection includes a selective gain (SCG) computation method.

9. The method of claim 1, further comprising:

calculating a Gaussian distribution for each parameter of the probabilistic dependency model.

10. The method of claim 1, further comprising:

calculating a Gaussian distribution for each selected feature of the probabilistic dependency model.

11. The method of claim 1, wherein:

the dependency model for a region L_i,j extending between word i and word j is calculated as P( L_i,j)=P( L_i,k)*P( L_k+1,j)*P(L_i,j| L_i,k, L_k+1,j)*e^{MI( Li,k, Lk+1,j)};

(i, k) is a first sub-region of the region L_i,j;

(k+1, j) is a second sub-region of the region L_i,j;

L_i,j is a triple of 1_i,j, W^i,k, and W^k+1,j;

1_i,j is a last dependency relation that connects the first and second sub-regions;

W^i,k is a head word representing the first sub-region; and

W^k+1,j is a head word representing the second sub-region.

12. A dialog system comprising:

a computer processor programmed to execute a spoken language understanding software module arrangement, the software module arrangement comprising;

a speech recognizer module, which, when executed by the processor, causes the processor to recognize speech as a sequence of words;

a part-of-speech tagger module, which, when executed by the processor, causes the processor to tag the sequence;

a statistical dependency parser, which the processor is configured to use to form dependency structures for the sequence;

a semantic mapper module, which, when executed by the processor, causes the processor to map grammatical features to the sequence;

a topic classifier module, which, when executed by the processor, causes the processor to classify the sequence into at least one semantic category; and

a dialog manager module, which, when executed by the processor, causes the processor to interpret meaning of the sequence in a context of a conversation;

wherein the statistical dependency parser is based on dependency relations of the spoken language via a probabilistic dependency model, and wherein the dependency model is decomposed into a model for a first sub-region, a second sub-region, and a component which includes a last dependency relation that connects the first and second sub-regions, with an adjustment of mutual information between the last dependency relation and the first and second sub-regions.

13. The spoken language understanding module arrangement of claim 12, wherein the parser systematically searches through pairs of head words bottom-up using a chart parsing technique, and at each step in the search, computes probabilistic scores for each pair based on a dependency model.

14. The spoken language understanding module arrangement of claim 13, wherein the dependency model includes a conditional maximum entropy model.

15. The spoken language understanding module arrangement of claim 14, wherein the conditional maximum entropy model includes a conditional maximum entropy model trained with Gaussian priors during parameter optimization.

16. The spoken language understanding module arrangement of claim 14, wherein the conditional maximum entropy model includes a conditional maximum entropy model trained with Gaussian priors during feature selection and during parameter optimization.

17. The spoken language understanding module arrangement of claim 16, wherein the feature selection includes a selective gain (SCG) computation method.

18. A computer-implemented method for modeling spoken language for a conversational dialog system, comprising:

modeling, by a computer processor, dependency relations of the spoken language via a probabilistic dependency model;

incorporating, by the computer processor, Gaussian priors during feature selection and during parameter optimization;

parsing, by the computer processor, a sequence of words, the parsing including systematically searching through pairs of head words bottom-up using a chart parsing technique; and

at each step in the search, computing, by the computer processor, the probabilistic scores for each pair based on the probabilistic dependency model and keeping n best candidate pairs for each region;

wherein the dependency model is decomposed into a model for a left branch of a parse tree, a right branch of the parse tree, a conditional probability of a top level that connects the left and right branches, and an adjustment of mutual information between the top level and the left and right branches.