Unified treatment of data-sparseness and data-overfitting in maximum entropy modeling

• US 20070100624A1
• Filed: 11/03/2005
• Published: 05/03/2007
• Est. Priority Date: 11/03/2005
• Status: Active Grant

• Current Assignee: Robert Bosch Corporation (Robert Bosch GmbH), Robert Bosch GmbH
• Original Assignee: Robert Bosch Corporation (Robert Bosch GmbH)
• Inventors: Zhao, Lin, Weng, Fuliang
• Granted Patent: US 8,700,403 B2
• Time in Patent Office: Days
• US Class Current: 704/257
• CPC Class Codes: G06F 40/216 using statistical methods
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/197 Probabilistic grammars, e.g. word n-grams

Abstract

A method of statistical modeling is provided which includes constructing a statistical model and incorporating Gaussian priors during feature selection and during parameter optimization for the construction of the statistical model.

Claims

1. A method of statistical modeling, comprising:

constructing a statistical model; and

incorporating Gaussian priors during feature selection and during parameter optimization for the construction of the statistical model.

2. The method of claim 1, wherein the statistical model includes a conditional maximum entropy model.

3. The method of claim 1, wherein feature selection includes a selective gain (SCG) computation method.

4. The method of claim 1, further comprising:

calculating a Gaussian distribution for each parameter of the statistical model.

5. The method of claim 1, further comprising:

calculating a Gaussian distribution for each selected feature of the statistical model.

6. A method for modeling spoken language for a conversational dialog system, comprising:

modeling dependency relations of the spoken language via a probabilistic dependency model;

incorporating Gaussian priors during feature selection and during parameter optimization;

parsing a sequence of words, the parsing including systematically searching through pairs of head words bottom-up using a chart parsing technique; and

at each step in the search, computing the probabilistic scores for each pair based on the probabilistic dependency model and keeping best candidate pairs for each region.

7. The method of claim 6, wherein the probabilistic dependency model includes a conditional maximum entropy model.

8. The method of claim 6, wherein feature selection includes a selective gain (SCG) computation method.

9. The method of claim 6, further comprising:

calculating a Gaussian distribution for each parameter of the probabilistic dependency model.

10. The method of claim 6, further comprising:

calculating a Gaussian distribution for each selected feature of the probabilistic dependency model.

11. A spoken language understanding module arrangement for use in a conversational dialog system, the arrangement comprising:

a speech recognizer to recognize speech as a sequence of words;

a part-of-speech tagger to tag the sequence;

a statistical dependency parser to form dependency structures for the sequence;

a semantic mapper to map grammatical features to the sequence;

a topic classifier to classify the sequence into at least one semantic category; and

a dialog manager to interpret meaning of the sequence in a context of a conversation.

12. The spoken language understanding module arrangement of claim 11, wherein the parser systematically searches through pairs of head words bottom-up using a chart parsing technique, and at each step in the search, computes probabilistic scores for each pair based on a dependency model.

13. The spoken language understanding module arrangement of claim 12, wherein the dependency model includes a conditional maximum entropy model.

14. The spoken language understanding module arrangement of claim 13, wherein the conditional maximum entropy model includes a conditional maximum entropy model trained with Gaussian priors during parameter optimization.

15. The spoken language understanding module arrangement of claim 13, wherein the conditional maximum entropy model includes a conditional maximum entropy model trained with Gaussian priors during feature selection and during parameter optimization.

16. The spoken language understanding module arrangement of claim 15, wherein the feature selection includes a selective gain (SCG) computation method.