Constrained line search optimization for discriminative training of HMMS

• US 8,239,332 B2
• Filed: 11/20/2007
• Issued: 08/07/2012
• Est. Priority Date: 11/20/2007
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Liu, Peng, Soong, Frank Kao-PingK, Jiang, Hui
• Primary Examiner(s): Gaffin, Jeffrey A
• Assistant Examiner(s): Tran, Mai T
• Application Number:
  US11/943,210
• Publication Number:
  US 20090132444A1
• Time in Patent Office: 1,722 Days
• Field of Search: 706/12
• US Class Current: 706/12
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

An exemplary method for optimizing a continuous density hidden Markov model (CDHMM) includes imposing a constraint for discriminative training, approximating an objective function as a smooth function of CDHMM parameters and performing a constrained line search on the smoothed function to optimize values of the CDHMM parameters. Various other methods, devices and systems are disclosed.

Claims

1. One or more computer-readable storage media comprising computer-executable instructions that, when executed by one or more processors, configure the one or more processors to perform acts comprising:

imposing a constraint for discriminative training, the constraint limiting a difference between an initial continuous density hidden Markov model (CDHMM) parameter value and an updated CDHMM parameter value;

approximating an objective function as a smooth function of CDHMM parameters; and

performing a constrained line search on the smoothed function to optimize values of the CDHMM parameters.

2. The one or more computer-readable storage media of claim 1, wherein the CDHMM comprises a speech recognition CDHMM.

3. The one or more computer-readable storage media of claim 1, wherein the imposing comprising imposing a constraint based on Kullback-Leibler divergence (KLD).

4. The one or more computer-readable storage media of claim 3, wherein the KLD comprises a divergence between the initial CDHMM parameter value and the updated CDHMM parameter value.

5. The one or more computer-readable storage media of claim 1, wherein the smooth function comprises a function having a first derivative with respect to a CDHMM parameter.

6. The one or more computer-readable storage media of claim 1, wherein the performing a constrained line search comprises deciding if a critical point exists.

7. The one or more computer-readable storage media of claim 6, wherein, if a critical point exists, the performing comprises performing a constrained line search along a line joining an initial point and the critical point.

8. The one or more computer-readable storage media of claim 6, wherein, if a critical point does not exist, the performing comprises performing a constrained line search along a gradient direction at an initial point.

9. The one or more computer-readable storage media of claim 1, wherein the imposing a constraint comprises imposing a distortion constraint.

10. The one or more computer-readable storage media of claim 9,wherein the distortion constraint allows for approximating a smooth model in a region near an initial point.

11. The one or more computer-readable storage media of claim 9, wherein the approximating an objective function comprises approximating the objective function as a quadratic function.

12. The one or more computer-readable storage media of claim 11, wherein the quadratic function comprises roots.

13. The one or more computer-readable storage media of claim 1, wherein the performing comprises selecting a trust region.

14. The one or more computer-readable storage media of claim 13, wherein the trust region acts to ensure accuracy of a quadratic solution to the smooth function.

15. The one or more computer-readable storage media of claim 13, wherein the selection of a positive or a negative root depends on whether a critical point exists in a selected trust region.

16. The one or more computer-readable storage media of claim 1, wherein the performing decides if a critical point does not exist or if a saddle point exists for the smooth function.

17. The one or more computer-readable storage media of claim 1, wherein the CDHMM comprises a CDHMM for pattern recognition.

18. One or more computer-readable storage media comprising computer-executable instructions that, when executed by one or more processors, configure the one or more processors to perform acts comprising:

imposing a constraint for discriminative training based on Kullback-Leibler divergence between an initial continuous density hidden Markov model (CDHMM) parameter value and an updated CDHMM parameter value;

approximating an objective function as a smooth function of CDHMM parameters; and

performing a constrained line search on the smoothed function to optimize values for the CDHMM parameters.

19. The one or more computer-readable storage media of claim 18, further comprising providing data and recognizing speech using optimized values.

20. A computing device comprising:

a processor;

memory coupled to the processor; and

control logic, maintained on the memory and operable on the processor, to impose a constraint for discriminative training, the constraint limiting a difference between an initial continuous density hidden Markov model (CDHMM) parameter value and an updated CDHMM parameter value, to approximate an objective function as a smooth function of CDHMM parameters and to perform a constrained line search on the smoothed function to optimize values for the CDHMM parameters.