System and method for parameter estimation for pattern recognition

• US 20030055640A1
• Filed: 04/30/2002
• Published: 03/20/2003
• Est. Priority Date: 05/01/2001
• Status: Abandoned Application

• Current Assignee: Ramot University Authority For Applied Research and Industrial Development Limited
• Original Assignee: Ramot University Authority For Applied Research and Industrial Development Limited
• Inventors: Ben-Yishai, Assaf, Burshtein, David
• Application Number:
  US10/134,451
• Publication Number:
  US 20030055640A1
• Time in Patent Office: Days
• US Class Current: 704/235
• CPC Class Codes: G06F 18/295 Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models
• CPC Class Codes: G10L 15/144 Training of HMMs

Abstract

A parameter estimator for estimating a set of parameters for pattern recognition has a recognizer for receiving a training set having members. The recognizer performs recognition on the members of the training set using a current set of parameters and based upon a predetermined group of elements. A set generator associated with the recognizer generates at least one equivalence set containing recognized members of the training set, which are used by a target function determiner associated with the set generator to calculate a target function using the set of parameters. A maximizer updates the parameter set so as to maximize the calculated target function.

Claims

1. A parameter estimator for estimating a set of parameters for pattern recognition, said parameter estimator comprising:

a recognizer for receiving a training set having members and performing recognition on said members using a current set of parameters and a predetermined group of elements,

a set generator associated with said recognizer for generating at least one equivalence set comprising recognized ones of said members,

a target function determiner associated with said set generator for calculating from at least one of said equivalence sets a target function using said set of parameters, and

a maximizer associated with said target function determiner for updating said set of parameters to maximize said target function.

2. A parameter estimator according to claim 1, wherein said target function comprises a summation, over the elements of said predetermined group of elements, of a difference between:

a first summation of logarithms of probability density functions as a function of said set of parameters, and

a second summation, of logarithms of probability density functions as a function of said set of parameters, multiplied by a discrimination rate, said discrimination rate being variable between zero and one.

3. A parameter estimator according to claim 2, wherein said target function comprises

4. A parameter estimator according to claim 1, further comprising an initial estimator associated with said recognizer for calculating an initial estimate of said parameter set.

5. A parameter estimator according to claim 4, wherein said initial estimate comprises a maximum likelihood estimate.

6. A parameter estimator according to claim 3, further comprising a discrimination rate tuner associated with said target function determiner for tuning said discrimination rate within said range.

7. A parameter estimator according to claim 6, wherein said discrimination rate tuner is operable to tune said discrimination rate to a constant value for all members of said training set.

8. A parameter estimator according to claim 6, wherein, for a given member of said training set, said discrimination rate tuner is operable to tune said discrimination rate to a respective discrimination rate level associated with said member.

9. A parameter estimator according to claim 6, wherein said discrimination rate is tunable so as to optimize said parameter set according to a predetermined optimization criterion.

10. A parameter estimator according to claim 1, wherein said maximizer is further operable to feed back said updated parameter set to said recognizer.

11. A parameter estimator according to claim 10, wherein said parameter estimator comprises an iterative device.

12. A parameter estimator according to claim 1, further comprising a parameter outputter associated with said maximizer and a statistical pattern recognition system for outputting at least some of said updated parameter set.

13. A parameter estimator according to claim 12, wherein said statistical pattern recognition system comprises a speech recognition system.

14. A parameter estimator according to claim 13, wherein said speech recognition system comprises a word-spotting system.

15. A parameter estimator according to claim 12, wherein said statistical pattern recognition system includes one of a group comprising:

image recognition, decryption, communications, sensory recognition, optical, optical character recognition (OCR), natural language processing (NLP), gesture and object recognition (for machine vision), text classification, and control systems.

16. A parameter estimator according to claim 3, wherein said maximizer comprises an iterative device comprising:

an auxiliary function determiner for forming an auxiliary function associated with said target function from a current estimate of said set of parameters, and

an auxiliary function maximizer for updating said set of parameters to maximize said auxiliary function.

17. A parameter estimator according to claim 16, wherein said auxiliary function comprises a summation, over the elements of said predetermined group of elements, of a difference between:

a first summation of conditional expected value functions as a function of said set of parameters, and

a second summation, of conditional expected value functions as a function of said set of parameters, multiplied by a discrimination rate, said discrimination rate being variable between zero and one.

18. A parameter estimator according to claim 17, wherein said auxiliary function comprises

19. A parameter estimator according to claim 18, wherein said second data set comprises a complete data set.

20. A parameter estimator according to claim 18, further comprising an initial estimator associated with said maximizer for calculating an initial estimate of said parameter set.

21. A parameter estimator according to claim 18, wherein said initial estimate comprises a maximum likelihood estimate.

22. A parameter estimator according to claim 1, wherein said statistical pattern recognition system comprises a speech recognition system, said members of said training set comprise utterances, and said predetermined group of elements comprises a predetermined vocabulary of words.

23. A parameter estimator according to claim 22, wherein said recognizer comprises a Viterbi recognizer.

24. A parameter estimator according to claim 1, wherein said parameters comprise parameters of a statistical model.

25. A parameter estimator according to claim 24, wherein said statistical model comprises a hidden Markov model (HMM).

26. A parameter estimator for estimating a set of parameters for word-spotting pattern recognition, said parameter estimator comprising:

a recognizer for receiving a training set, performing recognition on said training set using a current set of parameters and a predetermined group of elements, and providing recognized transcriptions of said training set,

a target function determiner associated with said recognizer for calculating from at least one of said recognized transcriptions a target function using said set of parameters, and

a maximizer associated with said target function determiner for updating said set of parameters to maximize said target function.

27. A parameter estimator according to claim 26, wherein said target function comprises a difference between:

a logarithm of a first probability density function as a function of said set of parameters, and

a logarithm of a second probability density function as a function of said set of parameters, multiplied by a discrimination rate, said discrimination rate being variable between zero and one.

28. A parameter estimator according to claim 27, wherein said target function comprises

log p_θ

(O|W)−

λ

log p_θ

(O|Ŵ

)

wherein W is a possible transcription of said training set,

Ŵ

is a recognized transcription of said training set,

O is said training set,

λ

is said discrimination rate,

θ

is said set of parameters, and

p_θ

(.|.) is a predetermined probability density function using said set of parameters.

29. A parameter estimator according to claim 26, further comprising an initial estimator associated with said recognizer for calculating an initial estimate of said parameter set.

30. A parameter estimator according to claim 28, wherein said initial estimate comprises a maximum likelihood estimate.

31. A parameter estimator according to claim 27, further comprising a discrimination rate tuner associated with said target function determiner for tuning said discrimination rate within said range.

32. A parameter estimator according to claim 31, wherein said discrimination rate is tunable so as to optimize said parameter set according to a predetermined optimization criterion.

33. A parameter estimator according to claim 26, wherein said maximizer is further operable to feed back said updated parameter set to said recognizer.

34. A parameter estimator according to claim 33, wherein said parameter estimator comprises an iterative device.

35. A parameter estimator according to claim 26, further comprising a parameter outputter associated with said maximizer and a word-spotting pattern recognition system for outputting at least some of said updated parameter set.

36. A parameter estimator according to claim 27, wherein said maximizer comprises an iterative device comprising:

an auxiliary function determiner for forming an auxiliary function associated with said target function from a current estimate of said set of parameters, and

an auxiliary function maximizer for updating said set of parameters to maximize said auxiliary function.

37. A pattern recognizer for performing statistical pattern recognition upon an input sequence, said pattern recognizer being operable to transcribe said input sequence into an output sequence, said output sequence comprising elements from a predetermined group of elements, said pattern recognizer comprising:

a transcriber for performing said transcription according to a predetermined statistical model having a set of parameters, and

a parameter estimator for providing said set of parameters, said parameter estimator comprising;

a recognizer for receiving a training set having members and performing recognition on said members using a current set of parameters and said predetermined group of elements,

a set generator associated with said recognizer for generating at least one equivalence set comprising recognized ones of said members,

a target function determiner associated with said set generator for calculating from at least one of said equivalence sets a target function using said set of parameters, and

a maximizer associated with said target function determiner for updating said set of parameters to maximize said target function.

38. A pattern recognizer according to claim 37, wherein said target function comprises a summation, over the elements of said predetermined group of elements, of a difference between:

a first summation of logarithms of probability density functions as a function of said set of parameters, and

a second summation, of logarithms of probability density functions as a function of said set of parameters, multiplied by a discrimination rate, said discrimination rate being variable between zero and one.

39. A pattern recognizer according to claim 38, wherein said target function comprises

40. A pattern recognizer according to claim 37, further comprising an initial estimator associated with said recognizer for calculating an initial estimate of said parameter set.

41. A pattern recognizer according to claim 37, wherein said maximizer is further operable to feed back said updated parameter set to said recognizer.

42. A pattern recognizer according to claim 41, wherein said parameter estimator comprises an iterative device.

43. A pattern recognizer according to claim 39, wherein said maximizer comprises an iterative device comprising:

an auxiliary function determiner for forming an auxiliary function associated with said target function from a current estimate of said set of parameters, and

an auxiliary function maximizer for updating said set of parameters to maximize said auxiliary function.

44. A pattern recognizer according to claim 40, wherein said auxiliary function comprises a summation, over the elements of said predetermined group of elements, of a difference between:

a first summation of conditional expected value functions as a function of said set of parameters, and

a second summation, of conditional expected value functions as a function of said set of parameters, multiplied by a discrimination rate, said discrimination rate being variable between zero and one.

45. A pattern recognizer according to claim 44, wherein said auxiliary function comprises

46. A pattern recognizer according to claim 37, wherein said statistical pattern recognition comprises speech recognition.

47. A pattern recognizer according to claim 46, wherein said members of said training set comprise utterances and said predetermined group of elements comprises a predetermined vocabulary of words.

48. A pattern recognizer according to claim 47, wherein said recognizer comprises a Viterbi recognizer.

49. A pattern recognizer according to claim 37, wherein said statistical pattern recognition system includes one of a group comprising:

image recognition, decryption, communications, sensory recognition, optical character recognition (OCR), natural language processing (NLP), gesture and object recognition (for machine vision), text classification, and control systems.

50. A pattern recognizer according to claim 37, wherein said statistical model comprises a hidden Markov model (HMM).

51. A pattern recognizer according to claim 37, wherein said input sequence comprises a continuous sequence.

52. A pattern recognizer according to claim 37, wherein said output sequence comprises a continuous sequence.

53. A speech recognizer for performing statistical speech processing upon an input sequence of utterances, said speech recognizer being operable to transcribe said input sequence into an output sequence, said output sequence comprising words from a predetermined vocabulary, said speech recognizer comprising:

a transcriber for performing said transcription according to a predetermined statistical model having a set of parameters, and

a parameter estimator for providing said set of parameters, said parameter estimator comprising;

a recognizer for receiving a training set having utterances and performing recognition on said utterances using a current set of parameters and said predetermined vocabulary,

a set generator associated with said recognizer for generating at least one equivalence set comprising recognized ones of said utterances,

a target function determiner associated with said set generator for calculating from at least one of said equivalence sets a target function using said set of parameters, and

a maximizer associated with said target function determiner for updating said set of parameters to maximize said target function.

54. A speech recognizer according to claim 53, wherein said statistical model comprises a hidden Markov model (HMM).

55. A speech recognizer according to claim 53, wherein said target function comprises a summation, over the elements of said predetermined group of elements, of a difference between:

a first summation of logarithms of probability density functions as a function of said set of parameters, and

a second summation, of logarithms of probability density functions as a function of said set of parameters, multiplied by a discrimination rate, said discrimination rate being variable between zero and one.

56. A speech recognizer according to claim 55, wherein said target function comprises

57. A speech recognizer according to claim 53, further comprising an initial estimator associated with said recognizer for calculating an initial estimate of said parameter set.

58. A speech recognizer according to claim 53, wherein said maximizer is further operable to feed back said updated parameter set to said recognizer.

59. A speech recognizer according to claim 58, wherein said parameter estimator comprises an iterative device.

60. A speech recognizer according to claim 56, wherein said maximizer comprises an iterative device comprising:

an auxiliary function determiner for forming an auxiliary function associated with said target function from a current estimate of said set of parameters, and

an auxiliary function maximizer for updating said set of parameters to maximize said auxiliary function.

61. A speech recognizer according to claim 60, wherein said auxiliary function comprises a summation, over the elements of said predetermined group of elements, of a difference between:

a first summation of conditional expected value functions as a function of said set of parameters, and

a second summation, of conditional expected value functions as a function of said set of parameters, multiplied by a discrimination rate, said discrimination rate being variable between zero and one.

62. A speech recognizer according to claim 61, wherein said auxiliary function comprises

63. A speech recognizer according to claim 53, wherein said recognizer comprises a Viterbi recognizer.

64. A speech recognizer according to claim 53, further comprising a converter for converting said input sequence of utterances into a sequence of samples representing a speech waveform.

65. A speech recognizer according to claim 64, further comprising a feature extractor for extracting from said sequence of samples a feature vector for processing by said transcriber, and wherein a dimension of said feature vector is less than a dimension of said sequence of samples.

66. A speech recognizer according to claim 53, further comprising a language modeler, for providing grammatical constraints to said transcriber.

67. A speech recognizer according to claim 53, further comprising an acoustic modeler for embedding acoustic constraints into said statistical model.

68. A speech recognizer according to claim 53, wherein said input sequence comprises a continuous speech sequence.

69. A speech recognizer according to claim 53, wherein said output sequence comprises a continuous speech sequence.

70. A speech recognizer according to claim 53, wherein said utterances comprise keywords and non-keywords, and wherein said speech recognizer is further operable to identify said keywords within said input sequence.

71. A parameter estimator for estimating a set of parameters for pattern recognition, said parameter estimator comprising:

a recognizer for receiving a training set having members and performing recognition on said members using a current set of parameters and a predetermined group of elements,

a set generator associated with said recognizer for generating at least one equivalence set comprising recognized ones of said members,

a numerator calculator, associated with said set generator, operable to calculate, for a given parameter and a set of indices of training set members, a respective numerator accumulator,

a denominator calculator associated with said set generator, operable to calculate, for said given parameter and a set of indices of training set members, a respective denominator accumulator, and

an evaluator, associated with said numerator calculator and said denominator calculator, for calculating for said given parameter a quotient between

the difference between

a first numerator accumulator, calculated for said given parameter and a set of indices of training set members corresponding to a given element v, and

a second numerator accumulator, calculated for said given parameter and a set of indices of training set members corresponding to an equivalence set associated with element v, multiplied by a discrimination rate, and,

the difference between

a first denominator accumulator, calculated for said given parameter and said set of indices of training set members corresponding to element v, and

a second denominator accumulator, calculated for said given parameter and said set of indices of training set members corresponding to said equivalence set associated with element v, multiplied by a discrimination rate,

said discrimination rate being variable between zero and one.

72. A parameter estimator according to claim 71, wherein said parameters comprise parameters of a statistical model.

73. A parameter estimator according to claim 72, wherein said statistical model comprises a hidden Markov model (HMM).

74. A parameter estimator according to claim 72, wherein said statistical model includes one of a group comprising:

Gaussian distribution, and Gaussian mixture distribution.

75. A parameter estimator according to claim 71, wherein said numerator calculator is operable to calculate said numerator accumulator for said given parameter in accordance with a maximum likelihood estimate of a numerator accumulator of said parameter.

76. A parameter estimator according to claim 71, wherein said quotient is

77. A parameter estimator according to claim 71, wherein said denominator calculator is operable to calculate said denominator accumulator for said given parameter in accordance with a maximum likelihood estimate of a denominator accumulator of said parameter.

78. A method for estimating a set of parameters for insertion into a statistical pattern recognition process, said method comprising:

determining initial values for said set of parameters; and

performing an estimation cycle comprising;

receiving a training set having members;

performing recognition on said members using a current set of parameters and a predetermined group of elements;

generating at least one equivalence set comprising recognized members of said training set;

using said equivalence sets and said set of parameters to calculate a target function;

maximizing said target function with respect to said set of parameters;

updating said set of parameters to maximize said target function;

if said set of parameters satisfies a predetermined estimation termination condition, outputting said parameters and discontinuing said parameter estimation method; and

if said set of parameters does not satisfy a predetermined estimation termination condition, performing another estimation cycle.

79. A method for estimating a set of parameters according to claim 78, wherein said target function comprises a summation, over the elements of said predetermined group of elements, of a difference between:

a first summation of logarithms of probability density functions as a function of said set of parameters, and

a second summation, of logarithms of probability density functions as a function of said set of parameters, multiplied by a discrimination rate, said discrimination rate being variable between zero and one.

80. A method for estimating a set of parameters according to claim 79, wherein said target function comprises

81. A method for estimating a set of parameters according to claim 79, further comprising tuning said discrimination rate.

82. A method for estimating a set of parameters according to claim 78, further comprising providing at least some of said updated parameter set to a statistical pattern recognition process.

83. A method for estimating a set of parameters according to claim 82, wherein said statistical pattern recognition process comprises a speech recognition process.

84. A method for estimating a set of parameters according to claim 82, wherein said statistical pattern recognition process includes one of a group comprising:

image recognition, decryption, communications, sensory recognition, optical, optical character recognition (OCR), natural language processing (NLP), gesture and object recognition (for machine vision), text classification, and control processes.

85. A method for estimating a set of parameters according to claim 78, wherein the step of maximizing said target function with respect to said set of parameters comprises:

performing a maximization cycle comprising;

using a current estimate of said set of parameters to calculate an auxiliary function associated with said target function;

maximizing said auxiliary function with respect to said set of parameters;

updating said set of parameters to maximize said target function;

if said set of parameters satisfies a predetermined maximization termination condition, outputting said parameters and discontinuing said parameter maximization; and

if said set of parameters does not satisfy a predetermined maximization termination condition, performing another maximization cycle.

86. A method for estimating a set of parameters according to claim 85, wherein said auxiliary function comprises a summation, over the elements of said predetermined group of elements, of a difference between:

a first summation of conditional expected value functions as a function of said set of parameters, and

a second summation, of conditional expected value functions as a function of said set of parameters, multiplied by a discrimination rate, said discrimination rate being variable between zero and one.

87. A method for estimating a set of parameters according to claim 86, wherein said auxiliary function comprises

88. A method for estimating a set of parameters according to claim 87, wherein said second data set comprises a complete data set.

89. A method for estimating a set of parameters according to claim 78, wherein said statistical pattern recognition process comprises a speech recognition process, said members of said training set comprise utterances, and said predetermined group of elements comprises a predetermined vocabulary of words.

90. A method for estimating a set of parameters according to claim 89, wherein said performing recognition on said members comprises performing Viterbi recognition on said members.

91. A method for estimating a set of parameters according to claim 78, wherein determining initial values for said set of parameters comprises performing maximum likelihood estimation to determine said initial values.

92. A method for estimating a set of parameters according to claim 78, wherein said statistical process uses a hidden Markov model (HMM).

93. A method for performing statistical pattern recognition upon an input sequence, thereby to transcribe said input sequence into an output sequence comprising elements from a predetermined group of elements, the method comprising the steps of:

receiving said input sequence;

estimating a set of parameters of a statistical model by;

determining initial values for said set of parameters; and

performing an estimation cycle comprising;

receiving a training set having members;

performing recognition on said members using a current set of parameters and said predetermined group of elements;

generating at least one equivalence set comprising recognized members of said training set;

using said equivalence sets and said set of parameters to calculate a target function;

maximizing said target function with respect to said set of parameters;

updating said set of parameters to maximize said target function;

if said set of parameters satisfies a predetermined estimation termination condition, discontinuing said parameter estimation; and

if said set of parameters does not satisfy a predetermined estimation termination condition, performing another estimation cycle;

transcribing said input sequence according to said statistical model having said estimated set of parameters.

94. A method for performing statistical pattern recognition according to claim 93, wherein said target function comprises a summation, over the elements of said predetermined group of elements, of a difference between:

a first summation of logarithms of probability density functions as a function of said set of parameters, and

a second summation, of logarithms of probability density functions as a function of said set of parameters, multiplied by a discrimination rate, said discrimination rate being variable between zero and one.

95. A method for performing statistical pattern recognition according to claim 94, wherein said target function comprises

96. A method for performing statistical pattern recognition according to claim 95, further comprising tuning said discrimination rate.

97. A method for performing statistical pattern recognition according to claim 93, wherein said statistical pattern recognition process comprises a speech recognition process.

98. A method for performing statistical pattern recognition according to claim 93, wherein said statistical pattern recognition process comprises one of said following types of processes:

image recognition, decryption, communications, sensory recognition, optical, optical character recognition (OCR), natural language processing (NLP), gesture and object recognition (for machine vision), text classification, and control.

99. A method for performing statistical pattern recognition according to claim 93, wherein the step of maximizing said target function with respect to said set of parameters comprises:

performing a maximization cycle comprising;

using a current estimate said set of parameters to calculate an auxiliary function associated with said target function;

maximizing said auxiliary function with respect to said set of parameters;

updating said set of parameters to maximize said target function;

if said set of parameters satisfies a predetermined maximization termination condition, outputting said parameters and discontinuing said parameter maximization; and

if said set of parameters does not satisfy a predetermined maximization termination condition, performing another maximization cycle.

100. A method for performing statistical pattern recognition according to claim 99, wherein said auxiliary function comprises a summation, over the elements of said predetermined group of elements, of a difference between:

a first summation of conditional expected value functions as a function of said set of parameters, and

a second summation, of conditional expected value functions as a function of said set of parameters, multiplied by a discrimination rate, said discrimination rate being variable between zero and one.

101. A method for performing statistical pattern recognition according to claim 100, wherein said auxiliary function comprises

102. A method for performing statistical pattern recognition according to claim 93, wherein said statistical pattern recognition comprises a speech recognition, said members of said training set comprise utterances, and said predetermined group of elements comprises a predetermined vocabulary of words.

103. A method for performing statistical pattern recognition according to claim 102, wherein performing recognition on said members comprises performing Viterbi recognition on said members.

104. A method for performing statistical pattern recognition according to claim 102, wherein transcribing said input sequence comprises performing Viterbi recognition upon said input sequence.

105. A method for performing statistical pattern recognition according to claim 93, wherein determining initial values for said set of parameters comprises performing maximum likelihood estimation to determine said initial values.

106. A method for performing statistical pattern recognition according to claim 93, wherein said statistical model comprises a hidden Markov model (HMM).

107. A method for performing statistical pattern recognition according to claim 93, wherein said input sequence comprises a continuous sequence.