Method of refining statistical pattern recognition models and statistical pattern recognizers

• US 7,509,259 B2
• Filed: 12/21/2004
• Issued: 03/24/2009
• Est. Priority Date: 12/21/2004
• Status: Active Grant

• Current Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Original Assignee: Motorola, Inc. (Motorola Solutions, Inc.)
• Inventors: Song, Jianming J.
• Primary Examiner(s): Edouard; Patrick N
• Assistant Examiner(s): GODBOLD, DOUGLAS
• Application Number:
  US11/018,271
• Publication Number:
  US 20060136205A1
• Time in Patent Office: 1,554 Days
• Field of Search: 704/236, 704/240, 704243-245, 704/256.2
• US Class Current: 704/256.2
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 2015/0635 updating or merging of old and new templates; Mean values; Weighting

Abstract

A device (800) performs statistical pattern recognition using model parameters that are refined by optimizing an objective function that includes a term for many items of training data for which recognition errors occur wherein each term depends on a relative magnitude of a first score for a recognition result for an item of training data and a second score calculated by evaluating a statistical pattern recognition model identified by a transcribed identity of the training data item with feature vectors extracted from the item of training data. The objective function does not include terms for items of training data for which there is a gross discrepancy between a transcribed identity and a recognized identity. Gross discrepancies can be detected by probability score or pattern identity comparisons. Terms, of the objective function are weighted based on the type of recognition error and weights can be increased for high priority patterns.

Claims

1. A method of operating a pattern recognition system for refining a plurality of statistical pattern recognition models that are used for statistical pattern recognition, the method including:

reading in initial values of a set of parameters for said plurality of statistical pattern recognition models;

reading a training data set that includes a plurality of training data items including training data items for each of said plurality of statistical pattern recognition models, along with a transcribed identity for each of said plurality of training data items;

obtaining feature vectors from each of the plurality of training data items;

using a processor to perform an optimization routine for optimizing an objective function in order to find refined values of said set of parameters for said plurality of said statistical pattern recognition models corresponding to an extremum of said objective function, wherein said objective function is dynamically defined for each of a succession of iterations of said optimization routine to include a subexpression for each k^th item of training data in, at least, a subset of said plurality of training data items that is defined by, at least, a first criterion that requires that said transcribed identity does not match a recognized identity for said k^th item of training data, and a second criterion that requires that there is not a gross discrepancy between said transcribed identity and said recognized identity, wherein each subexpression depends on a relative magnitude of a first probability score compared to a second probability score, wherein said first probability score is based on a value of a first statistical pattern recognition model corresponding to said recognized identity of said k^th item of training data evaluated with said one or more feature vectors obtained from said k^th item of training data and said second probability score is based on a value of a second statistical pattern recognition model corresponding to said transcribed identity of said k^th item of training data evaluated with said one or more feature vectors obtained from said k^th item of training data; and

using the refined statistical pattern recognition models to recognize a pattern.

2. The method according to claim 1 wherein optimizing said objective function comprises:

repeatedly evaluating said objective function in order to find an extremum of said objective function, wherein evaluating said objective function comprises;

for each k^th item of training data in said subset of said plurality of training data items;

performing pattern recognition using one or more feature vectors obtained from said k^th item of training data in order to determine a recognized identity for said k^th item of training data and compute a first probability score for said recognized identity of said k^th item of training data;

evaluating a pattern recognition model corresponding to said transcribed identity of said k^th item of training data with said one or more feature vectors obtained from said k^th item of training data in order to determine a second probability score for said transcribed identity of said k^th item of training data; and

changing a magnitude of said objective function by an amount that depends on a relative magnitude of said first probability score and said second probability score.

3. The method according to claim 1 wherein:

using a processor to perform said optimization routine comprises invoking an optimization routine that uses derivative information; and

wherein optimizing said objective function comprises;

while one or more stopping criteria is not met, for each of a succession of iterations of said optimization routine;

for each k^th item of training data in said subset of said plurality of training data items;

evaluating a gradient of a term of said objective function corresponding to said k^th item of training data with respect to model parameters for said first statistical pattern recognition model and said second statistical pattern recognition model at said one or more feature vectors obtained from said k^th item of training data to obtain a plurality of gradient component summands;

summing said plurality of gradient component summands over said subset of said plurality of training data items to obtain a plurality of gradient component sums;

using said optimization routine that uses derivative information to update values of said model parameters based on said gradient component sums; and

when said one or more stopping criteria is met, storing a last set of values of said model parameters.

4. The method according to claim 3 wherein said optimization routine that uses derivative information comprises the method of steepest descent.

5. The method according to claim 1 wherein said subexpression comprises a difference between said first probability score and said second probability score.

6. The method according to claim 5 wherein said subexpression comprises a sigmoid function of said difference.

7. The method according to claim 1 wherein reading said training data set that includes said plurality of training data items comprises:

reading a set of speech samples wherein said transcribed identity identifies words spoken in said set of speech samples; and

recognizing each particular item of training data using a search algorithm to find a highly likely path of a Hidden Markov Model.

8. The method according to claim 1 wherein for each of said succession of iterations, for each i^th item of training data in said training data set, determining whether said second criterion is met by a process comprising:

comparing a magnitude of a function that depends on a relative magnitude of said first probability score and said second probability score to a predetermined value.

9. The method according to claim 8 wherein said function that depends on said relative magnitude of said first probability score and said second probability score is a difference between said first probability score and said second probability score.

10. The method according to claim 1 wherein for each of said succession of iterations, for each i^th item of training data, determining whether said second criterion is satisfied by a process comprising checking that said recognized identity is within a confusability group for said transcribed identity.

11. The method according to claim 10 wherein said confusability group for said transcribed identity includes other patterns for which a likelihood of confusability with said transcribed identity is above a predetermined threshold.

12. The method according to claim 10 wherein said confusability group for said transcribed identity includes a number of other patterns that are most likely, among an entire set of patterns to be recognized, to be confused with said transcribed identity.

13. The method according to claim 10 wherein said confusability group for said transcribed identity includes a predetermined group of patterns.

14. A method of operating a pattern recognition system for refining a plurality of statistical pattern recognition models that are used for statistical pattern recognition, the method including:

reading in initial values of a set of parameters for said plurality of statistical pattern recognition models;

reading a training data set that includes a plurality of training data items including training data items for each of said plurality of statistical pattern recognition models, along with a transcribed identity for each of said plurality of training data items;

obtaining feature vectors from each of the plurality of training data items;

using a processor to perform an optimization routine for optimizing an objective function in order to find refined values of said set of parameters for said plurality of said statistical pattern recognition models corresponding to an extremum of said objective function, wherein said objective function is dynamically defined for each of a succession of iterations of said optimization routine to include a subexpression for each k^th item of training data in, at least, a subset of said plurality of training data items that is defined by, at least, a first criterion that requires that said transcribed identity does not match a recognized identity for said k^th item of training data, wherein each subexpression depends on a relative magnitude of a first probability score compared to a second probability score, wherein said first probability score is based on a value of a first statistical pattern recognition model corresponding to said recognized identity of said k^th item of training data evaluated with said one or more feature vectors obtained from said k^th item of training data and said second probability score is based on a value of a second statistical pattern recognition model corresponding to said transcribed identity of said k^th item of training data evaluated with said one or more feature vectors obtained from said k^th item of training data, wherein a value of said subexpression of said objective function for each k^th item of training data is increased by applying an emphasis function that is determined based, at least in part, on said transcribed identity; and

using the refined statistical pattern recognition models to recognize a pattern.

15. The method according to claim 14 wherein:

said emphasis function is chosen to increase said value of said subexpression to a greater degree if said transcribed identity is in a predetermined group of high priority patterns;

whereby, in refining said plurality of statistical pattern recognition models, emphasis is placed on said predetermined group of high priority patterns.

16. The method according to claim 14 wherein said emphasis function is determined based on, at least, said recognized identity.

17. The method according to claim 14 wherein applying said emphasis function comprises multiplying by a weight.

18. The method according to claim 17 wherein:

for each k^th item of training data in said subset of said plurality of training data items;

said transcribed identity and said recognized identity are compared to determine if a type of error selected from a group consisting of an insertion error, a deletion error and a substitution error has occurred;

a first predetermined value is used as said weight if an insertion error has occurred;

a second predetermined value is used as said weight if a deletion error has occurred; and

a third predetermined value is used as said weight if a substitution error has occurred.

19. The method according to claim 17 wherein:

for each k^th item of training data in said subset of said plurality of training data items;

if said transcribed identity has a probability of confusion with said recognized identity that is above a predetermined threshold, said weight is increased.

20. A method of performing statistical pattern recognition comprising a method of operating a pattern recognition system for refining a plurality of statistical pattern recognition models that are used for statistical pattern recognition, the method including:

reading in initial values of a set of parameters for said plurality of statistical pattern recognition models;

reading a set of training data including a plurality of items of training data each of which is identified by a transcribed identity;

obtaining one or more feature vectors from each k^th item of training data in said set of training data;

using a processor while one or more stopping criteria is not met to;

for each k^th item of training data in said set of training data training data;

performing pattern recognition on said k^th item of training data using a current set of values of said set of parameters for said plurality of statistical pattern recognition models in order to determine a recognized identity for said k^th item of training data by finding a first statistical pattern recognition model, corresponding to said recognized identity, among said plurality of statistical pattern recognition models that yields a highest score when evaluated with said one or more feature vectors obtained from said k^th item of training data;

if a training data qualification criteria, that requires that said recognized identity does not match said transcribed identity, and that there is not a gross discrepancy between said transcribed identity and said recognized identity, is met for said k^th item of training data;

evaluating a gradient of a function that depends on a relative magnitude of said first statistical pattern recognition model compared to a second statistical pattern recognition model corresponding to said transcribed identity of said k^th item of training data with respect to model parameters for said first statistical pattern recognition model and of said second statistical pattern recognition model at said one or more feature vectors obtained from said k^th item of training data to obtain a plurality of gradient component summands;

taking a plurality of component-by-component sums of said plurality of gradient component summands by summing over a subset of said set training data for which said training data qualification criteria is met;

using said plurality of component-by-component sums of said plurality of gradient component summands in a gradient-based optimization method to update said current set of values of said set of parameters for said plurality of statistical pattern recognition models;

when said one or more stopping criteria is met, storing said current set of values of said set of parameters for said plurality of statistical pattern recognition models; and

using the refined statistical pattern recognition models to recognize a pattern.

21. An apparatus for performing statistical pattern recognition, the apparatus comprising:

a data input for inputting data including one or more unknown patterns;

a processor coupled to said data input for receiving said data;

a memory including programming instructions coupled to said processor, wherein said processor is programmed by said programming instructions to perform statistical pattern recognition on said data using a plurality of statistical pattern recognition models that include model parameters that have values resulting from a process comprising;

reading in initial values of a set of parameters for said plurality of statistical pattern recognition models;

reading a training data set that includes a plurality of training data items including training data items for each of said plurality of statistical pattern recognition models, along with a transcribed identity for each of said plurality of training data items;

obtaining feature vectors from each of the plurality of training data items;

invoking an optimization routine for optimizing an objective function in order to find refined values of said set of parameters for said plurality of said statistical pattern recognition models corresponding to an extremum of said objective function, wherein said objective function is dynamically defined for each of a succession of iterations of said optimization routine to include a subexpression for each k^th item of training data in, at least, a subset of said plurality of training data items that is defined by, at least, a first criteria that requires that said transcribed identity does not match a recognized identity for said k^th item of training data, and a second criteria that requires that there is not a gross discrepancy between said transcribed identity and said recognized identity, wherein each subexpression depends on a relative magnitude of a first probability score compared to a second probability score, wherein said first probability score is based on a value of a first statistical pattern recognition model corresponding to said recognized identity of said k^th item of training data evaluated with said one or more feature vectors obtained from said k^th item of training data and said second probability score is based on a value of a second statistical pattern recognition model corresponding to said transcribed identity of said k^th item of training data evaluated with said one or more feature vectors obtained from said k^th item of training data; and

using the refined statistical pattern recognition models to recognize a pattern.

22. A computer readable medium storing programming instructions for operating a pattern recognition system for refining a plurality of statistical pattern recognition models that are used for statistical pattern recognition including programming instructions for:

reading in initial values of a set of parameters for said plurality of statistical pattern recognition models;

reading a training data set that includes a plurality of training data items including training data items for each of said plurality of statistical pattern recognition models, along with a transcribed identity for each of said plurality of training data items;

obtaining feature vectors from each of the plurality of training data items;

using a processor to perform an optimization routine for optimizing an objective function in order to find a plurality of values of parameters for said plurality of said statistical pattern recognition models corresponding to an extremum of said objective function, wherein said objective function is dynamically defined for each of a succession of iterations of said optimization routine to include a subexpression for each k^th item of training data in, at least, a subset of said plurality of training data items that is defined by, at least, a first criteria that requires that said transcribed identity does not match a recognized identity for said k^th item of training data, wherein each subexpression depends on a relative magnitude of a first probability score compared to a second probability score, wherein said first probability score is based on a value of a first statistical pattern recognition model corresponding to said recognized identity of said k^th item of training data evaluated with one or more feature vectors obtained from said k^th item of training data and said second probability score is based on a value of a second statistical pattern recognition model corresponding to said transcribed identity of said k^th item of training data evaluated with said one or more feature vectors obtained from said k^th item of training data, wherein a value of said subexpression of said objective function for each k^th item of training data is increased by applying an emphasis function that is determined based, at least in part, on said transcribed identity; and

using the refined statistical pattern recognition models to recognize a pattern.