Speech recognition using log-linear model

• US 10,134,394 B2
• Filed: 05/11/2015
• Issued: 11/20/2018
• Est. Priority Date: 03/20/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Caseiro, Diamantino Antonio, Biadsy, Fadi
• Primary Examiner(s): Goddard, Tammy Paige
• Assistant Examiner(s): Shah, Bharatkumar S
• Application Number:
  US14/708,465
• Publication Number:
  US 20160275946A1
• Time in Patent Office: 1,289 Days
• Field of Search: 704235
• CPC Class Codes: G06F 40/216 using statistical methods
• CPC Class Codes: G10L 15/197 Probabilistic grammars, e.g. word n-grams

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, relating to generating log-linear models. In some implementations, n-gram parameter values derived from an n-gram language model are obtained. N-gram features for a log-linear language model are determined based on the n-grams corresponding to the obtained n-gram parameter values. A weight for each of the determined n-gram features is determined, where the weight is determined based on (i) an n-gram parameter value that is derived from the n-gram language model and that corresponds to a particular n-gram, and (ii) an n-gram parameter value that is derived from the n-gram language model and that corresponds to an n-gram that is a sub-sequence within the particular n-gram. A log-linear language model having the determined n-gram features is generated, where the determined n-gram features in the log-linear language model have weights that are initialized based on the determined weights.

Claims

1. A method performed by one or more computers, the method comprising:

obtaining n-gram parameter values derived from an n-gram language model, the n-gram parameter values including n-gram parameter values for n-grams that include multiple words;

determining n-gram features for a log-linear language model based on the n-grams corresponding to the obtained n-gram parameter values;

determining a weight for each of the determined n-gram features, wherein for at least some of the n-gram features, the weight is determined based on (i) an n-gram parameter value that is derived from the n-gram language model and that corresponds to a particular n-gram of multiple words, and (ii) an n-gram parameter value that is derived from the n-gram language model and that corresponds to an n-gram that is a sub-sequence within the particular n-gram;

generating a log-linear language model having the determined n-gram features, the determined n-gram features in the log-linear language model having weights that are initialized based on the determined weights;

after generating the log-linear language model, training the log-linear language model to adjust the initialized weights;

after training the log-linear language model, using the log-linear language model to determine a transcription for an utterance; and

providing the transcription for the utterance.

2. The method of claim 1, wherein the n-gram language model is configured to assign, to an n-gram that does not have a corresponding parameter value in the n-gram language model, a score based on a parameter value in the n-gram language model for a sub-sequence of words within the n-gram.

3. The method of claim 1, wherein, for each of the n-grams that includes multiple words, the corresponding parameter value indicates a conditional probability of an occurrence of a last word in the n-gram given an occurrence of one or more words that precede the last word in the n-gram.

4. The method of claim 1, wherein generating the log-linear language model comprises generating the log-linear language model to indicate, for one or more n-grams, likelihoods of occurrence that equal to likelihoods of occurrence indicated by the n-gram language model for the one or more n-grams.

5. The method of claim 1, wherein determining the n-gram features for the log-linear language model comprises determining, for each n-gram parameter value in the n-gram language model, an n-gram feature that represents an occurrence of a particular word in a particular context comprising one or more words.

6. The method of claim 1, wherein determining the n-gram features for the log-linear language model comprises:

identifying each of the n-grams corresponding to the n-gram parameter values derived from the n-gram language model; and

determining the n-gram features to include a feature corresponding to each identified n-gram.

7. The method of claim 1, further comprising:

obtaining backoff parameter values derived from the n-gram language model, the backoff parameter values representing n-gram backoffs from one order of n-gram to a lower order of n-gram;

determining backoff features for the log-linear language model that represent the backoffs of the n-grams in the n-gram language model;

determining a weight for each backoff feature in the determined backoff features, wherein each of the weights for the determined backoff features is respectively determined based on a backoff parameter value derived from the n-gram language model; and

wherein generating the log-linear language model comprises generating the log-linear language model to have the backoff features and corresponding weights that are initialized based on the determined weights for the backoff features.

8. The method of claim 7, wherein the weights for the backoff features in the log-linear language model each represents a probability adjustment equivalent to the adjustment represented by a corresponding backoff parameter value in the n-gram language model.

9. The method of claim 1, wherein determining the weight for each n-gram feature comprises, for at least some of the weights, incorporating a backoff parameter value from the n-gram language model in the weight determination.

10. The method of claim 1, wherein the log-linear language model includes non-linguistic features corresponding to one or more aspects of non-linguistic context, the non-linguistic features including features indicative of a user characteristic, a time, a geographic location, an application, or an input field.

11. The method of claim 1, wherein training the log-linear language model comprises training the log-linear language model using stochastic gradient descent training and using a set of training data that is different from a set of training data used to train the n-gram language model.

19. The method of claim 1, wherein generating the log-linear language model comprises including, in the log-linear language model, features and corresponding weights representing one or more non-linguistic contextual factors;

wherein training the log-linear language model comprises adjusting the weights based on training data indicating non-linguistic contextual factors.

20. The method of claim 19, wherein the non-linguistic contextual factors comprise a characteristic of at least one of a physical environment, a physical state of a device, an operating state of a device, or a speaker.

12. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

obtaining n-gram parameter values derived from an n-gram language model, the n-gram parameter values including n-gram parameter values for n-grams that include multiple words;

determining n-gram features for a log-linear language model based on the n-grams corresponding to the obtained n-gram parameter values;

determining a weight for each of the determined n-gram features, wherein for at least some of the n-gram features, the weight is determined based on (i) an n-gram parameter value that is derived from the n-gram language model and that corresponds to a particular n-gram of multiple words, and (ii) an n-gram parameter value that is derived from the n-gram language model and that corresponds to an n-gram that is a sub-sequence within the particular n-gram;

generating a log-linear language model having the determined n-gram features, the determined n-gram features in the log-linear language model having weights that are initialized based on the determined weights;

after generating the log-linear language model, training the log-linear language model to adjust the initialized weights;

after training the log-linear language model, using the log-linear language model to determine a transcription for an utterance; and

providing the transcription for the utterance.

13. The system of claim 12, wherein the operations further comprise:

obtaining backoff parameter values derived from the n-gram language model, the backoff parameter values representing n-gram backoffs from one order of n-gram to a lower order of n-gram;

determining backoff features for the log-linear language model that represent the backoffs of the n-grams in the n-gram language model;

determining a weight for each backoff feature in the determined backoff features, wherein each of the weights for the determined backoff features is respectively determined based on a backoff parameter value derived from the n-gram language model; and

wherein generating the log-linear language model comprises generating the log-linear language model to have the backoff features and corresponding weights that are initialized based on the determined weights for the backoff features.

14. The system of claim 12, wherein generating the log-linear language model comprises generating the log-linear language model to indicate, for one or more n-grams, likelihoods of occurrence that equal to likelihoods of occurrence indicated by the n-gram language model for the one or more n-grams.

15. The system of claim 12, wherein determining the n-gram features for the log-linear language model comprises determining, for each n-gram parameter value in the n-gram language model, a n-gram feature that represents an occurrence of a particular word in a particular context comprising one or more words.

16. A non-transitory computer-readable medium storing a computer program, the program comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

obtaining n-gram parameter values derived from an n-gram language model, the n-gram parameter values including n-gram parameter values for n-grams that include multiple words;

determining n-gram features for a log-linear language model based on the n-grams corresponding to the obtained n-gram parameter values;

determining a weight for each of the determined n-gram features, wherein for at least some of the n-gram features, the weight is determined based on (i) an n-gram parameter value that is derived from the n-gram language model and that corresponds to a particular n-gram of multiple words, and (ii) an n-gram parameter value that is derived from the n-gram language model and that corresponds to an n-gram that is a sub-sequence within the particular n-gram;

generating a log-linear language model having the determined n-gram features, the determined n-gram features in the log-linear language model having weights that are initialized based on the determined weights;

using the log-linear language model to determine a transcription for an utterance; and

providing the transcription for the utterance.

17. The non-transitory computer-readable medium of claim 16, wherein the operations further comprise:

obtaining backoff parameter values derived from the n-gram language model, the backoff parameter values representing n-gram backoffs from one order of n-gram to a lower order of n-gram;

determining backoff features for the log-linear language model that represent the backoffs of the n-grams in the n-gram language model;

determining a weight for each backoff feature in the determined backoff features, wherein each of the weights for the determined backoff features is respectively determined based on a backoff parameter value derived from the n-gram language model; and

wherein generating the log-linear language model comprises generating the log-linear language model to have the backoff features and corresponding weights that are initialized based on the determined weights for the backoff features.

18. The non-transitory computer-readable medium of claim 16, wherein generating the log-linear language model comprises generating the log-linear language model to indicate, for one or more n-grams, likelihoods of occurrence that equal to likelihoods of occurrence indicated by the n-gram language model for the one or more n-grams.