Generating language models

• US 9,437,189 B2
• Filed: 05/29/2014
• Issued: 09/06/2016
• Est. Priority Date: 05/29/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Epstein, Mark Edward, Vasserman, Lucy
• Primary Examiner(s): APONTE, FRANCISCO JAVIER
• Application Number:
  US14/290,090
• Publication Number:
  US 20150348541A1
• Time in Patent Office: 831 Days
• Field of Search: 704200-278, 701 1-302, 717/104
• US Class Current: 1/1
• CPC Class Codes: G06F 8/10 Requirements analysis; Specification techniques
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating language models. In some implementations, data is accessed that indicates a set of classes corresponding to a concept. A first language model is generated in which a first class represents the concept. A second language model is generated in which second classes represent the concept. Output of the first language model and the second language model is obtained, and the outputs are evaluated. A class from the set of classes is selected based on evaluating the output of the first language model and the output of the second language model. In some implementations, the first class and the second class are selected from a parse tree or other data that indicates relationships among the classes in the set of classes.

Claims

1. A method performed by one or more computers, the method comprising:

accessing data indicating a set of classes that each represent a different level of specificity of a same, particular semantic concept, wherein the classes respectively correspond to different types of words or phrases, and each class includes multiple words or phrases of the corresponding type for the class;

identifying a language sequence including a particular word or phrase that corresponds to the particular semantic concept;

generating a first modified language sequence by replacing the particular word or phrase with a symbol representing the first class;

generating a second modified language sequence by replacing the particular word or phrase with a symbol representing at least one the second classes;

generating a first language model in which a single first class from the set of classes represents the particular semantic concept, the first language model being trained using the first modified language sequence;

generating a second language model in which multiple second classes from the set of classes represent the particular semantic concept at a greater level of specificity than the first class, each of the second classes being different from the first class, the second language model being trained using the second modified language sequence; and

selecting the first class or the multiple second classes based on output of the first language model and output of the second language model.

2. The method of claim 1, wherein generating the first language model comprises generating the first language model using the first class to represent the particular semantic concept and without using any of the second classes to represent the particular semantic concept, the first language model being trained using the first modified language sequence and not using the second modified language sequence;

and

wherein generating the second language model comprises generating the second language model using the second classes to represent the particular semantic concept and without using the first class to represent the particular semantic concept, the second language model being trained using the second modified language sequence and not using the first modified language sequence.

3. The method of claim 1, further comprising:

receiving audio data corresponding to an utterance; and

based on selecting the first class or the multiple second classes, determining a transcription for the utterance using a language model in which either (i) the selected first class represents the particular semantic concept and the multiple second classes do not represent the particular semantic concept, or (ii) the selected multiple second classes represent the particular semantic concept and the first class does not represent the particular semantic concept.

4. The method of claim 3, wherein selecting the first class or the multiple second classes comprises selecting the first class;

and

wherein determining the transcription for the utterance using a language model in which the selected one or more classes represents the concept comprises using the first language model to determine the transcription for the utterance.

5. The method of claim 3, wherein selecting the first class or the multiple second classes comprises selecting the first class and not selecting the second classes;

and

wherein determining the transcription for the utterance using a language model in which the selected one or more classes represent the concept comprises using the first language model to determine the transcription for the utterance and not using the second language model to determine the transcription for the utterance.

6. The method of claim 3, wherein selecting the first class or the multiple second classes comprises selecting the first class;

and

wherein the method further comprises, based on selecting the first class, generating a third language model in which the first class from the set of classes represents the concept; and

wherein using a language model in which the selected one or more classes represents the concept to determine a transcription for the utterance comprises using the third language model to determine the transcription.

7. The method of claim 1, further comprising:

determining a first score based on the output of the first language model;

determining a second score based on the output of the second language model; and

comparing the first score and the second score,

wherein selecting the first class or the multiple second classes is performed based on comparing the first score and the second score.

8. The method of claim 7, wherein determining the first score comprises determining a score that indicates a word error rate of the first language model;

and

wherein determining the second score comprises determining a score that indicates a word error rate of the second language model.

9. The method of claim 7, wherein determining the first score comprises determining a score that indicates a perplexity of the first language model;

and

determining the second score comprises determining a score that indicates a perplexity of the second language model.

10. The method of claim 1, wherein accessing data indicating the set of classes comprises accessing data indicating a hierarchy of classes corresponding to the particular semantic concept, wherein at least some of the classes in the hierarchy represent different forms of expressing at least a portion of the particular semantic concept.

11. The method of claim 10, wherein the hierarchy includes a top-level class, one or more lowest-level classes, and one or more intermediate classes located between the top-level class and the one or more lowest-level classes;

wherein selecting the first class or the multiple second classes comprises selecting one or more of the intermediate classes; and

wherein the method further comprises using a language model in which the selected one or more intermediate classes are used to represent the concept to determine a transcription for one or more utterances.

12. The method of claim 10, wherein the first class and the multiple second classes are located at different levels of the hierarchy, and the multiple second classes are sub-classes of the first class.

13. The method of claim 10, further comprising:

generating, for each additional set of classes in the hierarchy that represents the concept other than the first class and the multiple second classes, an additional language model in which the additional set of classes represents the concept;

obtaining output of each of the additional language models for the set of input data; and

evaluating the output of each of the additional language models;

selecting one or more of the classes in the hierarchy of classes based on evaluating the output of the first language model and the output of the second language model and based on evaluating the output of each of the additional language models.

14. The method of claim 1, further comprising identifying a transcription for an utterance;

wherein generating the first language model comprises training the first language model based on the transcription for the utterance, and wherein generating the second language model comprises training the second language model based on the transcription for the utterance.

15. A system comprising:

one or more computers; and

one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

accessing data indicating a set of classes that each represent a different level of specificity of a same, particular semantic concept, wherein the classes respectively correspond to different types of words or phrases, and each class includes multiple words or phrases of the corresponding type for the class;

identifying a language sequence including a particular word or phrase that corresponds to the particular semantic concept;

generating a first modified language sequence by replacing the particular word or phrase with a symbol representing the first class;

generating a second modified language sequence by replacing the particular word or phrase with a symbol representing at least one the second classes;

generating a first language model in which a single first class from the set of classes represents the particular semantic concept, the first language model being trained using the first modified language sequence;

generating a second language model in which multiple second classes from the set of classes represent the particular semantic concept at a greater level of specificity than the first class, each of the second classes being different from the first class, the second language model being trained using the second modified language sequence; and

selecting the first class or the multiple second classes based on output of the first language model and output of the second language model.

16. A non-transitory computer-readable storage device storing instructions that, when executed by a computer, cause the computer to perform operations comprising:

accessing data indicating a set of classes that each represent a different level of specificity of a same, particular semantic concept, wherein the classes respectively correspond to different types of words or phrases, and each class includes multiple words or phrases of the corresponding type for the class;

identifying a language sequence including a particular word or phrase that corresponds to the particular semantic concept;

generating a first modified language sequence by replacing the particular word or phrase with a symbol representing the first class;

generating a second modified language sequence by replacing the particular word or phrase with a symbol representing at least one the second classes;

generating a first language model in which a single first class from the set of classes represents the particular semantic concept, the first language model being trained using the first modified language sequence;

generating a second language model in which multiple second classes from the set of classes represent the particular semantic concept at a greater level of specificity than the first class, each of the second classes being different from the first class, the second language model being trained using the second modified language sequence; and

selecting the first class or the multiple second classes based on output of the first language model and output of the second language model.

17. The non-transitory computer-readable storage device of claim 16, wherein accessing data indicating the set of classes comprises accessing data indicating a hierarchy of classes representing the particular semantic concept at each of multiple different levels of specificity.

18. The non-transitory computer-readable storage device of claim 17, wherein the hierarchy includes a top-level class, one or more lowest-level classes, and multiple intermediate classes located between the top-level class and the one or more lowest-level classes, and wherein the multiple second classes are the multiple intermediate classes;

wherein selecting the first class or the multiple second classes comprises selecting the multiple intermediate classes; and

wherein the method further comprises using a language model in which the selected multiple intermediate classes are used to represent the concept to determine a transcription for one or more utterances.

19. The non-transitory computer-readable storage device of claim 17, wherein the operations further comprise:

identifying, in a language sequence, a word or phrase corresponding to the particular semantic concept; and

identifying, for the identified word or phrase, a subset of the classes in the hierarchy that includes classes at different levels of the hierarchy that that represent the semantic meaning of the identified word or phrase at different levels of specificity;

wherein generating the first language model comprises training the first language model based on the language sequence and one or more of the identified classes, and wherein generating the second language model comprises training the second language model based on the language sequence and one or more of the identified classes.

20. The non-transitory computer-readable storage device of claim 16, wherein generating the first language model comprises generating the first language model using the first class to represent the particular semantic concept and without using any of the second classes to represent the particular semantic concept, the first language model being trained using the first modified language sequence and not using the second modified language sequence;

and

wherein generating the second language model comprises generating the second language model using the second classes to represent the particular semantic concept and without using the first class to represent the particular semantic concept, the second language model being trained using the second modified language sequence and not using the first modified language sequence;

wherein selecting the first class or the multiple second classes comprises selecting the first class or the multiple second classes but not both the first class and the multiple second classes;

wherein the operations further comprise;

receiving audio data corresponding to an utterance; and

based on selecting the first class or the multiple second classes, determining a transcription for the utterance using a language model in which either (i) the selected first class represents the particular semantic concept and the multiple second classes do not represent the particular semantic concept, or (ii) the selected multiple second classes represent the particular semantic concept and the first class does not represent the particular semantic concept.