LANGUAGE MODELING OF COMPLETE LANGUAGE SEQUENCES

• US 20140278407A1
• Filed: 05/02/2013
• Published: 09/18/2014
• Est. Priority Date: 03/14/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Chelba, Ciprian I., Sak, Hasim, Schalkwyk, Johan
• Granted Patent: US 9,786,269 B2
• Time in Patent Office: Days
• US Class Current: 704/235
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/197 Probabilistic grammars, e.g. word n-grams

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for language modeling of complete language sequences. Training data indicating language sequences is accessed, and counts for a number of times each language sequence occurs in the training data are determined. A proper subset of the language sequences is selected, and a first component of a language model is trained. The first component includes first probability data for assigning scores to the selected language sequences. A second component of the language model is trained based on the training data, where the second component includes second probability data for assigning scores to language sequences that are not included in the selected language sequences. Adjustment data that normalizes the second probability data with respect to the first probability data is generated, and the first component, the second component, and the adjustment data are stored.

Claims

1. A method performed by data processing apparatus, the method comprising:

accessing training data indicating queries submitted by one or more users;

determining, for each of the queries, a count of a number of times the training data indicates the query was submitted;

selecting a proper subset of the queries based on the counts;

training a first component of a language model based on the counts, the first component including first probability data indicating relative frequencies of the selected queries among the training data;

training a second component of the language model based on the training data, the second component including second probability data for assigning scores to queries that are not included in the selected queries;

determining adjustment data that normalizes the second probability data with respect to the first probability data; and

storing the first component, the second component, and the adjustment data.

2. The method of claim 1, wherein training the first component of the language model based on the counts comprises generating a first probability distribution over a set of possible outcomes limited to the queries occurring in the training data;

and

wherein training the second component of the language model based on the training data comprises generating a second probability distribution for which the range of possible outcomes is not limited to a defined set of queries.

3. The method of claim 2, wherein determining the adjustment data comprises determining one or more weighting values for weighting the second probability distribution relative to the first probability distribution to form combined probability distribution.

4. The method of claim 1, wherein accessing training data indicating queries submitted by one or more users comprises accessing one or more query logs indicating voice queries spoken by different users;

wherein selecting the proper subset of the queries based on the counts comprises selecting queries having the highest counts.

5. The method of claim 1, wherein selecting the proper subset of the queries based on the counts comprises selecting queries having a count that equals or exceeds a minimum threshold value, the minimum threshold value being greater than one.

6. The method of claim 1, wherein selecting the proper subset of the queries based on the counts comprises:

selecting queries having a first number of terms based on a first threshold; and

selecting queries having a second number of terms based on a second threshold, the second number of terms being different from the first number of terms, and the second threshold being different from the first threshold.

7. The method of claim 1, wherein training the first component of the language model based on the counts comprises determining, for each of the selected queries, a score indicating a relative frequency of occurrence of the selected query, as a complete query, in the training data.

8. The method of claim 1, wherein training the first component of the language model comprises generating data indicating a first probability distribution for which a first sum of probabilities of occurrence of the selected queries is a first value;

wherein training the second component of the language model comprises generating data indicating a second probability distribution for which a second sum of probabilities of occurrence of the selected queries is a second value;

wherein determining the adjustment data comprises determining a weighting value based on the first value and the second value.

9. The method of claim 1, wherein determining the adjustment data comprises determining a weighting value to apply to output of the second component, the weighting value equalizing a portion of a probability distribution of the second component with a corresponding portion of a probability distribution of the first component.

10. The method of claim 1, further comprising:

determining a first score for a particular query using the first component of the language model;

determining a second score for the particular query using the second component of the language model;

determining that the first score and the second score do not satisfy a predetermined relationship; and

in response to determining that the first score and the second score do not satisfy the predetermined relationship, removing the particular query from the selected queries to generate an altered set of selected queries.

11. The method of claim 10, further comprising, after removing the particular query from the selected queries, determining second adjustment data based on the altered set of selected queries.

12. The method of claim 1, wherein training the second component of the language model based on the training data comprises training an n-gram model.

13. The method of claim 12, wherein training the n-gram model comprises determining conditional probabilities each indicating a probability of occurrence of a term given occurrence of one or more other terms.

14. The method of claim 1, wherein training the second component of the language model based on the training data comprises training the second component using a proper subset of the training data, the proper subset of the training data excluding instances of the selected queries.

15. The method of claim 1, wherein accessing training data indicating queries submitted by one or more users comprises (i) accessing first training data indicating first queries associated with a first geographical area and (ii) accessing second training data indicating second queries associated with a second geographical area that is larger than the first geographical area;

wherein determining, for each of the queries, a count of the number of times the training data indicates the query was submitted comprises determining, for each of the first queries, a count of a number of times the first training data indicates the query was submitted;

wherein selecting the proper subset of the queries based on the counts comprises selecting queries from among the first queries associated with the first geographical area;

wherein training the first component of the language model based on the counts comprises training the first component based on the counts indicating the number of times the first training data indicates that the selected queries were submitted; and

wherein training the second component of the language model based on the training data comprises training the second component of the language model based on the second training data indicating queries associated with the second geographical area.

16. The method of claim 15, further comprising:

receiving a query;

determining that the received query is associated with the first geographical area;

in response to determining that the received query is associated with the first geographical area, selecting the first component from among a plurality of language models corresponding to different geographical areas;

using the first component to evaluate one or more candidate transcriptions that are included in the selected queries; and

using the second component to evaluate one or more candidate transcriptions that are not included in the selected queries.

17. The method of claim 1, further comprising:

receiving a candidate transcription for one or more utterances;

determining that the candidate transcription is one of the selected queries;

in response to determining that the candidate transcription is one of the selected queries, determining a score for the candidate transcription using the first component of the language model; and

evaluating the candidate transcription based on the score.

18. The method of claim 1, further comprising:

receiving a candidate transcription for one or more utterances;

determining that the candidate transcription is not one of the selected queries;

in response to determining that the candidate transcription is not one of the selected queries, determining a score for the candidate transcription using the second component of the language model; and

evaluating the candidate transcription based on the score.

19. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

accessing training data indicating queries submitted by one or more users;

determining, for each of the queries, a count of a number of times the training data indicates the query was submitted;

selecting a proper subset of the queries based on the counts;

training a first component of a language model based on the counts, the first component including first probability data indicating relative frequencies of the selected queries among the training data;

training a second component of the language model based on the training data, the second component including second probability data for assigning scores to queries that are not included in the selected queries;

determining adjustment data that normalizes the second probability data with respect to the first probability data; and

storing the first component, the second component, and the adjustment data.

20. A computer storage medium storing a computer program, the program comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

accessing training data indicating queries submitted by one or more users;

determining, for each of the queries, a count of a number of times the training data indicates the query was submitted;

selecting a proper subset of the queries based on the counts;

training a first component of a language model based on the counts, the first component including first probability data indicating relative frequencies of the selected queries among the training data;

training a second component of the language model based on the training data, the second component including second probability data for assigning scores to queries that are not included in the selected queries;

determining adjustment data that normalizes the second probability data with respect to the first probability data; and

storing the first component, the second component, and the adjustment data.