Local and remote aggregation of feedback data for speech recognition

• US 10,157,609 B2
• Filed: 07/14/2015
• Issued: 12/18/2018
• Est. Priority Date: 06/09/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Plumpe, Michael D., Odell, Julian, Hamaker, Jon, Chambers, Rob, Le, Christopher, Domanic, Onur
• Primary Examiner(s): Shah, Paras D
• Application Number:
  US14/799,533
• Publication Number:
  US 20160012817A1
• Time in Patent Office: 1,253 Days
• Field of Search: 704231, 704236, 704251, 704252, 704257
• CPC Class Codes: G10L 15/01 Assessment or evaluation of speech recognition systems
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/065 Adaptation
• CPC Class Codes: G10L 15/14 using statistical models, e.g. Hidden Markov Models [HMMs] G10L15/18 takes precedence
• CPC Class Codes: G10L 15/187 Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/025 Phonemes, fenemes or fenones being the recognition units
• CPC Class Codes: G10L 2015/0631 Creating reference templates; Clustering
• CPC Class Codes: G10L 2015/0635 updating or merging of old and new templates; Mean values; Weighting

Abstract

A local feedback mechanism for customizing training models based on user data and directed user feedback is provided in speech recognition applications. The feedback data is filtered at different levels to address privacy concerns for local storage and for submittal to a system developer for enhancement of generic training models.

Claims

1. A computing device for providing speech recognition with local and remote feedback loops, the computing device composing:

a communications device configured to communicate with a remote system developer over a communications network;

one or more processors; and

a memory storing instructions that when executed by the one or more processors, cause the computing device to perform a method comprising;

collecting user data associated with a user, wherein the user data includes audio by the user and textual data from user generated documents;

filtering the collected data at distinct levels for local and generic models to protect private data;

updating one or more local models with the user data filtered at the local model level, each local model comprising at least one of a local acoustic model that models how phonemes sound or a local language model that models how words fit together to form sentences;

providing, over the communications network, user data filtered at the generic model level to the remote system developer to enable the remote system developer to update one or more generic models comprising at least one of a remote acoustic model that models how phonemes sound or a remote language model that models how words fit together to form sentences;

receiving speech inputs; and

recognizing, by a speech recognition system, the speech inputs based at least in part on the updated one or more local models and the updated one or more generic models.

2. The computing device of claim 1, wherein the method further comprises:

receiving, over the communications network, one or more updated generic models from the system developer; and

updating the one or more local models with the received one or more updated generic models.

3. The computing device of claim 1, wherein the method further comprises:

receiving directed user feedback to recognized speech, the user feedback comprising at least one of textual data or audio; and

aggregating the directed user feedback with the collected user data.

4. The computing device of claim 1, wherein the method further comprises:

processing the collected user data to detect trigrams, wherein filtering the collected data comprises removing the trigrams that include digits to address privacy concerns associated with the digits, wherein the digits describe one or more of;

phone numbers, social security numbers, or credit card numbers.

5. The computing device of claim 1, wherein collecting the user data comprises harvesting textual data from one or more documents in the computing device employing an n-gram method.

6. The computing device of claim 5, wherein collecting the user data comprises harvesting textual data from one or more of:

emails, word processing documents, or spreadsheets that employ trigrams.

7. The computing device of claim 1, wherein the method further comprises statistically analyzing the collected user data to determine updates to the one or more local models.

8. The computing device of claim 7, wherein the statistical analysis includes determination of words popularized since a release of the one or more local models.

9. A method for providing speech recognition with local and remote feedback loops, the method comprising:

collecting user data including audio and textual data from one or more computing devices associated with a user, wherein the textual data is collected from user generated documents;

receiving directed feedback from the user in response to recognized speech, the feedback comprising at least one of textual data or audio;

aggregating the collected user data and the directed feedback;

filtering the aggregated data at a first level to protect private data;

storing the aggregated data filtered at the first level in one or more local models for customizing current language and acoustic models;

filtering the aggregated data at a second level for one or more generic language and acoustic models to remove private data;

providing the aggregated data filtered at the second level to a system developer to enable the system developer to update the generic language and acoustic models;

receiving a speech input; and

recognizing, by a speech recognition system, the speech input based at least in part on the updated one or more local models or the updated one or more generic language and acoustic models.

10. The method of claim 9, further comprising:

prior to providing the aggregated data filtered at the second level to the system developer, enabling the user to provide permission for forwarding of the aggregated data filtered at the second level to the system developer; and

in response to receiving permission, providing the aggregated data filtered at the second level to the system developer.

11. The method of claim 10, wherein removing the private data from the aggregated data includes:

temporarily removing all private data; and

upon determining that a subset of the private data is non-private data, adding the non-private data to the filtered data prior to providing to the system developer.

12. The method of claim 10, wherein removing the private data from the aggregated data includes removing numbers from the aggregated data.

13. The method of claim 9, wherein the one or more computing devices execute a distributed speech recognition application that capture the user data in a distributed manner.

14. A computer-readable storage device with instructions stored thereon that when executed by one or more processing units, cause the one or more processing units to perform a method for speech recognition with local and remote feedback loops, the method comprising:

collecting user data including audio and textual data, wherein the textual data is from user generated documents;

receiving directed feedback from the user in response to recognized speech, the feedback comprising at least one of audio or textual data;

aggregating the collected user data and the directed feedback;

filtering the aggregated data at different levels for the local and the remote feedback loops to protect private data;

storing the data filtered at the local feedback level to customize current language and acoustic models;

providing the data filtered at the remote feedback level to a system developer to enable the system developer to update speech recognition generic language and acoustic models;

receiving speech inputs; and

recognizing, by a speech recognition system, the speech inputs based at least in part on the customized current language and acoustic models and the updated generic language and acoustic models.

15. The computer-readable storage device of claim 14, wherein the user data is collected through at least one of:

harvesting the textual data from at least one of;

emails, word processing documents, or spreadsheets associated with the user employing trigrams;

or

receiving a user spoken text passage.

16. The computer-readable storage device of claim 15, wherein the filtered data provided to the system developer includes trigrams containing words from original language models and a list of added words.

17. The computer-readable storage device of claim 14, wherein the filtered data is provided to the system developer through one of:

an email exchange, a file transfer, or an access through a hosted service.