LOCAL AND REMOTE AGGREGATION OF FEEDBACK DATA FOR SPEECH RECOGNITION

• US 20100312555A1
• Filed: 06/09/2009
• Published: 12/09/2010
• Est. Priority Date: 06/09/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Plumpe, Michael D., Odell, Julian, Domanic, Onur, Chambers, Rob, Le, Christopher, Hamaker, Jon
• Granted Patent: US 9,111,540 B2
• Time in Patent Office: Days
• US Class Current: 704/231
• CPC Class Codes: G10L 15/01 Assessment or evaluation of speech recognition systems
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/065 Adaptation
• CPC Class Codes: G10L 15/14 using statistical models, e.g. Hidden Markov Models [HMMs] G10L15/18 takes precedence
• CPC Class Codes: G10L 15/187 Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/025 Phonemes, fenemes or fenones being the recognition units
• CPC Class Codes: G10L 2015/0631 Creating reference templates; Clustering
• CPC Class Codes: G10L 2015/0635 updating or merging of old and new templates; Mean values; Weighting

Abstract

A local feedback mechanism for customizing training models based on user data and directed user feedback is provided in speech recognition applications. The feedback data is filtered at different levels to address privacy concerns for local storage and for submittal to a system developer for enhancement of generic training models.

Claims

1. A method to be executed at least in part in a computing device for providing speech recognition with local and remote feedback loops, the method comprising:

collecting user data at a computing device executing a speech recognition application for the user;

aggregating collected user data through a feedback mechanism;

providing the aggregated user data to an adaptation module of the speech recognition application locally through the feedback mechanism; and

providing the aggregated user data to a remote system developer through the feedback mechanism for updating generic training models for the speech recognition application.

2. The method of claim 1, further comprising:

receiving updated generic training models from the system developer; and

updating current training models for the speech recognition application with the received training models.

3. The method of claim 1, further comprising:

filtering the aggregated data prior to providing to the system developer to remove potentially private data; and

filtering the aggregated data prior to providing to the adaptation module to prevent storage of private information in the computing device.

4. The method of claim 3, wherein the aggregated data is filtered at distinct levels for the local adaptation module and the system developer.

5. The method of claim 1, further comprising:

receiving directed user feedback to recognized speech; and

aggregating the directed user feedback with the collected user data.

6. The method of claim 1, wherein the user data includes audio and textual data.

7. The method of claim 6, wherein the user data is collected through harvesting of documents in the computing device employing an n-gram method.

8. The method of claim 7, wherein the user data is collected through harvesting of at least one from a set of:

emails, word processing documents, and spreadsheets in the computing device employing trigrams.

9. The method of claim 1, wherein the training models include at least one of:

a language model defining how words fit together to form sentences and an acoustic model defining how phonemes sound.

10. The method of claim 1, wherein the filtered data provided to the system developer from a plurality of users is analyzed statistically to determine updates to existing language and acoustic models.

11. The method of claim 10, wherein the statistical analysis includes determination of words popularized since a release of existing the language and acoustic models.

12. A speech recognition system for providing speech recognition with local and remote feedback loops, the system comprising:

a computing device executing a speech recognition application, the application including;

a feedback mechanism configured to;

collect user data in audio and textual format from the computing device;

receive directed feedback from the user in response to recognized speech; and

aggregate the collected user data and the directed feedback;

a local adaptation module configured to;

receive the aggregated data;

provide the aggregated data to the speech recognition application for customizing current language and acoustic models of the speech recognition application; and

a communication module configured to;

provide the aggregated data to a system developer for updating generic language and acoustic models for the speech recognition application.

13. The system of claim 12, further comprising:

a filtering module configured to;

receive the aggregated data from the feedback mechanism;

remove potentially private data from the aggregated data;

enable the user to provide permission for forwarding of the filtered data to the system developer; and

provide the filtered data to the communication module for forwarding to the system developer.

14. The system of claim 13, wherein removing the potentially private data from the aggregated data includes:

temporarily removing all potentially private data; and

upon determining that a subset of the potentially private data is non-private data, adding the non-private data to the filtered data prior to providing to the system developer.

15. The system of claim 13, wherein removing the potentially private data from the aggregated data includes removing numbers from the aggregated data.

16. The system of claim 12, wherein the speech recognition application is a hosted service executed over a plurality of computing devices in a distributed manner.

17. A computer-readable storage medium with instructions stored thereon for speech recognition with integrated local and remote feedback loops, the instructions comprising:

collecting user data in audio and textual format;

receiving directed feedback from the user in response to recognized speech;

aggregating the collected user data and the directed feedback;

filtering the aggregated data at different levels for the integrated local and the remote feedback loops to protect private data;

providing the data filtered at the local feedback level to a speech recognition engine through a local adaptation module to customize current language and acoustic models of the speech recognition engine;

providing the data filtered at the remote feedback level to a system developer for updating generic language and acoustic models for the speech recognition engine.

18. The computer-readable medium of claim 17, wherein the user data is collected through one of:

harvesting of at least one from a set of;

emails, word processing documents, and spreadsheets associated with the user employing trigrams; and

receiving a user spoken text passage.

19. The computer-readable medium of claim 18, wherein the filtered data provided to the system developer includes trigrams containing words from original language models and a list of added words.

20. The computer-readable medium of claim 17, wherein the filtered data is provided to the system developer through one of:

an email exchange, a file transfer, and an access through a hosted service.