Identifying keyword occurrences in audio data

• US 8,423,363 B2
• Filed: 01/13/2010
• Issued: 04/16/2013
• Est. Priority Date: 01/13/2009
• Status: Active Grant

• Current Assignee: Centre de Recherche Informatique de Montréal
• Original Assignee: Centre de Recherche Informatique de Montréal
• Inventors: Gupta, Vishwa Nath, Boulianne, Gilles
• Primary Examiner(s): PULLIAS, JESSE SCOTT
• Application Number:
  US12/686,892
• Publication Number:
  US 20100179811A1
• Time in Patent Office: 1,189 Days
• Field of Search: 704/231, 704/235, 704/251, 704/257, 704/270, 704/270.1
• US Class Current: 704/255
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/088 Word spotting

Abstract

Occurrences of one or more keywords in audio data are identified using a speech recognizer employing a language model to derive a transcript of the keywords. The transcript is converted into a phoneme sequence. The phonemes of the phoneme sequence are mapped to the audio data to derive a time-aligned phoneme sequence that is searched for occurrences of keyword phoneme sequences corresponding to the phonemes of the keywords. Searching includes computing a confusion matrix. The language model used by the speech recognizer is adapted to keywords by increasing the likelihoods of the keywords in the language model. For each potential occurrences keywords detected, a corresponding subset of the audio data may be played back to an operator to confirm whether the potential occurrences correspond to actual occurrences of the keywords.

Claims

1. A method of identifying occurrences of a keyword within audio data, the method comprising:

a) providing a computer based processing entity programmed with software, the software implementing a language model to perform speech recognition;

b) inputting in the processing entity data conveying the keyword;

c) processing the data conveying the keyword with the software to adapt the language model to the keyword and generate an adapted language model, wherein processing the data conveying the keyword with the software to adapt the language model to the keyword comprises increasing a likelihood of the keyword in the language model;

d) processing the audio data with the adapted language model to determine if the audio data contains the keyword;

e) releasing result data at an output of the processing entity conveying results of the processing of the audio data with the adapted language model.

2. The method of claim 1, wherein increasing the likelihood of the keyword in the language model comprises boosting a log-likelihood of every n-gram that contains the keyword.

3. The method of claim 1, further comprising prior to increasing the likelihood of the keyword in the language model, adding the keyword to the language model.

4. The method of claim 1, wherein processing the audio data with the adapted language model comprises performing speech recognition on the audio data using the adapted language model to derive a transcript of the audio data.

5. The method of claim 4, wherein the speech recognition is continuous speech recognition.

6. The method of claim 4, wherein processing the audio data with the adapted language model further comprises performing a text-to-phoneme conversion on the transcript to derive a first phoneme sequence.

7. The method of claim 6, wherein the text-to-phoneme conversion is performed continuously.

8. The method of claim 6, wherein processing the audio data with the adapted language model further comprises mapping the phonemes in the first phoneme sequence to the audio data to derive a time-aligned phoneme sequence.

9. The method of claim 8, wherein mapping the phonemes in the first phoneme sequence to the audio data is performed continuously.

10. The method of claim 8, wherein processing the audio data with the adapted language model further comprises searching the time-aligned phoneme sequence for occurrences of a keyword phoneme sequence corresponding to the keyword.

11. The method of claim 10, wherein searching the time-aligned phoneme sequence for occurrences of a keyword phoneme sequence comprises computing a confusion matrix.

12. The method of claim 10, wherein the speech recognition is performed using a speaker-independent acoustic model.

13. The method of claim 10, wherein speech recognition is performed using an adapted acoustic model derived from an initial time-aligned phoneme sequence.

14. The method of claim 13, wherein the transcript of the audio data;

is a second transcript, the method further comprising;

a) performing speech recognition on the audio data using a speaker-independent acoustic model to derive an initial transcript of the audio data;

b) performing a text-to-phoneme conversion on the initial transcript of the audio data;

to derive an initial phoneme sequence;

c) mapping the phonemes in the initial phoneme sequence to the audio data to derive an initial time-aligned phoneme sequence; and

d) generating the adapted acoustic model on the basis of the initial time-aligned phoneme sequence and the speaker-independent acoustic model.

15. The method of claim 14, wherein the adapted acoustic model is generated using feature-based maximum likelihood linear regression (fMLLR).

16. The method of claim 10, further comprising performing speaker diarization on the audio data to identify segments of the audio data corresponding to individual speakers;

wherein the transcript of the audio data is a per-speaker transcript of the audio data derived by performing speech recognition using a per-speaker adapted acoustic model and an indication of the segments of the recording corresponding to individual speakers.

17. The method of claim 1, further comprising inputting in the processing entity a signal indicative of the audio data.

18. The method of claim 17, wherein the audio data comprises at least one recording, wherein the signal indicative of the audio data identifies the at least one recording from among a collection of recordings.

19. A method of identifying occurrences of keywords within audio recordings containing speech information, the method comprising:

a) providing a computer based processing entity programmed with software, the software implementing a language model to perform speech recognition;

b) inputting in the processing entity first data conveying a first keyword;

c) processing the first data with the software to adapt the language model to the first keyword and generate a language model adapted to the first keyword, wherein processing the first data with the software to adapt the language model to the first keyword comprises increasing a likelihood of the first keyword in the language model;

d) processing a first set of recordings with the language model adapted to the first keyword to determine if the first set of recordings contains the first keyword;

e) inputting in the processing entity second data conveying a second keyword;

f) processing the second data with the software to adapt the language model to the second keyword and generate a language model adapted to the second keyword, wherein processing the second data with the software to adapt the language model to the second keyword comprises increasing the likelihood of the second keyword in the language model;

g) processing a second set of recordings with the language model adapted to the second keyword to determine if the second set of recordings contains the second keyword;

releasing data at an output of the processing entity conveying results of the processing of the first and second sets of recordings with the language models adapted to the first and second keywords, respectively.

20. A method of identifying occurrences of a keyword within audio data, the method comprising:

a) providing a computer based processing entity programmed with software, the software implementing a language model to perform speech recognition;

b) performing speech recognition on the audio data using a speaker-independent acoustic model to derive a first transcript of the audio data;

c) performing a text-to-phoneme conversion on the first transcript of the recording to derive a first phoneme sequence;

d) mapping the phonemes in the first phoneme sequence to the audio data to derive a first time-aligned phoneme sequence; and

e) generating an adapted acoustic model on the basis of the first time-aligned phoneme sequence and the speaker-independent acoustic model;

f) inputting in the processing entity data conveying the keyword;

g) processing the data conveying the keyword with the software to adapt the language model to the keyword and generate an adapted language model;

h) processing the audio data with the adapted language model to determine if the audio data contains the keyword;

i) wherein processing the audio data with the adapted language model comprises;

performing speech recognition on the audio data using the adapted language model to derive a second transcript of the audio data, wherein speech recognition is performed using the adapted acoustic model;

performing a text-to-phoneme conversion on the second transcript to derive a second phoneme sequence;

mapping the phonemes in the second phoneme sequence to the audio data to derive a second time-aligned phoneme sequence;

searching the second time-aligned phoneme sequence for occurrences of a keyword phoneme sequence corresponding to the keyword;

j) releasing result data at an output of the processing entity conveying results of the processing of the audio data with the adapted language model.

21. The method of claim 20, wherein the adapted acoustic model is generated using feature-based maximum likelihood linear regression (fMLLR).