Method and System of Indexing Speech Data

• US 20090030680A1
• Filed: 07/23/2007
• Published: 01/29/2009
• Est. Priority Date: 07/23/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Mamou, Jonathan Joseph
• Granted Patent: US 8,831,946 B2
• Time in Patent Office: Days
• US Class Current: 704/235
• CPC Class Codes: G06F 16/685 using automatically derived transcript of audio data, e.g. lyrics speech recognition G10L15/00
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

A method and system of indexing speech data. The method includes indexing word transcripts including a timestamp for a word occurrence; and indexing sub-word transcripts including a timestamp for a sub-word occurrence. A timestamp in the index indicates the time and duration of occurrence of the word or sub-word in the speech data, and word and sub-word occurrences can be correlated using the timestamps. A method of searching speech transcripts is also provided in which a search query in the form of a phrase to be searched includes at least one in-vocabulary word and at least one out-of-vocabulary word. The method of searching includes extracting the search terms from the phrase, retrieving a list of occurrence of words for an in-vocabulary search term from an index of words having timestamps, retrieving a list of occurrences of sub-words for an out-of-vocabulary search term from an index of sub-words having timestamps, and merging the retrieved lists of occurrences of words and sub-words according to their timestamps.

Claims

1. A method of indexing speech data, the method comprising:

indexing word transcripts, including a timestamp for a word occurrence;

indexing sub-word transcripts, including a timestamp for a sub-word occurrence;

wherein a timestamp in the index indicates the time of occurrence of the word or sub-word in the speech data;

and wherein word and sub-word occurrences can be correlated using the timestamps.

2. The method of indexing of claim 1, wherein speech data is processed by a speech recognition system having a vocabulary of sub-words, to provide a sub-word transcript of the speech data, wherein sub-words are phones, morphemes, syllables, or sequences of phones from the vocabulary of sub-words.

3. The method of indexing of claim 1, wherein speech data is processed by a speech recognition system having a vocabulary of words, to provide a word transcript of the speech data, wherein each transcribed word is in the vocabulary of words.

4. The method of indexing as claimed in claim 1, wherein indexing word transcripts and sub-word transcripts includes an identifier of a source of the word or sub-word transcript.

5. The method of indexing as claimed in claim 1, wherein the timestamp is a start time and duration of the word or sub-word in the speech data.

6. The method of indexing as claimed in claim 1, wherein the method is applied to a video input, and image features are correlated to the indexing of words and sub-words transcripts according to the timestamps.

7. The method of indexing as claimed in claim 6, including indexing image features with a timestamp for the occurrence of an image feature in the video input.

8. A method of searching speech data including in-vocabulary and out-of-vocabulary words, comprising:

receiving a search query in the form of a phrase to be searched including at least one in-vocabulary word and at least one out-of-vocabulary word;

extracting the search terms from the phrase;

retrieving a list of occurrences of words for an in-vocabulary search term from an index of words having timestamps;

retrieving a list of occurrences of sub-words for an out-of-vocabulary search term from an index of sub-words having timestamps; and

merging the retrieved lists of occurrences of words and sub-words according to their timestamps.

9. The method of searching as claimed in claim 8, wherein sub-word transcripts are generated by a speech recognition system with a vocabulary of sub-words, wherein sub-words are phones, morphemes, syllables, or sequences of phones from the vocabulary of sub-words.

10. The method of searching as claimed in claim 8, wherein word transcripts are generated by a speech recognition system with a vocabulary of words, and in-vocabulary words are words in the vocabulary of words.

11. The method of searching of claim 8, including checking that the words and sub-words appear in the correct order according to start times of the timestamps.

12. The method of searching of claim 8, including checking that the words and sub-words are adjacent according to start times and durations of timestamps by checking that the time between adjacent words/sub-words is below a given threshold.

13. The method of searching of claim 8, wherein the search results are ranked according to their temporal proximity.

14. A system for searching speech transcripts from speech recognition systems, wherein a speech recognition system has a vocabulary of words, the system including an indexing system comprising:

indexing word transcripts, including a timestamp for a word occurrence;

indexing sub-word transcripts, including a timestamp for a sub-word occurrence;

wherein a timestamp in the index indicates the time of occurrence of the word or sub-word in the speech data;

and wherein word and sub-word occurrences can be correlated using the timestamps.

15. The system as claimed in claim 14, wherein the index of word transcripts and the index of sub-word transcripts include an identifier of a source of the word or sub-word transcript.

16. The system as claimed in claim 14, wherein the index of word transcripts and the index of sub-word transcripts are combined in an index with an indicator of the type of word or sub-word.

17. The system as claimed in claim 14, including a searching system comprising:

means for extracting search terms from a phrase including at least one in-vocabulary word and at least one out-of-vocabulary word;

means for retrieving a list of occurrences of words for an in-vocabulary search term from the index of words and for retrieving a list of occurrences of sub-words for an out-of-vocabulary search term from the index of sub-words; and

means for merging the retrieved lists of occurrences of words and sub-words according to their timestamps.

18. The system of claim 17, wherein the means for merging includes means for checking that the words and sub-words appear in the correct order according to start times of the timestamps.

19. The system of claim 17, wherein the means for merging includes means for checking that the words and sub-words are adjacent according to the timestamps.

20. A computer program product stored on a computer readable storage medium for indexing speech data, comprising computer readable program code means for performing the steps of:

indexing word transcripts, including a timestamp for a word occurrence;

indexing sub-word transcripts, including a timestamp for a sub-word occurrence;

wherein a timestamp in the index indicates the time of occurrence of the word or sub-word in the speech data;

and wherein word and sub-word occurrences can be correlated using the timestamps.