System and method of lattice-based search for spoken utterance retrieval

• US 8,670,977 B2
• Filed: 03/21/2011
• Issued: 03/11/2014
• Est. Priority Date: 08/23/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: AT&T Intellectual Property II LP (AT&T, Inc.)
• Inventors: Saraclar, Murat, Sproat, Richard William
• Primary Examiner(s): ALBERTALLI, BRIAN LOUIS
• Application Number:
  US13/052,819
• Publication Number:
  US 20110173226A1
• Time in Patent Office: 1,086 Days
• Field of Search: None
• US Class Current: 704/9
• CPC Class Codes: G06F 16/632 Query formulation
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 15/142 Hidden Markov Models [HMMs]
• CPC Class Codes: G10L 15/197 Probabilistic grammars, e.g. word n-grams
• CPC Class Codes: G10L 2015/085 Methods for reducing search complexity, pruning

Abstract

A system and method are disclosed for retrieving audio segments from a spoken document. The spoken document preferably is one having moderate word error rates such as telephone calls or teleconferences. The method comprises converting speech associated with a spoken document into a lattice representation and indexing the lattice representation of speech. These steps are performed typically off-line. Upon receiving a query from a user, the method further comprises searching the indexed lattice representation of speech and returning retrieved audio segments from the spoken document that match the user query.

Claims

1. A method comprising:

receiving a query from a user, the query comprising a query word;

retrieving, based on the query, a spoken document;

converting, via a processor, the query word into query word phoneme strings based on a query word pronunciation of the query word;

searching a phoneme-based index of a lattice representation of the spoken document for phoneme strings that correspond to the query word phoneme strings to yield search results, the phoneme-based index comprising an index for each arc label that records a lattice number, an input-state of each labeled arc, a probability mass leading to each state, a probability associated with each arc, and an index for a next state; and

returning audio segments from the spoken document that correspond to the query based on the search results.

2. The method of claim 1, wherein the spoken document is generated from a telephone call.

3. The method of claim 1, wherein when the query word pronunciation is not in a baseline pronunciation dictionary, generating the query word pronunciation via a text-to-speech module.

4. The method of claim 1, further comprising obtaining a phonetic representation of speech utterances associated with the spoken document by using a hybrid language model for out-of-vocabulary word detection.

5. The method of claim 4, wherein the hybrid language model is a combination of a phoneme level language model and a word-level language model.

6. The method of claim 1, wherein the query comprises one of text and speech.

7. The method of claim 1, wherein the phoneme-based index of the lattice representation of the spoken document is both a phoneme-based word index and a phoneme-based sub-word index.

8. The method of claim 1, wherein the phoneme-based index of the lattice representation of the spoken document is both a word index and a sub-word index.

9. The method of claim 8, wherein the word index and the sub-word index are generated based on high word error rate speech from the spoken document.

10. A system comprising:

a processor; and

a computer-readable storage medium having instructions stored which, when executed by the processor, cause the processor to perform operations comprising;

receiving a query from a user, the query comprising a query word;

retrieving, based on the query, a spoken document;

converting the query word into phoneme strings based on a word pronunciation of the query word;

searching a phoneme-based indexed lattice representation of the spoken document for phoneme strings that correspond to the phoneme strings to yield search results, the phoneme-based indexed lattice representation comprising an index for each arc label that records a lattice number, an input-state of each labeled arc, a probability mass leading to each state, a probability of each arc, and an index for a next state; and

returning audio segments from the spoken document that match the query based on the search results.

11. The system of claim 10, wherein the spoken document is generated from a telephone call.

12. The system of claim 10, the computer-readable storage medium having additional instructions stored which result in the operations further comprising:

upon receiving the query from the user, generating text-to-speech query word pronunciations when the query word pronunciation is not in a baseline pronunciation dictionary.

13. The system of claim 10, the computer-readable storage medium having additional instructions stored which result in the operations further comprising:

upon receiving the query from the user, generating a phonetic representation of speech utterances associated with the spoken document by using a hybrid language model for out-of-vocabulary word detection.

14. The system of claim 13, wherein the hybrid language model is a combination of a phoneme level language model and a word-level language model.

15. The system of claim 10, wherein the query comprises one of text and speech.

16. A computer-readable storage medium device having instructions stored which, when executed by a computing device, cause the computing device to perform operations comprising:

receiving a query from a user, the query comprising a query word;

retrieving, based on the query, a spoken document;

converting the query word into phoneme strings based on a query word pronunciation of the query word;

searching a phoneme-based indexed lattice representation of the spoken document for phoneme strings that correspond to the query word phoneme strings, to yield search results, the phoneme-based index comprising an index for each arc label that records a lattice number, an input-state of each labeled arc, a probability mass leading to each state, a probability of the arc itself, and an index for a next state; and

returning audio segments from the spoken document that correspond to the query based on the search results.

17. The computer-readable storage device of claim 16, wherein the phoneme-based indexed lattice representation of the spoken document is both a phoneme-based word index and a phoneme-based sub-word index.

18. The computer-readable storage device of claim 17, having additional instructions stored which result in the perations further comprising searching the phoneme-based sub-word index only after not finding corresponding results upon searching the phoneme-based word index.

19. The computer-readable storage device of claim 16, wherein the phoneme-based index of the lattice representation of the spoken document is both a word index and a sub-word index.

20. The computer-readable storage device of claim 19, wherein the word index and the sub-word index are generated based on high word error rate speech from the spoken document.