Indexing Digitized Speech With Words Represented In The Digitized Speech

• US 20080235021A1
• Filed: 03/20/2007
• Published: 09/25/2008
• Est. Priority Date: 03/20/2007
• Status: Active Grant

• Current Assignee: Innovationsgesellschaft Technische Universität mbH
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Jania, Frank L., Cross, Charles W.
• Granted Patent: US 8,515,757 B2
• Time in Patent Office: Days
• US Class Current: 704/257
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/19 Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules
• CPC Class Codes: G10L 15/193 Formal grammars, e.g. finite state automata, context free grammars or word networks
• CPC Class Codes: G10L 15/197 Probabilistic grammars, e.g. word n-grams
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/228 of application context
• CPC Class Codes: G10L 21/06 Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids G10L15/26 takes precedence

Abstract

Indexing digitized speech with words represented in the digitized speech, with a multimodal digital audio editor operating on a multimodal device supporting modes of user interaction, the modes of user interaction including a voice mode and one or more non-voice modes, the multimodal digital audio editor operatively coupled to an ASR engine, including providing by the multimodal digital audio editor to the ASR engine digitized speech for recognition; receiving in the multimodal digital audio editor from the ASR engine recognized user speech including a recognized word, also including information indicating where, in the digitized speech, representation of the recognized word begins; and inserting by the multimodal digital audio editor the recognized word, in association with the information indicating where, in the digitized speech, representation of the recognized word begins, into a speech recognition grammar, the speech recognition grammar voice enabling user interface commands of the multimodal digital audio editor.

Claims

1. A method of indexing digitized speech with words represented in the digitized speech, the method implemented with a multimodal digital audio editor operating on a multimodal device supporting multiple modes of user interaction with the multimodal digital audio editor, the modes of user interaction including a voice mode and one or more non-voice modes, the multimodal digital audio editor operatively coupled to an ASR engine, the method comprising:

providing by the multimodal digital audio editor to the ASR engine digitized speech for recognition;

receiving in the multimodal digital audio editor from the ASR engine recognized user speech including a recognized word, also including information indicating where, in the digitized speech, representation of the recognized word begins; and

inserting by the multimodal digital audio editor the recognized word, in association with the information indicating where, in the digitized speech, representation of the recognized word begins, into a speech recognition grammar, the speech recognition grammar voice enabling user interface commands of the multimodal digital audio editor.

2. The method of claim 1 further comprising visually displaying the digitized speech with the recognized word as an index of where in the digitized speech the representation of the recognized word begins.

3. The method of claim 1 further comprising recognizing by the ASR engine a word in the digitized speech, including identifying information indicating where, in the digitized speech, representation of the word begins.

4. The method of claim 3 wherein identifying the information indicating where, in the digitized speech, representation of the word begins further comprises.

organizing the digitized speech in sequential sets of time domain amplitude samples grouped in frames, each frame characterized in sequence by a unique and cardinal frame identification number, each frame containing the same number of time domain amplitude samples;

converting the digitized speech containing the word to the frequency domain beginning with one of the frames of time domain amplitude samples; and

deriving an index value indicating where, in the digitized speech, representation of the word begins by multiplying the one of the frame identification numbers by the number of amplitude samples in each frame.

5. The method of claim 1 wherein inserting the word into a speech recognition grammar further comprises associating the recognized word, as a non-optional terminal element in the speech recognition grammar, with a word representing a user interface command of the digital audio editor.

6. The method of claim 1 wherein the speech recognition grammar voice enabling user interface commands of the multimodal digital audio editor further comprises inserting by the multimodal digital audio editor into the grammar as part of a non-optional terminal grammar element the information indicating where, in the digitized speech, representation of the recognized word begins.

7. Apparatus for indexing digitized speech with words represented in the digitized speech, the apparatus implemented with a multimodal digital audio editor operating on a multimodal device supporting multiple modes of user interaction with the multimodal digital audio editor, the modes of user interaction including a voice mode and one or more non-voice modes, the multimodal digital audio editor operatively coupled to an ASR engine, the apparatus comprising a computer processor and a computer memory operatively coupled to the computer processor, the computer memory having disposed within it computer program instructions capable of:

providing from the multimodal digital audio editor to the ASR engine digitized speech for recognition;

receiving in the multimodal digital audio editor from the ASR engine recognized user speech including a recognized word, also including information indicating where, in the digitized speech, representation of the recognized word begins; and

inserting by the multimodal digital audio editor the recognized word, in association with the information indicating where, in the digitized speech, representation of the recognized word begins, into a speech recognition grammar, the speech recognition grammar voice enabling user interface commands of the multimodal digital audio editor.

8. The apparatus of claim 7 further comprising computer program instructions capable of visually displaying the digitized speech with the recognized word as an index of where in the digitized speech the representation of the recognized word begins.

9. The apparatus of claim 7 further comprising computer program instructions capable of recognizing by the ASR engine a word in the digitized speech, including identifying information indicating where, in the digitized speech, representation of the word begins.

10. The apparatus of claim 9 wherein identifying the information indicating where, in the digitized speech, representation of the word begins further comprises.

organizing the digitized speech in sequential sets of time domain amplitude samples grouped in frames, each frame characterized in sequence by a unique and cardinal frame identification number, each frame containing the same number of time domain amplitude samples;

converting the digitized speech containing the word to the frequency domain beginning with one of the frames of time domain amplitude samples; and

deriving an index value indicating where, in the digitized speech, representation of the word begins by multiplying the one of the frame identification numbers by the number of amplitude samples in each frame.

11. The apparatus of claim 7 wherein inserting the word into a speech recognition grammar further comprises associating the recognized word, as a non-optional terminal element in the speech recognition grammar, with a word representing a user interface command of the digital audio editor.

12. The apparatus of claim 7 wherein the speech recognition grammar voice enabling user interface commands of the multimodal digital audio editor further comprises inserting by the multimodal digital audio editor into the grammar as part of a non-optional terminal grammar element the information indicating where, in the digitized speech, representation of the recognized word begins.

13. A computer program product for indexing digitized speech with words represented in the digitized speech, the apparatus implemented with a multimodal digital audio editor operating on a multimodal device supporting multiple modes of user interaction with the multimodal digital audio editor, the modes of user interaction including a voice mode and one or more non-voice modes, the multimodal digital audio editor operatively coupled to an ASR engine, the computer program product disposed upon a computer-readable, signal-bearing medium, the computer program product comprising computer program instructions capable of:

providing from the multimodal digital audio editor to the ASR engine digitized speech for recognition;

receiving in the multimodal digital audio editor from the ASR engine recognized user speech including a recognized word, also including information indicating where, in the digitized speech, representation of the recognized word begins; and

inserting by the multimodal digital audio editor the recognized word, in association with the information indicating where, in the digitized speech, representation of the recognized word begins, into a speech recognition grammar, the speech recognition grammar voice enabling user interface commands of the multimodal digital audio editor.

14. The computer program product of claim 13 wherein the computer-readable, signal-bearing medium comprises a recordable medium.

15. The computer program product of claim 13 wherein the computer-readable, signal-bearing medium comprises a transmission medium.

16. The computer program product of claim 13 further comprising computer program instructions capable of visually displaying the digitized speech with the recognized word as an index of where in the digitized speech the representation of the recognized word begins.

17. The computer program product of claim 13 further comprising computer program instructions capable of recognizing by the ASR engine a word in the digitized speech, including identifying information indicating where, in the digitized speech, representation of the word begins.

18. The computer program product of claim 17 wherein identifying the information indicating where, in the digitized speech, representation of the word begins further comprises.

organizing the digitized speech in sequential sets of time domain amplitude samples grouped in frames, each frame characterized in sequence by a unique and cardinal frame identification number, each frame containing the same number of time domain amplitude samples;

converting the digitized speech containing the word to the frequency domain beginning with one of the frames of time domain amplitude samples; and

deriving an index value indicating where, in the digitized speech, representation of the word begins by multiplying the one of the frame identification numbers by the number of amplitude samples in each frame.

19. The computer program product of claim 13 wherein inserting the word into a speech recognition grammar further comprises associating the recognized word, as a non-optional terminal element in the speech recognition grammar, with a word representing a user interface command of the digital audio editor.

20. The computer program product of claim 13 wherein the speech recognition grammar voice enabling user interface commands of the multimodal digital audio editor further comprises inserting by the multimodal digital audio editor into the grammar as part of a non-optional terminal grammar element the information indicating where, in the digitized speech, representation of the recognized word begins.