Automatic indexing and aligning of audio and text using speech recognition

• US 5,649,060 A
• Filed: 10/23/1995
• Issued: 07/15/1997
• Est. Priority Date: 10/18/1993
• Status: Expired due to Term

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: Ellozy, Hamed A., Kanevsky, Dimitri, Nahamoo, David, Picheny, Michael Alan, Zadrozny, Wlodek Wlodzimierz, Kim, Michelle Y.
• Primary Examiner(s): MacDonald, Allen R.
• Assistant Examiner(s): Sartori, Michael A.
• Application Number:
  US08/547,113
• Time in Patent Office: 631 Days
• Field of Search: 395/2.4, 395/2.44, 395/2.55, 395/2.59, 395/2.6, 395/2.64, 395/2.79, 395/2.84, 395/2.85, 395/2.86, 395/2.87, 369/25, 364/419.02, 364/419.08
• US Class Current: 704/278
• CPC Class Codes: G06F 16/685 using automatically derived transcript of audio data, e.g. lyrics speech recognition G10L15/00
• CPC Class Codes: G10L 2015/226 using non-speech characteristics
• CPC Class Codes: G11B 27/028 with computer assistance
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence

Abstract

A method of automatically aligning a written transcript with speech in video and audio clips. The disclosed technique involves as a basic component an automatic speech recognizer. The automatic speech recognizer decodes speech (recorded on a tape) and produces a file with a decoded text. This decoded text is then matched with the original written transcript via identification of similar words or clusters of words. The results of this matching is an alignment of the speech with the original transcript. The method can be used (a) to create indexing of video clips, (b) for "teleprompting" (i.e. showing the next portion of text when someone is reading from a television screen), or (c) to enhance editing of a text that was dictated to a stenographer or recorded on a tape for its subsequent textual reproduction by a typist.

Claims

1. An apparatus for indexing an audio recording comprising:

an acoustic recorder for storing an ordered series of acoustic information signal units representing sounds generated from spoken words, said acoustic recorder having a plurality of recording locations, each recording location storing at least one acoustic information signal unit;

a timer connected to said acoustic recorder for time stamping said acoustic information signal units;

a speech recognizer connected to said acoustic recorder for generating an ordered series of recognized words having a high conditional probability of occurrence given the occurrence of the sounds represented by the acoustic information signal units from said acoustic recorder, each recognized word corresponding to at least one acoustic information signal unit and comprising a series of one or more characters, each recognized word having a context of at least one preceding or following recognized word;

a time alignment device connected to said speech recognizer and receiving time stamps of said acoustic information signal units for aligning said acoustic information signal units according to respective time stamps of said acoustic information signal units;

a text storage device for storing a transcript of text of the spoken words corresponding to ordered series of acoustic information signal units stored on said acoustic recorder;

mapping means connected to said text storage device for determining a size of an acoustic information signal unit to be passed to said speech recognizer from said acoustic recorder, said mapping means generating an ordered series of index words, said ordered series of index words comprising a representation of at least some of the spoken words represented by the acoustic information signal units, each index word having a context of at least one preceding or following index word and comprising a series of one or more characters;

a segmenter controlled by said mapping means for controlling playback of acoustic information signal units to said speech recognizer; and

alignment means connected to said acoustic recorder and to said mapping means for comparing the ordered series of recognized words with the ordered series of index words to pair recognized words and index words which are the same word and which have matching contexts, a recognized word being the same as an index word when both words comprise the same series of characters, a context of a target recognized word comprises the number of other recognized words preceding and following the target recognized word in the ordered series of recognized words, a context of a larger index word comprises the number of other index words preceding and following the target index word in the ordered series of index words, and the context of a recognized word matches the context of an index word if the context of the target recognized word is within a selected threshold value of the context of the target index word, said alignment means tagging each paired index word with the recording location of the acoustic information signal unit corresponding to the recognized word paired with the index word.

2. An apparatus as claimed in claim 1, wherein:

each index word which is not paired with a recognized word has a nearest preceding paired index word in the ordered series of index words, and has a nearest following paired index word in the ordered series of index words; and

the comparing means tags a non-paired index word with a recording location between the recording location of the nearest preceding paired index word and the recording location of the nearest following paired index word.

3. An apparatus as claimed in claim 2, wherein the speech recognizer aligns each recognized word with at least one acoustic information signal unit.

4. An apparatus as claimed in claim 3, wherein:

each recognized word which is not paired with an index word has a nearest preceding paired recognized word in the ordered series of recognized words, and has a nearest following paired recognized word in the ordered series of recognized words;

the context of a target recognized word comprises the number of other recognized words preceding the target recognized word and following the nearest preceding paired recognized word in the ordered series of recognized words;

the context of a target index word comprises the number of other index words preceding the target index word and following the nearest preceding paired index word in the ordered series of index words; and

the context of a recognized word matches the context of an index word if the context of the target recognized word is within a selected threshold value of the context of the target index word.

5. A method of indexing an audio recording comprising:

storing in an acoustic recorder an ordered series of acoustic information signal units representing sounds generated from spoken words, said acoustic recorder having a plurality of recording locations, each recording location storing at least one acoustic information signal unit;

time stamping said acoustic information signal units;

generating by a speech recognizer an ordered series of recognized words having a high conditional probability of occurrence given the occurrence of the sounds represented by the acoustic information signal units from said acoustic recorder, each recognized word corresponding to at least one acoustic information signal unit and comprising a series of one or more characters, each recognized word having a context of at least one preceding or following recognized word;

time aligning said acoustic information signal units according to respective time stamps of said acoustic information signal units;

storing a transcript of text of the spoken words corresponding to ordered series of acoustic information signal units on said acoustic recorder;

mapping the transcript of text to determine a size of an acoustic information signal unit to be passed to said speech recognizer from said acoustic recorder;

generating an ordered series of index words, said ordered series of index words comprising a representation of at least some of the spoken words represented by the acoustic information signal units, each index word having a context of at least one preceding or following index word and comprising a series of one or more characters;

segmenting said ordered series of index words to control playback of acoustic information signal units by said acoustic recorder;

comparing the ordered series of recognized words with the ordered series of index words to pair recognized words and index words which are the same word and which have matching contexts, a recognized word being the same as an index word when both words comprise the same series of characters, a context of a target recognized word comprises the number of other recognized words preceding and following the target recognized word in the ordered series of recognized words, a context of a target index word comprises the number of other index words preceding and following the target index word in the ordered series of index words, and the context of a recognized word matches the context of an index word if the context of the target recognized word is within a selected threshold value of the context of the target index word; and

tagging each paired index word with the recording location of the acoustic information signal unit corresponding to the recognized word paired with the index word.

6. A method as claimed in claim 5, wherein:

each index word which is not paired with a recognized word has a nearest preceding paired index word in the ordered series of index words, and has a nearest following paired index word in the ordered series of index words; and

the step of tagging comprises tagging a non-paired index word with a recording location between the recording location of the nearest preceding paired index word and the recording location of the nearest following paired index word.

7. A method as claimed in claim 6, further comprising the step of aligning each recognized word with at least one acoustic information signal unit.

8. A method as claimed in claim 7, wherein:

each recognized word which is not paired with an index word has a nearest preceding paired recognized word in the ordered series of recognized words, and has a nearest following paired recognized word in the ordered series of recognized words;

the context of a target recognized word comprises the number of other recognized words preceding the target recognized word and following the nearest preceding paired recognized word in the ordered series of recognized words;

the context of a target index word comprises the number of other index words preceding the target index word and following the nearest preceding paired index word in the ordered series of index words; and

the context of a recognized word matches the context of an index word if the context of the target recognized word is within a selected threshold value of the context of the target index word.