Method for automatically identifying sentence boundaries in noisy conversational data

• US 8,364,485 B2
• Filed: 08/27/2007
• Issued: 01/29/2013
• Est. Priority Date: 08/27/2007
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Nasukawa, Tetsuya, Punjani, Diwakar, Roy, Shourya, Subramaniam, L. Venkata, Takeuchi, Hironori
• Primary Examiner(s): COLUCCI, MICHAEL C
• Application Number:
  US11/845,462
• Publication Number:
  US 20090063150A1
• Time in Patent Office: 1,982 Days
• Field of Search: 704/235, 704/9, 704/4, 704/253, 704/10, 379/168, 379/75, 707/999.003, 715/259, 715/246
• US Class Current: 704/253
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

Sentence boundaries in noisy conversational transcription data are automatically identified. Noise and transcription symbols are removed, and a training set is formed with sentence boundaries marked based on long silences or on manual markings in the transcribed data. Frequencies of head and tail n-grams that occur at the beginning and ending of sentences are determined from the training set. N-grams that occur a significant number of times in the middle of sentences in relation to their occurrences at the beginning or ending of sentences are filtered out. A boundary is marked before every head n-gram and after every tail n-gram occurring in the conversational data and remaining after filtering. Turns are identified. A boundary is marked after each turn, unless the turn ends with an impermissible tail word or is an incomplete turn. The marked boundaries in the conversational data identify sentence boundaries.

Claims

1. A method for automatically identifying sentence boundaries in noisy conversational transcription data, comprising:

pre-processing on a computing device the noisy conversational transcription data to remove transcription symbols and noise to produce processed transcription data;

marking with the computing device sentence boundaries in the processed transcription data based on manually marked sentence boundaries in the processed transcription data, wherein said marked transcription data forms a training set;

determining frequencies of head and tail n-grams that occur at the beginning and ending of sentences in the training set;

determining the frequencies that the head and tail n-grams occur in the middle of sentences;

filtering out from the training set n-grams that occur a significant number of times in the middle of sentences in relation to the frequencies at which the n-gram occur at the beginning or ending of sentences;

marking a boundary in the conversational data before every head n-gram and after every tail n-gram that occurs in the conversational data and that also remains in the training set after filtering;

identifying turns occurring in the conversational data indicating a speaker change in the conversational data; and

marking a boundary in the conversational data after each turn, unless the turn ends with an impermissible tail word or includes a word indicating an incomplete turn;

wherein the steps of marking identify sentence boundaries in the conversational data.

2. The method of claim 1, further comprising removing false boundaries from the conversational data.

3. The method of claim 1, wherein the step of filtering includes using a threshold ratio of occurrences of head and tail n-grams to n-grams occurring in the middle of sentences in the training set.

4. The method of claim 1, wherein the step of filtering includes removing impermissible head or tail n-grams.