Word-level blind diarization of recorded calls with arbitrary number of speakers

• US 9,875,742 B2
• Filed: 01/26/2016
• Issued: 01/23/2018
• Est. Priority Date: 01/26/2015
• Status: Active Grant

• Current Assignee: Verint Systems Incorporated
• Original Assignee: Verint Systems Limited (Verint Systems Incorporated)
• Inventors: Gorodetski, Alex, Sidi, Oana, Wein, Ron, Shapira, Ido
• Primary Examiner(s): Shin, Seong-Ah A.
• Application Number:
  US15/006,572
• Publication Number:
  US 20160217792A1
• Time in Patent Office: 728 Days
• Field of Search: 704240, 704245, 704246, 704250, 348 1412, 381 7111
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 17/02 Preprocessing operations, e.g. segment selection; Pattern representation or modelling, e.g. based on linear discriminant analysis [LDA] or principal components; Feature selection or extraction
• CPC Class Codes: G10L 17/04 Training, enrolment or model building
• CPC Class Codes: G10L 17/16 Hidden Markov models [HMM]
• CPC Class Codes: G10L 25/84 for discriminating voice from noise

Abstract

Disclosed herein are methods of diarizing audio data using first-pass blind diarization and second-pass blind diarization that generate speaker statistical models, wherein the first pass-blind diarization is on a per-frame basis and the second pass-blind diarization is on a per-word basis, and methods of creating acoustic signatures for a common speaker based only on the statistical models of the speakers in each audio session.

Claims

1. A method of creating an acoustic signature for a speaker from multiple audio sessions and for performing diarization, the method comprising:

receiving, from an audio data source, audio data at an audio communications interface of a computing system, the audio data defining a training set containing a number of recorded audio sessions, wherein the computing system is configured to construct, from each audio session, a plurality of respective speaker models, wherein each speaker model is characterized by aggregating acoustic features into respective feature vectors that define a respective occupancy which is proportional to a total number of feature vectors used to construct the speaker model, and wherein the speaker models are Gaussian mixture models (GMMs) defined over a common set of Gaussian distributions that differ only by respective mixture probabilities for the acoustic features present in the feature vectors;

classifying the plurality of speaker models to identify a set of common speaker GMMs and a set of generic speaker GMMs, wherein the classifying includes constructing an undirected similarity graph having vertices corresponding to the plurality of respective speaker models of all the recorded audio sessions in the training set and classifying the plurality of speaker models according to a degree of similarity between the corresponding vertices in the undirected similarity graph in relation to at least one threshold degree of similarity;

generating an acoustic signature by at least;

constructing a super-GMM for the set of common speaker GMMs, and

constructing a second super-GMM for the set of generic speaker GMMs by generating a set of random vectors and training a second GMM over these random vectors, wherein a respective acoustic signature for a common speaker is given as a super-model pair of the two constructed super-GMMs;

storing the two constructed super-GMMs in a computing system memory;

receiving additional audio data at the audio communications interface;

identifying the common speaker using the super-model pair; and

labeling the additional audio data with an identified common speaker label.

2. A The method according to claim 1, wherein the audio data comprises one of a .WAV format, a PCM format, and a LPCM format.

3. The method according to claim 1, further comprising, decoding at least one segment of the audio data, transcribing the segment of audio data, producing a diarized transcript, and using a voice activity detector to classify the audio data into speech and non-speech segments by assessing a dynamic energy range for each segment of audio data.

4. The method according to claim 1, wherein the audio data originates from a recording stored on a server.

5. The method according to claim 1, wherein the audio data is a real time stream of audio data.

6. The method according to claim 1, further comprising displaying the undirected graph on a graphical display connected to the computing system.

7. The method of for creating a plurality of acoustic signatures and for performing diarization, comprising:

receiving audio data at a communication interface of a computing system on a frame by frame basis,

creating a speech to text transcription of the audio data;

clustering respective segments of the audio data according to word sequences;

classifying the segments to identify a set of common speaker Gaussian mixture models (GMMs) and a set of generic speaker GMMs, wherein the classifying includes constructing an undirected similarity graph having vertices corresponding to a plurality of speaker models of previously recorded audio sessions in a training set;

wherein the classifying further includes determining with a processor in the computing system a degree of similarity between the corresponding vertices in the undirected similarity graph in relation to at least one threshold degree of similarity;

generating an acoustic signature by at least;

constructing a super-GMM for the set of common speaker GMMs, and

constructing a second super-GMM for the set of generic speaker GMMs by generating a set of random vectors and training a GMM over these random vectors, wherein the acoustic signature for respective common speakers is given as a super-model pair of the two constructed super-GMMs; and

storing the two constructed super-GMMs in a computing system memory;

receiving additional audio data at the communication interface;

identifying a respective common speaker using the super-model pair; and

labeling the additional audio data with an identified common speaker label.

8. The method according to claim 7, further comprising utilizing a diagonal Gaussian distribution for the clusters to calculate the log likelihood.

9. The method according to claim 7, wherein the prerecorded training sets reside on a server.

10. The method according to claim 7, further comprising filtering short utterances as background audio.

11. The method according to claim 7, further comprising filtering out short utterances on a time duration basis.

12. The method according to claim 7, further comprising classifying the clusters according to Mel-frequency cepstral coefficients (MFCC) for each frame.

13. The method according to claim 7, further comprising:

determining a cluster of segments to be comprised of respective utterances and representing a distribution of feature vectors in the respective utterances;

characterizing each feature vector in terms of its probability of being present in one of the clusters;

calculating a distance metric between utterances according to the probability;

identifying time between speakers in the audio stream.

14. The method according to claim 13, further comprising:

using distances between utterances to construct an affinity matrix based upon respective distances;

computing a stochastic matrix from the affinity matrix;

computing eigenvalues and corresponding eigenvectors of the stochastic matrix; and

computing an embedding of the utterances into dimensional vectors;

identifying embedded utterances in a frame as an additional speaker or as additional background audio.

15. The method according to claim 7, further comprising using the processor to determine the degree of similarity by calculating a distance (δ

) between the corresponding vertices of the speaker models.

16. The method according to claim 7, further comprising displaying the undirected graph on a graphical display connected to the computing system.