Channel-compensated low-level features for speaker recognition

• US 10,347,256 B2
• Filed: 09/19/2017
• Issued: 07/09/2019
• Est. Priority Date: 09/19/2016
• Status: Active Grant

• Current Assignee: Pindrop Security, Inc.
• Original Assignee: Pindrop Security, Inc.
• Inventors: Khoury, Elie, Garland, Matthew
• Primary Examiner(s): Azad, Abul K
• Application Number:
  US15/709,024
• Publication Number:
  US 20180082692A1
• Time in Patent Office: 658 Days
• CPC Class Codes: G10L 17/02 Preprocessing operations, e.g. segment selection; Pattern representation or modelling, e.g. based on linear discriminant analysis [LDA] or principal components; Feature selection or extraction
• CPC Class Codes: G10L 17/04 Training, enrolment or model building
• CPC Class Codes: G10L 17/18 Artificial neural networks; Connectionist approaches
• CPC Class Codes: G10L 17/20 Pattern transformations or operations aimed at increasing system robustness, e.g. against channel noise or different working conditions
• CPC Class Codes: G10L 19/028 Noise substitution, i.e. substituting non-tonal spectral components by noisy source comfort noise for discontinuous speech transmission G10L19/012

Abstract

A system for generating channel-compensated features of a speech signal includes a channel noise simulator that degrades the speech signal, a feed forward convolutional neural network (CNN) that generates channel-compensated features of the degraded speech signal, and a loss function that computes a difference between the channel-compensated features and handcrafted features for the same raw speech signal. Each loss result may be used to update connection weights of the CNN until a predetermined threshold loss is satisfied, and the CNN may be used as a front-end for a deep neural network (DNN) for speaker recognition/verification. The DNN may include convolutional layers, a bottleneck features layer, multiple fully-connected layers and an output layer. The bottleneck features may be used to update connection weights of the convolutional layers, and dropout may be applied to the convolutional layers.

Claims

1. A system for generating channel-compensated low level features for speaker recognition, the system comprising:

an acoustic channel simulator configured to receive a recognition speech signal, degrade the recognition speech signal to include characteristics of an audio channel, and output a degraded speech;

a first feed forward convolutional neural network configured, in a training mode, to receive the degraded speech signal, and to derive from the degraded speech signal a plurality of channel-compensated low-level features, and further configured, in a test and enrollment mode, to receive the recognition speech signal and to calculate from the recognition speech signal a plurality of the channel-compensated low-level features;

a speech signal analyzer configured, in the training mode, to extract features of the recognition speech signal;

a loss function processor configured to calculate a loss based on the features from the speech analyzer and the channel-compensated low-level features from the first feed forward convolutional neural network;

wherein, the calculated loss at each of a plurality of training iterations is lowered by modifying one or more connection weights of the first feed forward convolutional neural network, and

if the calculated loss is less than or equal to the threshold loss, or a maximum number of training iterations has been met, the training mode is terminated.

2. The system according to claim 1, wherein the acoustic channel simulator comprises one or more of an environmental noise simulator, a reverberation simulator, an audio acquisition device characteristic simulator, and a transcoding noise simulator, each selectably or programmatically configurable to perform a portion of said degradation of the recognition speech signal.

3. The system according to claim 2, wherein the environmental noise simulator introduces to the recognition speech signal at least one environmental noise type selected from a plurality of environmental noise types.

4. The system according to claim 3, wherein the environmental noise simulator introduces the at least one selected environmental noise type at a signal-to-noise ratio (SNR) selected from a plurality of signal-to-noise ratios (SNRs).

5. The system according to claim 2, wherein the reverberation simulator is configured to simulate reverberation according to a direct-to-reverberation ratio (DRR) selected from a plurality of DRRs.

6. The system according to claim 5, wherein each DRR in the plurality of DRRs has a corresponding time-to-reverberation at 60 dB.

7. The system according to claim 2, wherein the audio acquisition device characteristic simulator simulates audio characteristics of an audio acquisition device selectable from a plurality of stored audio acquisition device profiles each having one or more selectable audio characteristics.

8. The system according to claim 7, wherein each audio acquisition device profile includes at least one of:

a frequency/equalization characteristic, an amplitude characteristic, a filtering characteristic, an electrical noise characteristic, and a physical noise characteristic.

9. The system according to claim 2, wherein the transcoding noise simulator is configured to selectively add audio channel transcoding characteristics selectable from a plurality of stored transcoding characteristic profiles.

10. The system according to claim 9, wherein each transcoding characteristic profile includes at least one of a quantization error noise characteristic, a sampling rate audio artifact characteristic, and a data compression audio artifact characteristic.

11. The system according to claim 1, wherein the features from the speech signal analyzer and the channel-compensated features from the first feed forward convolutional neural network each include a corresponding at least one of Mel-frequency cepstrum coefficients (MFCC), low-frequency cepstrum coefficients (LFCC), and perceptual linear prediction (PLP) coefficients.

12. The system according to claim 1, further comprising a second, speaker-aware, convolutional neural network configured, in the test and enrollment mode, to receive the plurality of channel-compensated features, and to extract from the channel-compensated features a plurality of speaker-aware bottleneck features.

13. The system according to claim 12, wherein the second convolutional neural network includes a plurality of convolutional layers and a bottleneck layer, the bottleneck layer configured to output the speaker-aware bottleneck features.

14. The system according to claim 13, wherein the second convolutional neural network further includes a plurality of fully connected layers, an output layer, and a second loss function processor each used during training of the second convolutional neural network.

15. The system according to claim 14, wherein at least one of the fully connected layers employs a dropout technique to avoid overfitting.

16. The system according to claim 15, wherein a dropout ratio of the dropout technique is about 30%.

17. The system according to claim 14, wherein the second convolutional neural network further includes a max pooling layer configured to pool over a time axis.

18. The system according to claim 12, wherein the second convolutional neural network takes as input at least one set of other features side by side with the channel-compensated features, the at least one set of other features extracted from the speech signal.

19. A method of training a deep neural network (DNN) with channel-compensated low-level features for speaker recognition, the method comprising:

receiving a recognition speech signal;

degrading the recognition speech signal to produce a channel-compensated speech signal;

extracting, using a first feed forward convolutional neural network, a plurality of low-level features from the channel-compensated speech signal;

calculating a loss result using the channel-compensated low-level features extracted from the channel-compensated speech signal and hand-crafted features extracted from the recognition speech signal;

modifying connection weights of the first feed forward convolutional neural network to lower the calculated loss result at each of a plurality of training iterations; and

terminating the training if the calculated loss result is less than or equal to a threshold loss or a maximum number of training iterations has been met.

20. The method according to claim 19, wherein said degrading the speech signal includes selectively or programmatically adding one or more of environmental noise, reverberation, an audio acquisition device characteristic, and an audio channel transcoding artifact.

21. The method according to claim 20, wherein said adding environmental noise includes adding to the speech signal at least one environmental noise type selected from a plurality of environmental noise types.

22. The method according to claim 21, wherein the selected environmental noise type is added at a signal-to-noise ratio (SNR) selected from a plurality of signal-to-noise ratios (SNRs).

23. The method according to claim 20, wherein said adding reverberation utilizes a direct-to-reverberation ratio (DRR) selected from a plurality of DRRs.

24. The method according to claim 23, wherein each DRR in the plurality of DRRs has a corresponding time-to-reverberation at 60 dB.

25. The method according to claim 20, wherein said acquisition device characteristic simulator simulates characteristics of an audio acquisition device selectable from a plurality of stored audio acquisition device profiles each having one or more selectable audio characteristics.

26. The method according to claim 25, wherein each audio acquisition device profile includes at least one of a frequency/equalization characteristic, an amplitude characteristic, a filtering characteristic, and electrical noise characteristic, and a physical noise characteristic.

27. The method according to claim 20, wherein said adding of transcoding characteristics includes selectively adding audio channel transcoding characteristics selectable from a plurality of stored transcoding characteristic profiles.

28. The method according to claim 27, wherein each transcoding characteristic profile includes at least one of a quantization error characteristic, a sampling rate audio artifact characteristic, and a data compression audio artifact characteristic.