Low-footprint adaptation and personalization for a deep neural network

• US 9,324,321 B2
• Filed: 03/07/2014
• Issued: 04/26/2016
• Est. Priority Date: 03/07/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Xue, Jian, Li, Jinyu, Yu, Dong, Seltzer, Michael L., Gong, Yifan
• Primary Examiner(s): Baker, Charlotte M
• Application Number:
  US14/201,704
• Publication Number:
  US 20150255061A1
• Time in Patent Office: 781 Days
• Field of Search: 704/232
• US Class Current: 1/1
• CPC Class Codes: G06N 3/082 modifying the architecture, e.g. adding, deleting or silencing nodes or connections
• CPC Class Codes: G10L 15/075 supervised, i.e. under machine guidance
• CPC Class Codes: G10L 15/16 using artificial neural networks

Abstract

The adaptation and personalization of a deep neural network (DNN) model for automatic speech recognition is provided. An utterance which includes speech features for one or more speakers may be received in ASR tasks such as voice search or short message dictation. A decomposition approach may then be applied to an original matrix in the DNN model. In response to applying the decomposition approach, the original matrix may be converted into multiple new matrices which are smaller than the original matrix. A square matrix may then be added to the new matrices. Speaker-specific parameters may then be stored in the square matrix. The DNN model may then be adapted by updating the square matrix. This process may be applied to all of a number of original matrices in the DNN model. The adapted DNN model may include a reduced number of parameters than those received in the original DNN model.

Claims

1. A method of adapting and personalizing a deep neural network (DNN) model for automatic speech recognition (ASR), comprising:

receiving, by a computing device, at least one utterance comprising a plurality of speech features for one or more speakers from one or more ASR tasks;

applying, by the computing device, a decomposition process to two or more matrices in the DNN model;

in response to applying the decomposition process, adapting the DNN model to include a decomposed matrix that is generated from decomposition processing of the two or more matrices; and

exposing the adapted DNN model as a service.

2. The method of claim 1, wherein the adapting of the DNN model further comprises replacing the two or more matrices in a layer of the DNN model with the decomposed matrix.

3. The method of claim 1, wherein the adapting of the DNN model further comprises adding a new layer into the DNN model, wherein the new layer comprises the decomposed matrix a non-linear layer.

4. The method of claim 1, wherein applying, by the computing device, the decomposition process further comprises applying singular value decomposition (SVD) to the two or more matrices in the DNN model.

5. The method of claim 1, wherein the decomposition process further comprises applying affine transformations to inputs and outputs of the DNN model.

6. The method of claim 1, wherein the exposing further comprises at least one selected from a group consisting of:

providing the adapted DNN model as a web service and incorporating the adapted DNN model in an application that is distributed to a client device.

7. A system for adapting and personalizing a deep neural network (DNN) model for automatic speech recognition (ASR), comprising:

a memory for storing executable program code; and

a processor, functionally coupled to the memory, the processor being responsive to computer-executable instructions contained in the program code and operative to;

receive at least one utterance comprising a plurality of speech features for one or more speakers from one or more ASR tasks;

determine an adapted DNN model from the DNN model, the DNN model comprising a plurality of unadapted matrices and the adapted DNN model comprising a plurality of adapted matrices;

calculate a difference between the plurality adapted matrices and the plurality of unadapted matrices to determine a plurality of delta matrices;

apply a decomposition process to the plurality of delta matrices;

convert the plurality of delta matrices into a subset of one or more small matrices; and

store the subset of the one or more small matrices within an adapted DNN model as a speaker dependent feature set.

8. The system of claim 7, wherein the processor, in applying the decomposition process to the plurality of delta matrices, is operative to apply singular value decomposition (SVD) to each of the plurality of delta matrices.

9. The system of claim 7, wherein the processor, in converting each of the plurality of delta matrices into a subset of the one or more small matrices, is operative to convert the product of two low-rank matrices.

10. The system of claim 7, wherein the processor, in storing the subset of the one or more small matrices, the subset of the one or more small matrices comprising a small percentage of a plurality of parameters in the DNN model, is operative to only store the subset of the one or more small matrices for each of one or more speakers.

11. The system of claim 7, further comprising at least one selected from a group consisting of:

providing the adapted DNN model as a web service and incorporating the adapted DNN model in an application that is distributed to a client device.

12. The system of claim 7, further comprising receiving a spoken utterance, and analyzing the spoken utterance using the adapted DNN model.

13. A computer-readable storage medium storing computer executable instructions which, when executed by a computer, will cause computer to perform a method of adapting and personalizing a deep neural network (DNN) model for automatic speech recognition (ASR), the method comprising:

receiving a plurality of utterances, each of the plurality of utterances comprising a plurality of speech features for a plurality of speakers from one or more ASR tasks;

applying a decomposition process to two or more matrices in the DNN model; and

in response to applying the decomposition process, adapting the DNN model to include a decomposed matrix that is generated from decomposition processing of the two or more matrices.

14. The computer-readable storage medium of claim 13, wherein the adapting of the DNN model further comprises updating a layer of the DNN model to include the decomposed matrix.

15. The computer-readable storage medium of claim 14, wherein the adapting of the DNN model further comprises adding at least one new layer into the DNN model, wherein the new layer comprises the decomposed matrix a non-linear layer.

16. The computer-readable storage medium of claim 15, wherein the at least one of new layer is a linear layer, the linear layer comprising a linear function.

17. The computer-readable storage medium of claim 13, wherein applying the decomposition process further comprises applying singular value decomposition (SVD) to the two or more matrices in the DNN model.

18. The computer-readable storage medium of claim 13, wherein the applying of the decomposition process further comprises applying affine transformations to inputs and outputs of the DNN model.

19. The computer-readable storage medium of claim 13, wherein the method further comprising:

using the adapted DNN model as a web service.

20. The computer-readable storage medium of claim 13, wherein the method further comprising:

incorporating the adapted DNN model as component in an application service.