FIXED POINT NEURAL NETWORK BASED ON FLOATING POINT NEURAL NETWORK QUANTIZATION

• US 20160328646A1
• Filed: 10/22/2015
• Published: 11/10/2016
• Est. Priority Date: 05/08/2015
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: MAJUMDAR, Somdeb, LIN, Dexu, ANNAPUREDDY, Venkata Sreekanta Reddy, HOWARD, David Edward, JULIAN, David Jonathan, BELL, William Richard II
• Granted Patent: US 10,373,050 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/048 Activation functions
• CPC Class Codes: G06N 3/06 Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
• CPC Class Codes: G06N 3/063 using electronic means
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 3/10 Interfaces, programming languages or software development kits, e.g. for simulating neural networks
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/82 using neural networks

Abstract

A method of quantizing a floating point machine learning network to obtain a fixed point machine learning network using a quantizer may include selecting at least one moment of an input distribution of the floating point machine learning network. The method may also include determining quantizer parameters for quantizing values of the floating point machine learning network based at least in part on the at least one selected moment of the input distribution of the floating point machine learning network to obtain corresponding values of the fixed point machine learning network.

Claims

1. A method of quantizing a floating point machine learning network to obtain a fixed point machine learning network using a quantizer, comprising:

determining quantizer parameters for quantizing values of the floating point machine learning network based at least in part on at least one moment of an input distribution of the floating point machine learning network to obtain corresponding values of the fixed point machine learning network.

2. The method of claim 1, in which the at least one moment of the input distribution of the floating point machine learning network comprises a mean of the input distribution.

3. The method of claim 2, further comprising:

shifting activation values to create a zero-mean distribution for each layer of the fixed point machine learning network; and

shifting a coordinate of any non-linear function applied to a shifted activation value by an amount corresponding to the shifted activation value.

4. The method of claim 1, in which the at least one moment of the input distribution of the floating point machine learning network comprises a variance of the input distribution.

5. The method of claim 1, in which the quantizer parameters for quantizing values of the floating point machine learning network comprise a maximum dynamic encoding range of the fixed point machine learning network.

6. The method of claim 1, in which the quantizer parameters for quantizing values of the floating point machine learning network comprise a quantizer step size.

7. The method of claim 6, further comprising determining the quantizer step size of the quantizer for bias, weight, and/or activation values based at least in part on a mean value and standard deviation of the bias, the weight, and/or the activation values from the floating point machine learning network.

8. The method of claim 1, in which the quantizer parameters for quantizing values of the floating point machine learning network comprise a bit width.

9. The method of claim 1, in which the quantizer parameters for quantizing values of the floating point machine learning network comprise a signed or unsigned indicator value.

10. The method of claim 1, in which the corresponding values of the fixed point machine learning network comprise a bias, a weight, and/or activation values.

11. A method of converting a first machine learning network to a second machine learning network, comprising:

incorporating a mean value of a distribution of activation values of the first machine learning network into a network bias of the second machine learning network.

12. The method of claim 11, in which activation values for each layer of the second machine learning network have an approximately zero mean.

13. The method of claim 11, further comprising determining the network bias for internal layers differently from determining the network bias for an output layer of the second machine learning network.

14. The method of claim 11, further comprising modifying a non-linear activation function when the mean value is incorporated into the network bias.

15. The method of claim 11, in which the first machine learning network comprises a floating point neural network and the second machine learning network comprises a fixed point neural network.

16. An apparatus for quantizing a floating point machine learning network to obtain a fixed point machine learning network using a quantizer, comprising:

means for selecting at least one moment of an input distribution of the floating point machine learning network; and

means for determining quantizer parameters for quantizing values of the floating point machine learning network based at least in part on the at least one selected moment of the input distribution of the floating point machine learning network to obtain corresponding values of the fixed point machine learning network.

17. The apparatus of claim 16, in which the at least one moment of the input distribution of the floating point machine learning network comprises a mean of the input distribution or a variance of the input distribution.

18. The apparatus of claim 16, in which the quantizer parameters for quantizing values of the floating point machine learning network comprise a maximum dynamic encoding range of the fixed point machine learning network, a bit width, a signed or unsigned indicator value and/or a quantizer step size.

19. The apparatus of claim 18, further comprising means for determining the quantizer step size of the quantizer for bias, weight, and/or activation values based at least in part on a mean value and standard deviation of the bias, the weight, and/or the activation values from the floating point machine learning network.

20. The apparatus of claim 16, in which the corresponding values of the fixed point machine learning network comprise a bias, a weight, and/or activation values.

21. An apparatus for quantizing a floating point machine learning network to obtain a fixed point machine learning network using a quantizer, comprising:

a memory unit; and

at least one processor coupled to the memory unit, the at least one processor configured;

to select at least one moment of an input distribution of the floating point machine learning network; and

to determine quantizer parameters for quantizing values of the floating point machine learning network based at least in part on the at least one selected moment of the input distribution of the floating point machine learning network to obtain corresponding values of the fixed point machine learning network.

22. The apparatus of claim 21, in which the at least one moment of the input distribution of the floating point machine learning network comprises a mean of the input distribution or a variance of the input distribution.

23. The apparatus of claim 21, in which the quantizer parameters for quantizing values of the floating point machine learning network comprise a maximum dynamic encoding range of the fixed point machine learning network, a bit width, a signed or unsigned indicator value and/or a quantizer step size.

24. The apparatus of claim 23, in which the at least one processor is further configured to determine the quantizer step size of the quantizer for bias, weight, and/or activation values based at least in part on a mean value and standard deviation of the bias, the weight, and/or the activation values from the floating point machine learning network.

25. The apparatus of claim 21, in which the corresponding values of the fixed point machine learning network comprise a bias, a weight, and/or activation values.

26. A non-transitory computer-readable medium having program code recorded thereon for quantizing a floating point machine learning network to obtain a fixed point machine learning network using a quantizer, the program code being executed by a processor and comprising:

program code to select at least one moment of an input distribution of the floating point machine learning network; and

program code to determine quantizer parameters for quantizing values of the floating point machine learning network based at least in part on the at least one selected moment of the input distribution of the floating point machine learning network to obtain corresponding values of the fixed point machine learning network.

27. The computer-readable medium of claim 26, in which the at least one moment of the input distribution of the floating point machine learning network comprises a mean of the input distribution or a variance of the input distribution.

28. The computer-readable medium of claim 26, in which the quantizer parameters for quantizing values of the floating point machine learning network comprise a maximum dynamic encoding range of the fixed point machine learning network, a bit width, a signed or unsigned indicator value and/or a quantizer step size.

29. The computer-readable medium of claim 28, further comprising program code to determine the quantizer step size of the quantizer for bias, weight, and/or activation values based at least in part on a mean value and standard deviation of the bias, the weight, and/or the activation values from the floating point machine learning network.

30. The computer-readable medium of claim 26, in which the corresponding values of the fixed point machine learning network comprise a bias, a weight, and/or activation values.