APPARATUS AND METHODS FOR TRAINING IN CONVOLUTIONAL NEURAL NETWORKS

• US 20190065959A1
• Filed: 10/29/2018
• Published: 02/28/2019
• Est. Priority Date: 04/29/2016
• Status: Active Grant

• Current Assignee: Cambricon Technologies Corporation Limited
• Original Assignee: Cambricon Technologies Corporation Limited
• Inventors: Chen, Yunji, Zhi, Tian, Liu, Shaoli, Guo, Qi, Chen, Tianshi
• Granted Patent: US 10,643,129 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 15/7867 with reconfigurable architecture
• CPC Class Codes: G06F 7/5443 Sum of products for applications thereof, see the relevant places, e.g. G06F17/10, H03H17/00
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/048 Activation functions
• CPC Class Codes: G06N 3/063 using electronic means
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

Aspects for backpropagation of a convolutional neural network are described herein. The aspects may include a direct memory access unit configured to receive input data from a storage device and a master computation module configured to select one or more portions of the input data based on a predetermined convolution window. Further, the aspects may include one or more slave computation modules respectively configured to convolute one of the one or more portions of the input data with one of one or more previously calculated first data gradients to generate a kernel gradient, wherein the master computation module is further configured to update a prestored convolution kernel based on the kernel gradient.

Claims

1. An apparatus for backpropagation of a convolutional neural network, comprising:

a master computation module configured to;

receive input data, and

select one or more portions of the input data based on a predetermined convolution window in response to an instruction; and

one or more slave computation modules respectively configured to convolute one of the one or more portions of the input data with one of one or more calculated first data gradients to generate a kernel gradient, wherein the master computation module is further configured to update a prestored convolution kernel based on the kernel gradient.

2. The apparatus of claim 1, wherein the one or more slave computation modules are respectively configured to multiply at least a portion of the prestored convolution kernel with the one or more first data gradients.

3. The apparatus of claim 2, wherein the master computation module is further configured to calculate one or more second data gradients based on a derivative of an activation function and a sum of one or more multiplication results between the first data gradients and the portion of the prestored convolution kernel, wherein the activation function is a function selected from the group consisting a sigmoid function, a tanh function, a relu function, and a softmax function.

4. The apparatus of claim 1, further comprising an interconnection unit configured to sum the one or more kernel gradients respectively generated by the one or more slave computation modules to generate a summed kernel gradient.

5. The apparatus of claim 4, wherein the master computation module is further configured to calculate a quadratic mean of the summed kernel gradient generated by the interconnection unit.

6. The apparatus of claim 1, further comprising an interconnection unit configured to channel data between the master computation module and the one or more slave computation modules.

7. The apparatus of claim 6, wherein the interconnection unit is configured to a sum of one or more multiplication results between the first data gradients and the portion of the prestored convolution kernel.

8. The apparatus of claim 1, wherein each of the one or more slave computation modules includes a slave neuron caching unit configured to store one of the one or more portions of the input data.

9. The apparatus of claim 1, each of the one or more slave computation modules includes a weight value caching unit configured to store a portion of the prestored convolution kernel that correspond to the slave computation module.

10. The apparatus of claim 1, wherein the master computation module includes a window controller configured to adjust the predetermined convolution window to select the one or more portions of the input data.

11. The apparatus of claim 1, wherein the instruction is selected from the group consisting of a convolution network sigmoid derivative instruction, a convolution network tanh derivative instruction, a convolution network relu derivative instruction, and a convolution network softmax derivative instruction.

12. The apparatus of claim 11, wherein the convolution network sigmoid derivative instruction includes an address of the first data gradients, a size of the first data gradients, an address of the portion of the convolution kernel, a size of the portion of the convolution kernel, and an indication of a derivative of a sigmoid function.

13. The apparatus of claim 11, wherein the convolution network tanh derivative instruction includes an address of the first data gradients, a size of the first data gradients, an address of the portion of the convolution kernel, a size of the portion of the convolution kernel, and an indication of a derivative of a tanh function.

14. The apparatus of claim 11, wherein the convolution network relu derivative instruction includes an address of the first data gradients, a size of the first data gradients, an address of the portion of the convolution kernel, a size of the portion of the convolution kernel, and an indication of a derivative of a relu function.

15. The apparatus of claim 11, wherein the convolution network softmax derivative instruction includes an address of the first data gradients, a size of the first data gradients, an address of the portion of the convolution kernel, a size of the portion of the convolution kernel, and an indication of a derivative of a softmax function.

16. A method for backpropagation of a convolutional neural network, comprising:

receiving, by a direct memory access unit, input data from a storage device;

selecting, by a master computation module, one or more portions of the input data based on a predetermined convolution window in response to an instruction;

convoluting, by one or more slave computation modules, one of the one or more portions of the input data with one of one or more previously calculated first data gradients to generate a kernel gradient; and

updating, by the master computation module, a prestored convolution kernel based on the kernel gradient.

17. The method of claim 16, further comprising multiplying, by the one or more slave computation modules, at least a portion of the prestored convolution kernel with the one or more first data gradients.

18. The method of claim 17, further comprising calculating, by the master computation module, one or more second data gradients based on a derivative of an activation function and a sum of one or more multiplication results between the first data gradients and the portion of the prestored convolution kernel.

19. The method of claim 16, further comprising summing, by an interconnection unit, the one or more kernel gradients respectively generated by the one or more slave computation modules to generate a summed kernel gradient.

20. The method of claim 19, further comprising calculating, by the master computation module, a quadratic mean of the summed kernel gradient respectively generated by the interconnection unit.

21. The method of claim 16, further comprising channeling data, by an interconnection unit, between the master computation module and the one or more slave computation modules.

22. The method of claim 21, further comprising calculating, by the interconnection unit, a sum of one or more multiplication results between the first data gradients and the portion of the prestored convolution kernel.

23. The method of claim 16, further comprising storing, by a slave neuron caching unit of each of the one or more slave computation modules, one of the one or more portions of the input data.

24. The method of claim 16, further comprising storing, by a weight value caching unit of each of the one or more slave computation modules, a portion of the convolution kernel that correspond to the slave computation module.

25. The method of claim 16, further comprising adjusting, by a window controller of the master computation module, the predetermined convolution window to select the one or more portions of the input data.

26. The method of claim 16, wherein the instruction is selected from the group consisting of a convolution network sigmoid derivative instruction, a convolution network tanh derivative instruction, a convolution network relu derivative instruction, and a convolution network softmax derivative instruction.

27. The method of claim 26, wherein the convolution network sigmoid derivative instruction includes an address of the first data gradients, a size of the first data gradients, an address of the portion of the convolution kernel, a size of the portion of the convolution kernel, and an indication of a derivative of a sigmoid function.

28. The method of claim 26, wherein the convolution network tanh derivative instruction includes an address of the first data gradients, a size of the first data gradients, an address of the portion of the convolution kernel, a size of the portion of the convolution kernel, and an indication of a derivative of a tanh function.

29. The method of claim 26, wherein the convolution network relu derivative instruction includes an address of the first data gradients, a size of the first data gradients, an address of the portion of the convolution kernel, a size of the portion of the convolution kernel, and an indication of a derivative of a relu function.

30. The method of claim 26, wherein the convolution network softmax derivative instruction includes an address of the first data gradients, a size of the first data gradients, an address of the portion of the convolution kernel, a size of the portion of the convolution kernel, and an indication of a derivative of a softmax function.