Apparatus and methods for training in convolutional neural networks

• US 10,643,129 B2
• Filed: 10/29/2018
• Issued: 05/05/2020
• Est. Priority Date: 04/29/2016
• Status: Active Grant

• Current Assignee: Cambricon Technologies Corporation Limited
• Original Assignee: Cambricon Technologies Corporation Limited
• Inventors: Chen, Yunji, Zhi, Tian, Liu, Shaoli, Guo, Qi, Chen, Tianshi
• Primary Examiner(s): Sitiriche, Luis A
• Application Number:
  US16/174,165
• Publication Number:
  US 20190065959A1
• Time in Patent Office: 554 Days
• CPC Class Codes: G06F 15/7867 with reconfigurable architecture
• CPC Class Codes: G06F 7/5443 Sum of products for applications thereof, see the relevant places, e.g. G06F17/10, H03H17/00
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/048 Activation functions
• CPC Class Codes: G06N 3/063 using electronic means
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

Aspects for backpropagation of a convolutional neural network are described herein. The aspects may include a direct memory access unit configured to receive input data from a storage device and a master computation module configured to select one or more portions of the input data based on a predetermined convolution window. Further, the aspects may include one or more slave computation modules respectively configured to convolute one of the one or more portions of the input data with one of one or more previously calculated first data gradients to generate a kernel gradient, wherein the master computation module is further configured to update a prestored convolution kernel based on the kernel gradient.

Claims

1. An apparatus for backpropagation of a convolutional neural network, comprising:

a master computation circuit configured to;

receive input data, and

select one or more portions of the input data based on a predetermined convolution window in response to an instruction; and

one or more slave computation circuits respectively configured to convolute one of the one or more portions of the input data with one of one or more calculated first data gradients to generate a kernel gradient,

wherein the master computation circuit is further configured to update a prestored convolution kernel based on the kernel gradient,

wherein the one or more slave computation circuits are respectively configured to multiply at least a portion of the prestored convolution kernel with the one or more first data gradients, and

wherein the master computation circuit is further configured to calculate one or more second data gradients based on a derivative of an activation function and a sum of one or more multiplication results between the first data gradients and the portion of the prestored convolution kernel.

2. The apparatus of claim 1, wherein the activation function is a function selected from the group consisting a sigmoid function, a tanh function, a relu function, and a softmax function.

3. The apparatus of claim 1, further comprising an interconnection circuit configured to sum the one or more kernel gradients respectively generated by the one or more slave computation circuits to generate a summed kernel gradient.

4. The apparatus of claim 3, wherein the master computation circuit is further configured to calculate a quadratic mean of the summed kernel gradient generated by the interconnection circuit.

5. The apparatus of claim 1, further comprising an interconnection circuit configured to channel data between the master computation circuit and the one or more slave computation circuits.

6. The apparatus of claim 5, wherein the interconnection circuit is configured to calculate a sum of one or more multiplication results between the first data gradients and the portion of the prestored convolution kernel.

7. The apparatus of claim 1, wherein each of the one or more slave computation circuits includes a slave neuron caching circuit configured to store one of the one or more portions of the input data.

8. The apparatus of claim 1, each of the one or more slave computation circuits includes a weight value caching circuit configured to store a portion of the prestored convolution kernel that correspond to the slave computation circuit.

9. The apparatus of claim 1, wherein the master computation circuit includes a window controller circuit configured to adjust the predetermined convolution window to select the one or more portions of the input data.

10. The apparatus of claim 1, wherein the instruction is selected from the group consisting of a convolution network sigmoid derivative instruction, a convolution network tanh derivative instruction, a convolution network relu derivative instruction, and a convolution network softmax derivative instruction.

11. The apparatus of claim 10, wherein the convolution network sigmoid derivative instruction includes an address of the first data gradients, a size of the first data gradients, an address of the portion of the convolution kernel, a size of the portion of the convolution kernel, and an indication of a derivative of a sigmoid function.

12. The apparatus of claim 10, wherein the convolution network tanh derivative instruction includes an address of the first data gradients, a size of the first data gradients, an address of the portion of the convolution kernel, a size of the portion of the convolution kernel, and an indication of a derivative of a tanh function.

13. The apparatus of claim 10, wherein the convolution network relu derivative instruction includes an address of the first data gradients, a size of the first data gradients, an address of the portion of the convolution kernel, a size of the portion of the convolution kernel, and an indication of a derivative of a relu function.

14. The apparatus of claim 10, wherein the convolution network softmax derivative instruction includes an address of the first data gradients, a size of the first data gradients, an address of the portion of the convolution kernel, a size of the portion of the convolution kernel, and an indication of a derivative of a softmax function.

15. A method for backpropagation of a convolutional neural network, comprising:

receiving, by a direct memory access circuit, input data from a storage device;

selecting, by a master computation circuit, one or more portions of the input data based on a predetermined convolution window in response to an instruction;

convoluting, by one or more slave computation circuits, one of the one or more portions of the input data with one of one or more previously calculated first data gradients to generate a kernel gradient;

updating, by the master computation circuit, a prestored convolution kernel based on the kernel gradient,

multiplying, by the one or more slave computation circuits, at least a portion of the prestored convolution kernel with the one or more first data gradients;

calculating, by the master computation circuit, one or more second data gradients based on a derivative of an activation function and a sum of one or more multiplication results between the first data gradients and the portion of the prestored convolution kernel.

16. The method of claim 15, further comprising summing, by an interconnection circuit, the one or more kernel gradients respectively generated by the one or more slave computation circuits to generate a summed kernel gradient.

17. The method of claim 16, further comprising calculating, by the master computation circuit, a quadratic mean of the summed kernel gradient respectively generated by the interconnection circuit.

18. The method of claim 15, further comprising channeling data, by an interconnection circuit, between the master computation circuit and the one or more slave computation circuits.

19. The method of claim 18, further comprising calculating, by the interconnection circuit, a sum of one or more multiplication results between the first data gradients and the portion of the prestored convolution kernel.

20. The method of claim 15, further comprising storing, by a slave neuron caching circuit of each of the one or more slave computation circuits, one of the one or more portions of the input data.

21. The method of claim 15, further comprising storing, by a weight value caching circuit of each of the one or more slave computation circuits, a portion of the convolution kernel that correspond to the slave computation circuit.

22. The method of claim 15, further comprising adjusting, by a window controller circuit of the master computation circuit, the predetermined convolution window to select the one or more portions of the input data.

23. The method of claim 15, wherein the instruction is selected from the group consisting of a convolution network sigmoid derivative instruction, a convolution network tanh derivative instruction, a convolution network relu derivative instruction, and a convolution network softmax derivative instruction.

24. The method of claim 23, wherein the convolution network sigmoid derivative instruction includes an address of the first data gradients, a size of the first data gradients, an address of the portion of the convolution kernel, a size of the portion of the convolution kernel, and an indication of a derivative of a sigmoid function.

25. The method of claim 23, wherein the convolution network tanh derivative instruction includes an address of the first data gradients, a size of the first data gradients, an address of the portion of the convolution kernel, a size of the portion of the convolution kernel, and an indication of a derivative of a tanh function.

26. The method of claim 23, wherein the convolution network relu derivative instruction includes an address of the first data gradients, a size of the first data gradients, an address of the portion of the convolution kernel, a size of the portion of the convolution kernel, and an indication of a derivative of a relu function.

27. The method of claim 23, wherein the convolution network softmax derivative instruction includes an address of the first data gradients, a size of the first data gradients, an address of the portion of the convolution kernel, a size of the portion of the convolution kernel, and an indication of a derivative of a softmax function.