LOW RANK MATRIX COMPRESSION

• US 20180293758A1
• Filed: 04/08/2017
• Published: 10/11/2018
• Est. Priority Date: 04/08/2017
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Bar-On, Tomer, Subag, Jacob, Fais, Yaniv, Dreyfuss, Jeremie, Novik, Gal, Leibovich, Gal, Schwartz, Tomer, Cohen, Ehud, Faivishevsky, Lev, Sarel, Uzi, Armon, Amitai, Shadmiy, Yahav
• Granted Patent: US 11,037,330 B2
• Time in Patent Office: Days
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/048 Activation functions
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 3/088 Non-supervised learning, e.g. competitive learning
• CPC Class Codes: G06T 9/002 using neural networks
• CPC Class Codes: H04N 19/42 characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation H04N19/635 takes precedence
• CPC Class Codes: H04N 19/436 using parallelised computational arrangements

Abstract

In an example, an apparatus comprises logic, at least partially including hardware logic, to implement a lossy compression algorithm which utilizes a data transform and quantization process to compress data in a convolutional neural network (CNN) layer. Other embodiments are also disclosed and claimed.

Claims

1. An apparatus comprising a processor to:

detect one or more independent rows of a matrix comprising weights of a neural network;

determine a scalar associated with each of the one or more independent rows of the matrix;

encode a plurality of the one or more independent rows with the scalar associated with the row to generate encoded weight data;

apply delta compression to compress the encoded weight data; and

store the encoded weight data in a memory.

2. The apparatus of claim 1, the processor to:

compress at least a portion of the encoded weight data in a frequency domain.

3. The apparatus of claim 2, the processor to:

quantize the at least a portion of the encoded weight data in the frequency domain.

4. The apparatus of claim 2, the processor to:

compress the at least a portion of the encoded weight data via K-means compression.

5. The apparatus of claim 2, the processor to:

apply an inversed transform to the neural network layer.

6. (canceled)

7. (canceled)

8. (canceled)

9. (canceled)

10. (canceled)

11. A method, comprising:

detecting one or more independent rows of a matrix comprising weights of a neural network;

determining a scalar associated with each of the one or more independent rows of the matrix;

encoding a plurality of the one or more independent rows with the scalar associated with the row to generate encoded weight data;

implementing a delta compression algorithm to compress the encoded weight data; and

storing the encoded weight data in a memory.

12. The method of claim 11, further comprising:

compressing at least a portion of the encoded weight data in a frequency domain.

13. The method of claim 12, further comprising:

quantizing the at least a portion of the encoded weight data in the frequency domain.

14. The method of claim 12, further comprising:

compress the at least a portion of the encoded weight data via K-means compression.

15. The method of claim 12, further comprising:

applying an inversed transform to the neural network layer.

16. An electronic device comprising:

a computer readable memory; and

a processor communicatively coupled to the computer readable memory to;

detect one or more independent rows of a matrix comprising weights of a neural network;

determine a scalar associated with each of the one or more independent rows of the matrix;

encode a plurality of the one or more independent rows with the scalar associated with the row to generate encoded weight data;

implement a delta compression algorithm to compress the encoded weight data; and

store the encoded weight data in the computer readable memory.

17. The electronic device of claim 16, the processor to:

compress at least a portion of the encoded weight data in a frequency domain.

18. The electronic device of claim 17, the processor to:

quantize the at least a portion of the encoded weight data in the frequency domain.

19. The electronic device of claim 17, the processor to:

compress the at least a portion of the encoded weight data.

20. The electronic device of claim 17, the processor to:

apply an inversed transform to the neural network layer.