Convolutional neural network

• US 9,665,799 B1
• Filed: 01/29/2016
• Issued: 05/30/2017
• Est. Priority Date: 01/29/2016
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia, Inc.)
• Original Assignee: FotoNation Limited (Adeia, Inc.)
• Inventors: Munteanu, Mihai Constantine, Zaharia, Corneliu, Caliman, Alexandru
• Primary Examiner(s): Werner, Brian P
• Application Number:
  US15/010,418
• Time in Patent Office: 487 Days
• CPC Class Codes: G06F 18/24137 Distances to cluster centroïds
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 10/955 using specific electronic processors

Abstract

A convolutional neural network (CNN) for an image processing system comprises an image cache responsive to a request to read a block of N×M pixels extending from a specified location within an input map to provide a block of N×M pixels at an output port. A convolution engine reads blocks of pixels from the output port, combines blocks of pixels with a corresponding set of weights to provide a product, and subjects the product to an activation function to provide an output pixel value. The image cache comprises a plurality of interleaved memories capable of simultaneously providing the N×M pixels at the output port in a single clock cycle. A controller provides a set of weights to the convolution engine before processing an input map, causes the convolution engine to scan across the input map by incrementing a specified location for successive blocks of pixels and generates an output map within the image cache by writing output pixel values to successive locations within the image cache.

Claims

1. A convolutional neural network (CNN) for an image processing system comprising:

an image cache comprising an input port and an output port, said image cache being responsive to a request to read a block of N×

M pixels extending from a specified location within an input map to provide said block of N×

M pixels at said output port;

a convolution engine being arranged to read at least one block of N×

M pixels from said image cache output port, to combine said at least one block of N×

M pixels with a corresponding set of weights to provide a product, and to subject said product to so an activation function to provide an output pixel value;

said image cache being configured to write output pixel values to a specified write address via said image cache input port;

said image cache comprising a plurality of interleaved memories, each memory storing a block of pixel values at a given memory address, the image cache being arranged to determine for a block of N×

M pixels to be read from said image cache;

a respective one address within each of said interleaved memories in which said pixels of said block of N×

M pixels are stored;

a respective memory of said plurality of interleaved memories within which each pixel of said block of N×

M pixels is stored; and

a respective offset for each pixel of said block of N×

M pixels within each memory address, so that said image cache can simultaneously provide said N×

M pixels at said output port in a single clock cycle; and

a controller arranged to provide a set of weights to said convolution engine before processing at least one input map, to cause said convolution engine to process said at least one input map by specifying locations for successive blocks of N×

M pixels and to generate an output map within said image cache by writing said output pixel values to successive locations within said image cache.

2. A CNN according to claim 1 further comprising a sub-sampler arranged to selectively sub-sample output pixel values from said convolution engine and to provide output pixel values at a data output interface, said image cache being configured to write output pixel values from said sub-sampler data output interface to a specified write address via said image cache input port.

3. A CNN according to claim 1 further comprising a read controller arranged to acquire image information across a system bus, the read controller being arranged to provide a data output interface to provide at least one pixel to be written to a specified write address within said image cache.

4. A CNN according to claim 1 further comprising a write controller arranged to provide image cache data across a system bus, the write controller being arranged to provide a data input interface to receive at least one pixel to be read from a specified address within said image cache.

5. A CNN according to claim 1 comprising A interleaved memories and wherein at each memory address there can be stored A×

A adjacent pixels for a map.

6. A CNN according to claim 5 wherein A=4.

7. A CNN according to claim 5 wherein where N≦

A+1 and M≦

A+1, said controller is arranged to freely access a block of N×

M pixels at any location within a map.

8. A CNN according to claim 6 wherein where M=N=6, said controller is arranged to access blocks of N×

M pixels at even numbered locations within a map.

9. A CNN according to claim 6 wherein where 6<

M≦

8 and 6<

N≦

M, said controller is arranged to access blocks of N×

M pixels at only every fourth location within a map.

10. A CNN according to claim 1 wherein said controller is arranged to concatenate pixels from a plurality of output maps into a single vector of 1×

W cells, and wherein said convolution engine is arranged to combine vector cells with a corresponding set of weights to provide a product, and to subject said product to an activation function to provide an output vector cell value.

11. A CNN according to claim 1 wherein said controller is arranged to write output maps to said image cache at different memory locations than input maps used to generate said output maps.

12. A CNN according to claim 1 wherein said controller is arranged to overwrite input maps within said image cache with output maps from successive convolution layers.

13. A CNN according to claim 1 wherein said image cache comprises an cache control block and for each block of N×

M pixels to be accessed within said image cache, where M≦

8 and N≦

8, said cache control block is arranged to calculate an 8×

8 (MS) matrix where each cell indicates a respective memory of said plurality of interleaved memories within which each pixel of said block of N×

M pixels is stored;

to calculate an 8×

8 (BS) matrix where each cell indicates a respective offset for each pixel of said block of N×

M pixels within each memory address;

to combined said MS and BS matrices and to mask said combination according to the size of said block of N×

M pixels to generate a respective byte enable (BE) signal for each interleaved memory of said image cache.

14. A CNN according to claim 13 wherein said MS matrix is calculated using a combination of sums and compares.

15. A CNN according to claim 13 wherein said BS matrix is calculated using a combination of shifts and a one-hot function.

16. A CNN according to claim 1 further comprising a weights cache and a read controller arranged to read weights for each layer of convolution processing across a system bus into said weights cache.

17. A CNN according to claim 16 wherein said weights cache comprises a plurality of interleaved memories, each memory storing a block of weights at a given memory address, the weights cache being arranged to determine for a block of N×

M weights to be read from said weights cache;

a respective one address within each of said interleaved memories in which said weights of said block of N×

M weights are stored;

a respective memory of said plurality of interleaved memories within which each weight of said block of N×

M weights is stored; and

a respective offset for each weight of said block of N×

M weight within each memory address, so that said weights cache can simultaneously provide said N×

M weights at an output port in a single clock cycle.

18. A CNN according to claim 1 wherein a width of said image cache in pixels is a power of 2.

19. A CNN according to claim 1 further comprising a read controller for said image cache, said read controller being arranged to simultaneously provide a block of N×

M pixel values to said image cache input port, the image cache being arranged to determine for said block of N×

M pixels to be written to said image cache;

a respective one address within each of said interleaved memories in which said pixels of said block of N×

M pixels are to be stored;

a respective memory of said plurality of interleaved memories within which each pixel of said block of N×

M pixels is to be stored; and

a respective offset for each pixel of said block of N×

M pixels within each memory address, so that said image cache can simultaneously write said N×

M pixels at said input port to said interleaved memories.

20. A CNN according to claim 19 wherein said block of pixels provided by said read controller comprises 1 row of pixels.

21. An image processing system comprising:

an image cache comprising an input port and an output port, said image cache being responsive to a request to read a block of N×

M pixels extending from a specified location within an input map to provide said block of N×

M pixels at said output port;

an image processor being arranged to read at least one block of N×

M pixels from said image cache output port, and to process said at least one block of N×

M pixels to provide at least one output pixel value;

said image cache being configured to write output pixel values to a specified write address via said image cache input port;

said image cache comprising a plurality of interleaved memories, each memory storing a block of pixel values at a given memory address, the image cache being arranged to determine for a block of N×

M pixels to be read from said image cache;

a respective one address within each of said interleaved memories in which said pixels of said block of N×

M pixels are stored;

a respective memory of said plurality of interleaved memories within which each pixel of said block of N×

M pixels is stored; and

a respective offset for each pixel of said block of N×

M pixels within each memory address, so that said image cache can simultaneously provide said N×

M pixels at said output port in a single clock cycle; and

a controller arranged to cause said image processor to read from a specified location of said at least one input map a configurable block of N×

M pixels and to write output pixel values to specified locations within said image cache.

22. An image processing system according to claim 21 comprising A interleaved memories and wherein at each memory address there can be stored A×

A adjacent pixels for a map.

23. An image processing system according to claim 22 wherein A=4.

24. An image processing system according to claim 22 wherein where N≦

A+1 and M≦

A+1, said controller is arranged to freely access a block of N×

M pixels at any location within a map.

25. An image processing system according to claim 23 wherein where M=N=6, said controller is arranged to access blocks of N×

M pixels at even numbered locations within a map.

26. An image processing system according to claim 23 wherein where 6<

M≦

8 and 6<

N≦

8, said controller is arranged to access blocks of N×

M pixels at only every fourth location within a map.