Feature extraction using a neurosynaptic system for object classification

• US 9,798,972 B2
• Filed: 07/02/2014
• Issued: 10/24/2017
• Est. Priority Date: 07/02/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Appuswamy, Rathinakumar, Esser, Steven K., Modha, Dharmendra S.
• Primary Examiner(s): Vincent, David
• Application Number:
  US14/322,776
• Publication Number:
  US 20160004961A1
• Time in Patent Office: 1,210 Days
• Field of Search: 706 12, 706 45
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/049 Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
• CPC Class Codes: G06N 3/063 using electronic means
• CPC Class Codes: G06N 3/08 Learning methods

Abstract

Embodiments of the invention provide a neurosynaptic system comprising a first set of one or more neurosynaptic core circuits configured to receive input data comprising multiple input regions, and extract a first set of features from the input data. The features of the first set are computed based on different input regions. The system further comprises a second set of one or more neurosynaptic core circuits configured to receive the first set of features, and generate a second set of features by combining the first set of features based on synaptic connectivity information of the second set of core circuits.

Claims

1. A neurosynaptic system comprising:

a first set of hardware neurosynaptic core circuits configured to;

receive input data comprising different input regions, wherein the input data comprises at least one of audio data or visual data; and

extract a first set of features from the input data, wherein features of the first set are computed based on the different input regions; and

a second set of hardware neurosynaptic core circuits configured to;

receive the first set of features; and

generate output data comprising a second set of features, wherein the second set of features are linear combinations computed by combining the first set of features based on synaptic connectivity information of the second set of hardware neurosynaptic core circuits, and each feature of the second set of features represents a characteristic across the different input regions;

wherein the output data is provided to a classifier comprising a third set of hardware neurosynaptic core circuits for object classification, the classifier configured to classify each object of interest within the input data based on the output-data; and

wherein each hardware neurosynaptic core circuit comprises a plurality of electronic neurons, a plurality of electronic axons, and a plurality of electronic synapses for interconnecting the neurons with the axons.

2. The system of claim 1, wherein the input data comprises at least one of a sequence of images, audio, or videos, and the output data generated is used for scene understanding.

3. The system of claim 2, wherein:

for each hardware neurosynaptic core circuit of the first set of hardware neurosynaptic core circuits;

one or more axons of the hardware neurosynaptic core circuit are of an axon type corresponding to a particular receptive filter configured to process a particular input region of the input data.

4. The system of claim 3, wherein each receptive filter processes a particular input region of the input data based on the presence of a particular shape in the particular input region of the input data.

5. The system of claim 3, wherein:

each hardware neurosynaptic core circuit of the first set of hardware neurosynaptic core circuits is configured to extract a first set of features from the input data by;

computing multiple inner products, wherein the inner products are computed by simultaneously processing different input regions of the input data using receptive filters, and wherein the receptive filters are matched to different input regions of the input data; and

computing linear combinations of the inner products computed, wherein each feature of the first set is a combination of two or more inner products computed.

6. The system of claim 2, wherein the synaptic connectivity information includes learned synaptic weights for synapses of the second set of hardware neurosynaptic core circuits.

7. The system of claim 6, further comprising:

a linear classifier for generating a weight matrix, wherein each row of the weight matrix represents a relevance of a particular feature in predicting different classifications;

wherein the linear classifier is trained based on a set of features extracted from the input data using the first set of hardware neurosynaptic core circuits; and

wherein learned synaptic weights for synapses of a hardware neurosynaptic core circuit of the second set of hardware neurosynaptic core circuits are obtained based on the weight matrix.

8. The system of claim 7, wherein:

the synaptic weights for the synapses of the hardware neurosynaptic core circuit are set based on quantized values of a covariance matrix of the weight matrix, wherein the covariance matrix includes multiple entries, and wherein each entry of the covariance matrix represents a covariance between a pair of features.

9. The system of claim 7, wherein one or more correlation coefficients for the features is determined based on the weight matrix.

10. The system of claim 1, wherein:

the input data comprises an image with multiple image sub-patches;

each neuron of the first set of hardware neurosynaptic core circuits has a receptive filter for each image sub-patch; and

the system extracts one or more salient features and one or more discriminative features from the input data in parallel.

11. A computerized method for feature extraction, comprising:

at a first set of hardware neurosynaptic core circuits;

receiving input data comprising different input regions, wherein the input data comprises at least one of audio data or visual data; and

extracting a first set of features from the input data, wherein features of the first set are computed based on the different input regions; and

at a second set of hardware neurosynaptic core circuits;

receiving the first set of features; and

generating output data comprising a second set of features, wherein the second set of features are linear combinations computed by combining the first set of features based on synaptic connectivity information of the second set of hardware neurosynaptic core circuits, and each feature of the second set of features represents a characteristic across the different input regions;

wherein the output data is provided to a classifier comprising a third set of hardware neurosynaptic core circuits for object classification, the classifier configured to classify each object of interest within the input data based on the output-data; and

wherein each hardware neurosynaptic core circuit comprises a plurality of electronic neurons, a plurality of electronic axons, and a plurality of electronic synapses for interconnecting the neurons with the axons.

12. The method of claim 11, wherein extracting a first set of features from the input data, comprises:

computing multiple inner products, wherein the inner products are computed by simultaneously processing different input regions of the input data using receptive filters, and wherein the receptive filters are matched to different input regions of the input data; and

computing linear combinations of the inner products computed, wherein each feature of the first set is a combination of two or more inner products computed.

13. The method of claim 12, wherein each receptive filter processes a particular input region of the input data by looking for a particular shape in the particular input region of the input data.

14. The method of claim 11, wherein the synaptic connectivity information includes learned synaptic weights.

15. The method of claim 14, further comprising:

training a linear classifier to generate a weight matrix, wherein each row of the weight matrix represents a relevance of a particular feature in predicting different classifications, wherein the linear classifier is trained based on a set of features extracted from the input data, and wherein the learned synaptic weights are obtained based on the weight matrix.

16. The method of claim 15, further comprising:

determining a covariance matrix of the weight matrix, wherein the covariance matrix includes multiple entries, and wherein each entry of the covariance matrix represents a covariance between a pair of features;

wherein the learned synaptic weights are quantized values of the covariance matrix.

17. The method of claim 16, further comprising:

based on the weight matrix, determining one or more correlation coefficients for the features.

18. The method of claim 11, wherein:

the input data comprises is image with multiple image sub-patches; and

each image sub-patch has a corresponding receptive filter.

19. The method of claim 11, further comprising:

extracting one or more salient features and one or more discriminative features from the input data in parallel.

20. A non-transitory computer program product for feature extraction, the computer program product comprising a computer-readable storage medium having program code embodied therewith, the program code being executable by a computer to:

at a first set of hardware neurosynaptic core circuits;

receiving input data comprising different input regions, wherein the input data comprises at least one of audio data or visual data; and

extracting a first set of features from the input data, wherein features of the first set are computed based on the different input regions; and

at a second set of hardware neurosynaptic core circuits;

receiving the first set of features; and

generating output data comprising a second set of features, wherein the second set of features are linear combinations computed by combining the first set of features based on synaptic connectivity information of the second set of hardware neurosynaptic core circuits, and each feature of the second set of features represents a characteristic across the different input regions;

wherein the output data is provided to a classifier comprising a third set of hardware neurosynaptic core circuits for object classification, the classifier configured to classify each object of interest within the input data based on the output-data; and

wherein each hardware neurosynaptic core circuit comprises a plurality of electronic neurons, a plurality of electronic axons, and a plurality of electronic synapses for interconnecting the neurons with the axons.