Scene understanding using a neurosynaptic system

• US 9,373,058 B2
• Filed: 05/29/2014
• Issued: 06/21/2016
• Est. Priority Date: 05/29/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Andreopoulos, Alexander, Appuswamy, Rathinakumar, Datta, Pallab, Esser, Steven K., Modha, Dharmendra S.
• Primary Examiner(s): AHMED, SAMIR ANWAR
• Application Number:
  US14/289,890
• Publication Number:
  US 20150347870A1
• Time in Patent Office: 754 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06N 3/065 Analogue means
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/451 with interaction between the filter responses, e.g. cortical complex cells
• CPC Class Codes: G06V 10/955 using specific electronic processors
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 30/194 References adjustable by an adaptive method, e.g. learning
• CPC Class Codes: H04N 19/136 Incoming video signal characteristics or properties
• CPC Class Codes: H04N 9/67 for matrixing camera processing pipelines for matrixing of colour signals H04N23/85

Abstract

Embodiments of the invention provide a method for scene understanding based on a sequence of image frames. The method comprises converting each pixel of each image frame to neural spikes, and extracting features from the sequence of image frames by processing neural spikes corresponding to pixels of the sequence of image frames. The method further comprises encoding the extracted features as neural spikes, and classifying the extracted features.

Claims

1. A method for scene understanding based on a sequence of image frames, comprising:

converting, utilizing a first set of neurosynaptic core circuits, each pixel of each image frame to one or more corresponding neural spikes;

extracting, utilizing a second set of neurosynaptic core circuits, one or more features from said sequence of image frames by processing neural spikes corresponding to pixels of said sequence of image frames;

encoding, utilizing a third set of neurosynaptic core circuits, said one or more extracted features as one or more neural spikes; and

classifying, utilizing said third set of neurosynaptic core circuits, said one or more extracted features;

wherein each neurosynaptic core circuit comprises a plurality of electronic neurons, a plurality of electronic axons, and a synaptic crossbar array including multiple electronic synapse devices interconnecting the plurality of electronic neurons to the plurality of electronic axons.

2. The method of claim 1, wherein converting each pixel of each image frame to one or more corresponding neural spikes comprises converting each pixel of each image frame to one or more corresponding neural spikes based on one of the following:

a temporal coding scheme, or a spatial coding scheme.

3. The method of claim 2, wherein:

said temporal coding scheme is rate coding; and

in rate coding;

each pixel of each image frame has a corresponding spike count based on a corresponding intensity value for said pixel, wherein said corresponding spike count represents the number of neural spikes delivered to one or more electronic axons of the plurality of electronic axons within a pre-determined time window; and

a corresponding intensity value for each pixel of each image frame is mapped to a corresponding spike count for said pixel based on one of a linear mapping or a non-linear mapping.

4. The method of claim 2, wherein:

said temporal coding scheme is stochastic coding; and

in stochastic coding;

each pixel of each image frame has a corresponding spiking probability based on a corresponding intensity value for said pixel, wherein said corresponding spiking probability represents the probability that a neural spike is delivered to one or more electronic axons of the plurality of electronic axons in each time step; and

a corresponding intensity value for each pixel of each image frame is mapped to a corresponding spiking probability for said pixel based on one of a linear mapping or a non-linear mapping.

5. The method of claim 2, wherein:

said spatial coding scheme is single line coding; and

in single line coding, neural spikes for each pixel of each image frame are delivered to an electronic axon of the plurality of electronic axons.

6. The method of claim 2, wherein:

said spatial coding scheme is population coding; and

in population coding, neural spikes for each pixel of each image frame is delivered to multiple electronic axons of the plurality of electronic axons, wherein each electronic axon of the multiple electronic axons is configured to encode values having one of a particular sign and a particular amplitude.

7. The method of claim 1, wherein:

extracting one or more features from said sequence of image frames by processing neural spikes corresponding to pixels of said sequence of image frames comprises extracting one or more features from said sequence of image frames using a neural network including one or more neurons;

each neuron of said neural network receives input via one or more weighted incoming lines;

for each neural spike delivered to each electronic axon, a neuron of said neural network receives said neural spike via a weighted incoming line, and said neuron integrates a weight corresponding to said weighted incoming line as input;

each neuron generates one or more outgoing neural spikes based on an activation function and input integrated by said neuron;

each input received by each neuron represents input from any of the following;

an external source, one or more neurons of said neural network;

said neural network includes a set of neurons for output, wherein outgoing neural spikes generated by said set of neurons encode said one or more extracted features, and wherein each neuron of said set generates an outgoing neural spike when input integrated by said neuron exceeds a pre-determined threshold.

8. The method of claim 7, wherein:

said neural network further includes a synaptic crossbar array for interconnecting said one or more neurons;

said array includes one or more axons, one or more dendrite, and one or more weighted synaptic connections interconnecting said one or more axons with said one or more dendrites;

one or more neural spikes are delivered to at least one electronic axon of the plurality of electronic axons; and

each dendrite of said array is a weighted incoming line that delivers one or more weighted neural spikes to a particular neuron of said neural network.

9. The method of claim 8, wherein:

said one or more weighted synaptic connections are binary; and

each neuron assigns a weight to an incoming neural spike based on an axon that said incoming neural spike was delivered to.

10. The method of claim 1, wherein:

said one of more extracted features comprise at least one of the following;

one or more mathematically defined features, and one or more learned features;

said one or more mathematically defined feature include operators for extracting at least one of the following;

edges, textures, colors, or shapes; and

said one or more learned features include at least one feature learned from training data using one or more of the following;

a K-means clustering algorithm applied to the training data, and an input/desired output covariance algorithm applied to the training data.

11. The method of claim 1, wherein encoding said one or more extracted features as one or more neural spikes comprises encoding said one or more extracted features as one or more neural spikes based on one of the following:

a temporal coding scheme, or both a temporal coding scheme and a spatial coding scheme.

12. The method of claim 11, wherein:

said temporal coding scheme is rate coding; and

in rate coding;

each extracted feature has a corresponding spike count based on a corresponding level for said extracted feature, wherein said corresponding spike count represents the number of outgoing neural spikes generated by one or more associated neurons within a pre-determined time window; and

a corresponding level for each extracted feature is mapped to a corresponding spike count for said extracted feature based on one of a linear mapping or a non-linear mapping.

13. The method of claim 11, wherein:

said temporal coding scheme is stochastic coding; and

in stochastic coding;

each extracted feature has a corresponding spiking probability based on a corresponding level for said extracted feature, wherein said corresponding spiking probability represents the probability that an outgoing neural spike is generated by one or more associated neurons in each time step; and

a corresponding level for each extracted feature is mapped to a corresponding spiking probability for said pixel based on one of a linear mapping or a non-linear mapping.

14. The method of claim 11, wherein:

said spatial coding scheme is single line coding; and

in single line coding, an extracted feature is encoded using a single neuron.

15. The method of claim 11, wherein:

said spatial coding scheme is population coding; and

in population coding, an extracted feature is encoded using multiple neurons, wherein each neuron of said multiple neurons is configured to encode values having one of a particular sign and a particular amplitude.

16. The method of claim 1, wherein:

classifying said one or more extracted features comprises assigning one or more classification labels to said one or more extracted features using a neural network including one or more neurons;

each neuron of said neural network receives input via one or more weighted electronic axons;

for each neural spike delivered to each electronic axon, a neuron of said neural network receives said neural spike via a weighted electronic axon, and said neuron integrates a weight corresponding to said weighted electronic axon as input;

each neuron generates one or more outgoing neural spikes based on an activation function and input integrated by said neuron;

each input received by each neuron represents input from one of the following;

an external source, and at least other neuron of said neural network; and

said one or more neurons include a set of neurons for output, wherein outgoing neural spikes generated by said set of neurons encode said one or more classification labels.

17. The method of claim 16, further comprising

creating weights for classification using one of the following:

supervised learning, and reinforcement learning.

18. The method of claim 16, comprising:

translating said one or more classification labels into a human-readable format;

wherein, for each classification label, the classifier generates one or more outgoing neural spikes, wherein said one or more outgoing neural spikes are delivered to one or more output lines in a pre-determined time window, wherein a sum of the number of outgoing neural spikes delivered to said one or more output lines in said pre-determined time window represents the number of votes for said classification label; and

wherein a classification label having the most number of votes represents a predicted class for said one or more extracted features.

19. The method of claim 1, wherein said sequence of image frames constitute a video.

20. A system for scene understanding based on a sequence of image frames, comprising:

at least one processor; and

a non-transitory processor-readable memory device storing instructions that when executed by the at least one processor causes the at least one processor to perform operations including;

converting, utilizing a first set of neurosynaptic core circuits, each pixel of each image frame to one or more corresponding neural spikes;

extracting, utilizing a second set of neurosynaptic core circuits, one or more features from said sequence of image frames by processing neural spikes corresponding to pixels of said sequence of image frames;

encoding, utilizing a third set of neurosvnaptic core circuits, said one or more extracted features as one or more neural spikes; and

classifying, utilizing said third set of neurosynaptic core circuits, said one or more extracted features;

wherein each neurosynaptic core circuit comprises a plurality of electronic neurons, a plurality of electronic axons, and a synaptic crossbar array including multiple electronic synapse devices interconnecting the plurality of electronic neurons to the plurality of electronic axons.