Scene understanding using a neurosynaptic system

• US 10,140,551 B2
• Filed: 05/30/2018
• Issued: 11/27/2018
• Est. Priority Date: 05/29/2014
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Andreopoulos, Alexander, Appuswamy, Rathinakumar, Datta, Pallab, Esser, Steven K., Modha, Dharmendra S.
• Primary Examiner(s): Ahmed, Samir
• Application Number:
  US15/993,482
• Publication Number:
  US 20180276502A1
• Time in Patent Office: 181 Days
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06N 3/065 Analogue means
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/451 with interaction between the filter responses, e.g. cortical complex cells
• CPC Class Codes: G06V 10/955 using specific electronic processors
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 30/194 References adjustable by an adaptive method, e.g. learning
• CPC Class Codes: H04N 19/136 Incoming video signal characteristics or properties
• CPC Class Codes: H04N 9/67 for matrixing camera processing pipelines for matrixing of colour signals H04N23/85

Abstract

Embodiments of the invention provide a method for scene understanding based on a sequence of image frames. The method comprises converting each pixel of each image frame to neural spikes, and extracting features from the sequence of image frames by processing neural spikes corresponding to pixels of the sequence of image frames. The method further comprises encoding the extracted features as neural spikes, and classifying the extracted features.

Claims

1. A method comprising:

for an image frame of a sequence of image frames;

converting, utilizing a first neurosynaptic core circuit, at least one pixel of the image frame to one or more neural spikes;

processing, utilizing a second neurosynaptic core circuit, the one or more neural spikes to extract one or more salient features from the image frame;

interpreting the one or more salient features into a human-readable format; and

classifying an object of interest in the image frame based on the human-readable format;

wherein each neurosynaptic core circuit comprises a plurality of electronic synapse devices interconnecting a plurality of electronic neurons to a plurality of electronic axons.

2. The method of claim 1, further comprising:

assigning one or more classification labels to the one or more salient features utilizing a third neurosynaptic core circuit.

3. The method of claim 2, wherein interpreting the one or more salient features into a human-readable format comprises:

for each classification label, determining a corresponding sum of neural spikes generated by the third neurosynaptic core circuit within a specified time window, wherein the corresponding sum represents a number of votes for the classification label.

4. The method of claim 3, wherein classifying an object of interest in the image frame based on the human-readable format comprises:

classifying an object of interest in the image frame with a classification label having a most number of votes across the one or more classification labels.

5. The method of claim 1, wherein the sequence of image frames comprise a video.

6. A system comprising:

at least one processor; and

a non-transitory processor-readable memory device storing instructions that when executed by the at least one processor causes the at least one processor to perform operations including;

for an image frame of a sequence of image frames;

converting, utilizing a first neurosynaptic core circuit, at least one pixel of the image frame to one or more neural spikes;

processing, utilizing a second neurosynaptic core circuit, the one or more neural spikes to extract one or more salient features from the image frame;

interpreting the one or more salient features into a human-readable format; and

classifying an object of interest in the image frame based on the human-readable format;

wherein each neurosynaptic core circuit comprises a plurality of electronic synapse devices interconnecting a plurality of electronic neurons to a plurality of electronic axons.

7. The system of claim 6, wherein the operations further comprise:

assigning one or more classification labels to the one or more salient features utilizing a third neurosynaptic core circuit.

8. The system of claim 7, wherein interpreting the one or more salient features into a human-readable format comprises:

for each classification label, determining a corresponding sum of neural spikes generated by the third neurosynaptic core circuit within a specified time window, wherein the corresponding sum represents a number of votes for the classification label.

9. The system of claim 8, wherein classifying an object of interest in the image frame based on the human-readable format comprises:

classifying an object of interest in the image frame with a classification label having a most number of votes across the one or more classification labels.

10. The system of claim 6, wherein the sequence of image frames comprise a video.

11. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor of a machine, cause the machine to perform operations comprising:

for an image frame of a sequence of image frames;

converting, utilizing a first neurosynaptic core circuit, at least one pixel of the image frame to one or more neural spikes;

processing, utilizing a second neurosynaptic core circuit, the one or more neural spikes to extract one or more salient features from the image frame;

interpreting the one or more salient features into a human-readable format; and

classifying an object of interest in the image frame based on the human-readable format;

wherein each neurosynaptic core circuit comprises a plurality of electronic synapse devices interconnecting a plurality of electronic neurons to a plurality of electronic axons.

12. The computer program product of claim 11, wherein the operations further comprise:

assigning one or more classification labels to the one or more salient features utilizing a third neurosynaptic core circuit.

13. The computer program product of claim 12, wherein interpreting the one or more salient features into a human-readable format comprises:

for each classification label, determining a corresponding sum of neural spikes generated by the third neurosynaptic core circuit within a specified time window, wherein the corresponding sum represents a number of votes for the classification label.

14. The computer program product of claim 13, wherein classifying an object of interest in the image frame based on the human-readable format comprises:

classifying an object of interest in the image frame with a classification label having a most number of votes across the one or more classification labels.

15. The computer program product of claim 11, wherein the sequence of image frames comprise a video.