Scene understanding using a neurosynaptic system

• US 10,043,110 B2
• Filed: 10/19/2016
• Issued: 08/07/2018
• Est. Priority Date: 05/29/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Andreopoulos, Alexander, Appuswamy, Rathinakumar, Datta, Pallab, Esser, Steven K., Modha, Dharmendra S.
• Primary Examiner(s): AHMED, SAMIR ANWAR
• Application Number:
  US15/298,137
• Publication Number:
  US 20170039429A1
• Time in Patent Office: 657 Days
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06N 3/065 Analogue means
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/451 with interaction between the filter responses, e.g. cortical complex cells
• CPC Class Codes: G06V 10/955 using specific electronic processors
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 30/194 References adjustable by an adaptive method, e.g. learning
• CPC Class Codes: H04N 19/136 Incoming video signal characteristics or properties
• CPC Class Codes: H04N 9/67 for matrixing camera processing pipelines for matrixing of colour signals H04N23/85

Abstract

Embodiments of the invention provide a method for scene understanding based on a sequence of image frames. The method comprises converting each pixel of each image frame to neural spikes, and extracting features from the sequence of image frames by processing neural spikes corresponding to pixels of the sequence of image frames. The method further comprises encoding the extracted features as neural spikes, and classifying the extracted features.

Claims

1. A method comprising:

receiving, utilizing a first neurosynaptic core circuit, an input video comprising a plurality of image frames in a first color space;

converting, utilizing a second neurosynaptic core circuit, the input video to a second color space by converting each image frame of the input video to one or more channels, wherein each channel corresponds to a dimension of the second color space, and the second color space is different from the first color space; and

converting, utilizing a third neurosynaptic core circuit, at least one pixel of at least one image frame of the one or more channels to one or more neural spikes based on a pre-determined coding scheme;

wherein each neurosynaptic core circuit comprises a plurality of electronic synapse devices interconnecting a plurality of electronic neurons to a plurality of electronic axons.

2. The method of claim 1, wherein the one or more neural spikes are processed to extract one or more salient features from the at least one image frame.

3. The method of claim 1, wherein the converting, utilizing the third neurosynaptic core circuit, the at least one pixel of the at least one image frame of the one or more channels to the one or more neural spikes based on the pre-determined coding scheme comprises:

mapping a pixel intensity value of the at least one pixel to a number of neural spikes, wherein the mapping is one of a linear mapping or a non-linear mapping, and the number of neural spikes are delivered to at least one electronic axon of the plurality of axons within a pre-determined time window.

4. The method of claim 1, wherein the converting, utilizing the third neurosynaptic core circuit, the at least one pixel of the at least one image frame of the one or more channels to the one or more neural spikes based on the pre-determined coding scheme comprises:

mapping a pixel intensity value of the at least one pixel to a probability value, wherein the mapping is one of a linear mapping or a non-linear mapping, and the probability value indicates a probability a neural spike is delivered to at least one electronic axon of the plurality of axons in each time step.

5. The method of claim 1, wherein the first color space comprises a red green blue (RGB) color space, and the second color space is a Lab color space.

6. A system comprising:

at least one processor; and

a non-transitory processor-readable memory device storing instructions that when executed by the at least one processor causes the at least one processor to perform operations including;

receiving, utilizing a first neurosynaptic core circuit, an input video comprising a plurality of image frames in a first color space;

converting, utilizing a second neurosynaptic core circuit, the input video to a second color space by converting each image frame of the input video to one or more channels, wherein each channel corresponds to a dimension of the second color space, and the second color space is different from the first color space; and

converting, utilizing a third neurosynaptic core circuit, at least one pixel of at least one image frame of the one or more channels to one or more neural spikes based on a pre-determined coding scheme;

wherein each neurosynaptic core circuit comprises a plurality of electronic synapse devices interconnecting a plurality of electronic neurons to a plurality of electronic axons.

7. The system of claim 6, wherein the one or more neural spikes are processed to extract one or more salient features from the at least one image frame.

8. The system of claim 6, wherein the converting, utilizing the third neurosynaptic core circuit, the at least one pixel of the at least one image frame of the one or more channels to the one or more neural spikes based on the pre-determined coding scheme comprises:

mapping a pixel intensity value of the at least one pixel to a number of neural spikes, wherein the mapping is one of a linear mapping or a non-linear mapping, and the number of neural spikes are delivered to at least one electronic axon of the plurality of axons within a pre-determined time window.

9. The system of claim 6, wherein the converting, utilizing the third neurosynaptic core circuit, the at least one pixel of the at least one image frame of the one or more channels to the one or more neural spikes based on the pre-determined coding scheme comprises:

mapping a pixel intensity value of the at least one pixel to a probability value, wherein the mapping is one of a linear mapping or a non-linear mapping, and the probability value indicates a probability a neural spike is delivered to at least one electronic axon of the plurality of axons in each time step.

10. The system of claim 6, wherein the first color space comprises a red green blue (RGB) color space, and the second color space is a Lab color space.

11. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor of a machine, cause the machine to perform operations comprising:

receiving, utilizing a first neurosynaptic core circuit, an input video comprising a plurality of image frames in a first color space;

converting, utilizing a second neurosynaptic core circuit, the input video to a second color space by converting each image frame of the input video to one or more channels, wherein each channel corresponds to a dimension of the second color space, and the second color space is different from the first color space; and

converting, utilizing a third neurosynaptic core circuit, at least one pixel of at least one image frame of the one or more channels to one or more neural spikes based on a pre-determined coding scheme;

wherein each neurosynaptic core circuit comprises a plurality of electronic synapse devices interconnecting a plurality of electronic neurons to a plurality of electronic axons.

12. The computer-readable medium of claim 11, wherein the one or more neural spikes are processed to extract one or more salient features from the at least one image frame.

13. The computer-readable medium of claim 11, wherein the converting, utilizing the third neurosynaptic core circuit, the at least one pixel of the at least one image frame of the one or more channels to the one or more neural spikes based on the pre-determined coding scheme comprises:

mapping a pixel intensity value of the at least one pixel to a number of neural spikes, wherein the mapping is one of a linear mapping or a non-linear mapping, and the number of neural spikes are delivered to at least one electronic axon of the plurality of axons within a pre-determined time window.

14. The computer-readable medium of claim 11, wherein the converting, utilizing the third neurosynaptic core circuit, the at least one pixel of the at least one image frame of the one or more channels to the one or more neural spikes based on the pre-determined coding scheme comprises:

mapping a pixel intensity value of the at least one pixel to a probability value, wherein the mapping is one of a linear mapping or a non-linear mapping, and the probability value indicates a probability a neural spike is delivered to at least one electronic axon of the plurality of axons in each time step.

15. The computer-readable medium of claim 11, wherein the first color space comprises a red green blue (RGB) color space, and the second color space is a Lab color space.