Method for pseudo-recurrent processing of data using a feedforward neural network architecture

• US 10,152,673 B2
• Filed: 06/21/2013
• Issued: 12/11/2018
• Est. Priority Date: 06/21/2013
• Status: Active Grant

• Current Assignee: Aselsan Elektronik Sanayi Ve Ticaret Anonim Sirketi
• Original Assignee: Aselsan Elektronik Sanayi Ve Ticaret Anonim Sirketi
• Inventors: Yilmaz, Ozgur, Ozkan, Huseyin
• Primary Examiner(s): Chace, Christian
• Assistant Examiner(s): Ray, Negel G.
• Application Number:
  US14/900,177
• Publication Number:
  US 20160140434A1
• Time in Patent Office: 1,999 Days
• Field of Search: None
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06F 18/23213 with fixed number of clusters, e.g. K-means clustering
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/063 using electronic means

Abstract

Recurrent neural networks are powerful tools for handling incomplete data problems in machine learning thanks to their significant generative capabilities. However, the computational demand for algorithms to work in real time applications requires specialized hardware and software solutions. We disclose a method for adding recurrent processing capabilities into a feedforward network without sacrificing much from computational efficiency. We assume a mixture model and generate samples of the last hidden layer according to the class decisions of the output layer, modify the hidden layer activity using the samples, and propagate to lower layers. For an incomplete data problem, the iterative procedure emulates feedforward-feedback loop, filling-in the missing hidden layer activity with meaningful representations.

Claims

1. A computer implemented method for recurrent data processing, comprising the steps of:

computing activity of multiple layers of hidden layer nodes in a feed forward neural network, given an input data instance,

forming memories of hidden layer activities, utilizing clustering and filtering methods, as a training phase in a recurrent processing,

finding memories that are closest to a presented test data instance according to a class decision of the feedforward network, and imputing the test data hidden layer activity with computed closest memories in an iterative fashion,

wherein the step of forming memories of hidden layer activities, utilizing clustering and filtering methods, as a training phase in a recurrent processing further comprises the substeps of;

computing hidden layer activities of every training data instance, then low-pass filtering and stacking the hidden layer activities in a data structure;

keeping a first and second hidden layer activity memory, indexed by class label;

forming both class specific and class independent cluster centers as quantized memories of the training data'"'"'s second hidden layer activity, via k-means clustering, using each class data separately or using all the data together depending on a choice of class specificity;

keeping quantized second hidden layer memories, indexed by class labels or non-indexed, depending on the class specificity choice;

training a cascade of classifiers for enabling multiple hypotheses generation of a network, via utilizing a subset of the input data as the training data; and

keeping a classifier memory, indexed with the set of data used during training;

wherein the step of finding memories that are closest to the presented test data instance according to the class decision of the feed forward network, and imputing the test data hidden layer activity with computed closest memories in an iterative fashion further comprises the substeps of;

determining first, second and third class label choices of the neural network as multiple hypotheses, via a cascaded procedure utilizing a sequence of classifier decisions;

computing a set of candidate samples for the second layer, that are closest Euclidian distance hidden layer memories to the test data'"'"'s second hidden layer activity, using the multiple hypotheses class decisions of the network and a corresponding memory database then assigning the second hidden layer sample as one of the candidate hidden layer memories, via max or averaging operations depending on a choice of multi-hypotheses competition;

merging the second hidden layer sample with the test data'"'"'s second hidden layer activity via weighted averaging operation, creating an updated second hidden layer activity;

using the updated second hidden layer activity to compute the closest Euclidian distance first hidden layer memory, and assigning as the first hidden layer sample, merging the first hidden layer sample with the test data first hidden layer activity via weighted averaging operation, creating an updated first hidden layer activity;

computing the feedforward second hidden layer activity from updated first hidden layer activity, and merging this feed-forward second hidden layer activity with updated second hidden layer activity, via weighted averaging operation; and

repeating these steps for multiple iterations starting from the step of determining the first, second and third class label choices of the neural network as multiple hypotheses, via a cascaded procedure utilizing a sequence of classifier decisions, and using the output of step of computing the feedforward second hidden layer activity from updated first hidden layer activity, and merging this feed forward second hidden layer activity with updated second hidden layer activity, via weighted averaging operation in the beginning of the next iteration.

2. A computer implemented method according to claim 1 for enabling a feedforward network to mimic a recurrent neural network via making a class decision at the output layer of feedforward neural network, and selecting an appropriate memory to estimate hidden layer activities, then inserting the selected memory to the hidden layer activity as if the selected memory is a feedback from a higher layer network in classical recurrent networks.