Neural device and method of constructing the device

• US 5,649,067 A
• Filed: 06/05/1995
• Issued: 07/15/1997
• Est. Priority Date: 12/16/1992
• Status: Expired due to Fees

• Current Assignee: US Philips Corporation (Koninklijke Philips N.V.)
• Original Assignee: US Philips Corporation (Koninklijke Philips N.V.)
• Inventors: Gentric, Philippe, Minot, Joel
• Primary Examiner(s): Hafiz, Tariq R.
• Application Number:
  US08/464,923
• Time in Patent Office: 771 Days
• Field of Search: 382/155-159, 395/20-25, 395/27
• US Class Current: 706/25
• CPC Class Codes: G06F 18/24137 Distances to cluster centroïds
• CPC Class Codes: G06N 3/082 modifying the architecture, e.g. adding, deleting or silencing nodes or connections
• CPC Class Codes: G06V 30/194 References adjustable by an adaptive method, e.g. learning

Abstract

On the basis of a device without hidden neurons, arbitrary samples are taken from a set of learning samples so as to be presented as objects to be classified. Each time if the response is not correct, a hidden neuron (H_i) is introduced with a connection to the output neuron (O_j) of the class of the sample, whereas if the response is correct, no neuron whatsoever is added. During this introduction phase for hidden neurons, the neurons are subdivided into groups by searching for each of the introduced neurons whether it falls within an existing group, in which case it is incorporated therein; otherwise a new group is created around this neuron. The association with a group is defined as a function of the distance from the "creator" neuron. This device could be applied in character recognition systems as one example.

Claims

1. A neural device produced according to a method of constructing a neural device for classification of objects, said neural device being trained using a set of learning samples of objects having known classes, each object to be classified being defined by an input vector which is represented by a point in hyperspace, said neural device comprising:

a layer of input neurons, each of which corresponds to one of the dimensions of the hyperspace;

a layer of hidden neurons having inputs, the inputs being connected exclusively to the input neurons, and activation of each hidden neuron being based on coordinates of a respective reference point of the hyperspace; and

a layer of output neurons, each of which corresponds to a respective class of the objects;

said method comprising;

a) starting from a neural device without a hidden neuron;

b) choosing an arbitrary sample from said set of learning samples;

c) subsequently placing a first hidden neuron in the device, while defining the respective reference point associated with said first neuron as the point in hyperspace representing the sample;

d) establishing an excitatory connection of positive weight between said first neuron and the output neuron corresponding to a class of the sample;

e) taking a new sample from said set of learning samples;

f) applying the new sample to the device for classification;

g) if, as a result of step f), a response of the device to the new sample does not give a correct classification, introducing into the device a new hidden neuron, corresponding to the new sample, by;

I) defining the respective reference point associated with the new hidden neuron as being the point representing the new sample and;

II) establishing an excitatory connection of positive weight between the new hidden neuron and the output neuron corresponding to the class of the new sample;

h) if the response does give the correct classification, skipping step g);

i) treating all remaining samples according to steps f), g) and h) until no samples remain in said set of learning samples; and

j) defining groups of neurons, with a neuron that is representative of each group, as the new hidden neurons are introduced, according to the following process;

I) determining, for each respective new hidden neurons of the new hidden neurons, whether the respective new hidden neuron forms part of a previously defined group;

II) in response to a positive result from the determining step, incorporating the respective hidden neuron into the group of which the respective new hidden neuron forms a part; and

III) in response to a negative result from the determining step, forming a new group for which the respective new hidden neurons constitutes a representative neuron, the first neuron thus being defined as being representative of a first group.

2. The device according to claim 1, wherein a neuron is considered to form part of a given group if the distance between a reference point associated with the neuron and the reference point associated with the neuron which is representative of the given group, is smaller than a given distance.

3. The device of claim 2, wherein:

the groups are considered to be of a first level;

at least one second given distance is defined which is greater than the first distance; and

at least second groups of a second level are defined based on the second given distance during the process.

4. The device of claim 1 wherein when two hidden neurons corresponding to two learning samples have representative points whose coordinates neighbor one another, without said samples belonging to the same class, an inhibitory synapse of negative weight is inserted between the hidden neuron corresponding to one of the samples and the output neuron corresponding to the class of the other sample.

5. The device of claim 1, wherein a supplementary hidden neuron which corresponds to a learning sample is inserted with a connection to the corresponding output neuron if for this sample, presented as an object to be classified, an output neuron is effectively activated for the desired class, but the difference between the activity of the output neuron with the highest activation level and that of the output neuron with the highest activation level but one is below a given threshold.

6. The device of claim 1 wherein:

the objects fall into J distinct classes;

the input vectors have K coordinates, represented by points in a hyperspace of K dimensions;

the layer of input neurons has K input neurons, each of which corresponds to one of said dimensions;

the layer of output neurons comprises J output neurons, each of which corresponds to one of said distinct classes; and

activation of the hidden neurons is calculated as a function of coordinates in the hyperspace of the respective reference point associated with each hidden neuron, the function decreasing with the geometrical distance in the hyperspace between the respective reference point associated with the hidden neuron and the point representing the input vector;

said device further comprises;

means for, during a first stage, operating only the hidden neurons which are representative of the groups, and determining which groups have a highest activity; and

means for, during a second stage, operating the hidden neurons of one or more groups which had the highest activity during the first stage.

7. The device of claim 6 wherein a classification result is indicated by one of the output neurons having a highest activity, so long as a difference, between the output neuron having the highest activity and a second output neuron having a second highest activity, exceeds a threshold value.

8. The device of claim 6 wherein activation of the output neurons is a function of the activation of one or several input neurons, wherein said function presents an input threshold.

9. The device of claim 6 wherein activation of a hidden neuron having an index i is calculated according to the formula ##EQU6## where w_ki represents the K coordinates of the sample having triggered introduction of the hidden neuron having index i,

σ

_i is an adjustable coefficient associated with the hidden neuron having index i,

x_k represents the K coordinates of the input vector to be classified;

n is an integer; and

function f is a function which increases as its argument tends towards zero.

10. The device of claim 9 wherein n=2.

11. The device of claim 3 wherein

the objects fall into J distinct classes,

the input vectors have K coordinates, represented by points in a hyperspace of K dimensions,

the layer of input neurons has K input neurons, each of which corresponds to one of said dimensions,

the layer of output neurons comprises J output neurons, each of which corresponds to one of said distinct classes,

activation of the hidden neurons is calculated as a function of coordinates in the hyperspace of the respective reference point associated with each hidden neuron, function decreasing with the geometrical distance in the hyperspace between the respective reference point associated with the hidden neuron and the point representing the input vector,

the device further comprises means for

during a first stage, operating only the hidden neurons which are representative of the groups of a highest level, and determining which groups of the highest level have a highest activity;

and

during a second stage, operating one or more groups within the one or more groups having the highest activity level the first stage,

during subsequent stages, selecting lower level groups according to activity levels observed in previous stages until the first level is reached.

12. The device of claim 3 wherein:

the objects fall into J distinct classes;

the input vectors have K coordinates, represented by points in a hyperspace of K dimensions;

the layer of input neurons has K input neurons, each of which corresponds to one of said dimensions;

the layer of output neurons comprises J output neurons, each of which corresponds to one of said distinct classes;

activation of the hidden neurons is calculated as a function of coordinates in the hyperspace of the respective reference point associated with each hidden neuron, function decreasing with the geometrical distance in the hyperspace between the respective reference point associated with the hidden neuron and the point representing the input vector; and

the device further comprises;

means for, during a first stage, operating only the hidden neurons which are representative of the groups of a highest level, and determining which groups of the highest level have a highest activity;

means for, during a second stage, operating one or more groups within the one or more groups having the highest activity level the first stage, and

means for, during subsequent stages, selecting lower level groups according to activity levels observed in previous stages until the first level is reached.

13. The device of claim 2 wherein activation of a hidden neuron having an index i is calculated according to the formula ##EQU7## where:

w_ki represents the K coordinates of the sample having triggered introduction of the hidden neuron having index i;

σ

_i is an adjustable coefficient associated with the hidden neuron having index i;

x_k represents the K coordinates of the input vector to be classified;

n is an integer; and

function f is a function which increases as its argument tends towards zero.