Neural network device for evolving appropriate connections

• US 7,080,053 B2
• Filed: 08/16/2001
• Issued: 07/18/2006
• Est. Priority Date: 08/16/2000
• Status: Expired due to Fees

• Current Assignee: The Research Foundation for The State University of New York (State University of New York)
• Original Assignee: The Research Foundation for The State University of New York (State University of New York)
• Inventors: Pinezich, John D., Adams, Paul, Cox, Kingsley J. A.
• Primary Examiner(s): Knight, Anthony
• Assistant Examiner(s): Datskovskiy, Sergey
• Application Number:
  US10/333,667
• Publication Number:
  US 20040128004A1
• Time in Patent Office: 1,797 Days
• Field of Search: 706/25, 706/14, 706/39
• US Class Current: 706/25
• CPC Class Codes: G06N 3/082 modifying the architecture, e.g. adding, deleting or silencing nodes or connections

Abstract

A method for evolving appropriate connections among units in a neural network includes a) calculating weight changes at each existing connection and incipient connections between units for each training example; and b) determining a K ratio using the weight changes, wherein said K ratio comprises the weight change of existing connections, and wherein if the K ratio the weight change of incipient connections exceeds a threshold, further including b1) increasing a weight of the existing connection; b2) creating new connections at the incipient connections. The method further includes c) pruning weak connections between the units.

Claims

1. A computer-implemented method for evolving appropriate connections among units in a neural network comprising the steps of:

a) calculating weight changes at each existing connection and incipient connections between units for each training example;

b) determining a K ratio using the weight changes, wherein said K ratio comprises

$\frac{\text{the <br><br> <br><br>weight <br><br> <br><br>change <br><br> <br><br>of <br><br> <br><br>existing <br><br> <br><br>connections}}{\text{the <br><br> <br><br>weigh <br><br> <br><br>tchange <br><br> <br><br>of <br><br> <br><br>incipient <br><br> <br><br>connections}},$

and wherein if the K ratio exceeds a threshold, further comprising the steps of;

b1) increasing a weight of the existing connection;

b2) creating new connections at the incipient connections; and

c) pruning weak connections between the units.

2. The computer-implemented method of claim 1, wherein the incipient connections comprises connections adjacent to the existing connection which are yet-to-be-formed depending on the K ratio.

3. The computer-implemented method of claim 1, wherein a parameter E comprises approximately:

$\frac{1}{\sqrt{\mathrm{nm}}},$

where n is a number of J cells and m is a number of I cells.

4. The computer-implemented method of claim 1, wherein the K ratio is determined for a center cell of the neural network, wherein upon the creation of the new connections at the incipient connections, feedforward connections to the center cell from an input layer of the neural network and an output layer of the neural network are updated.

5. The computer-implemented method of claim 1, wherein if the K ratio does not exceed the threshold, further comprising returning to step (a).

6. The computer-implemented method of claim 1, wherein the weak connections comprise connections having weights below a specified strength.

7. The computer-implemented method of claim 1, wherein the step of increasing the weight of the existing connection further comprises increasing said weight by (1−

E)×

(total weight change of existing connections).

8. The computer-implemented method of claim 1, wherein the new connections comprise a weight of (E)×

(total weight change of existing connections).

9. A program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform the method steps for evolving appropriate connections among units in a neural network, the method steps comprising the steps of:

a) calculating weight changes at each existing connection and each incipient connection between units for each training example;

b) determining a K ratio using the weight changes, wherein said K ratio comprises

$\frac{\text{the <br><br> <br><br>weight <br><br> <br><br>change <br><br> <br><br>of <br><br> <br><br>existing <br><br> <br><br>connections}}{\text{the <br><br> <br><br>weight <br><br> <br><br>change <br><br> <br><br>of <br><br> <br><br>incipien <br><br> <br><br>tconnections}},$

and the weight change of incipient connections wherein if the K ratio exceeds a threshold, further comprising the steps of;

b1) increasing a weight of the existing connection;

b2) creating new connections at the incipient connections; and

c) pruning weak connections between the units.

10. The program storage device of claim 9, wherein the incipient connection comprises connections adjacent to the existing connection which are yet-to-be-formed depending on the K ratio.

11. The program storage device of claim 9, wherein a parameter E comprises approximately:

$\frac{1}{\sqrt{\mathrm{nm}}},$

where n is a number of J cells and m is a number of I cells.

12. The program storage device of claim 9, wherein if the K ratio does not exceed the threshold, further comprising returning to step (a).

13. The program storage device of claim 9, wherein the weak connections comprise connections having weights below a specified strength.

14. The program storage device of claim 9, wherein the step of increasing the weight of the existing connection further comprises increasing said weight by (1−

E)×

(total weight change of existing connections).

15. The program storage device of claim 9, wherein the new connections comprise a weight of (E)×

(total weight change of existing connections).

16. The program storage device of claim 9, wherein the K ratio is determined for a center cell of the neural network, wherein upon the creation of the new connections at the incipient connections, feedforward connections to the center cell from an input layer of the neural network and an output layer of the neural network are updated.