Self-organizing data driven learning hardware with local interconnections

• US 7,293,002 B2
• Filed: 06/18/2002
• Issued: 11/06/2007
• Est. Priority Date: 06/19/2001
• Status: Expired due to Fees

• Current Assignee: Ohio University
• Original Assignee: Ohio University
• Inventors: Starzyk, Janusz A.
• Primary Examiner(s): Vincent; David
• Assistant Examiner(s): FERNANDEZ RIVAS, OMAR F
• Application Number:
  US10/174,038
• Publication Number:
  US 20030055799A1
• Time in Patent Office: 1,967 Days
• Field of Search: 706/15, 706/16, 706/19, 706/20, 706/25, 706/26, 706/27, 706/31, 706/32, 706/39
• US Class Current: 706/15
• CPC Class Codes: G06N 3/088 Non-supervised learning, e.g. competitive learning

Abstract

A method for organizing processors to perform artificial neural network tasks is provided. The method provides a computer executable methodology for organizing processors in a self-organizing, data driven, learning hardware with local interconnections. A training data is processed substantially in parallel by the locally interconnected processors. The local processors determine local interconnections between the processors based on the training data. The local processors then determine, substantially in parallel, transformation functions and/or entropy based thresholds for the processors based on the training data.

Claims

1. A self-organizing apparatus, comprising:

a plurality of artificial neurons arranged in multiple layers;

a plurality of initial interconnections interconnecting the plurality of artificial neurons to form an initial configuration of a self-organizing learning array in which less than total interconnection is undertaken between adjacent layers;

one or more learning array input nodes, each learning array input node connected to at least one artificial neuron; and

the plurality of artificial neurons including a first self-organizing artificial neuron comprising;

an entropy-based evaluator determining entropy of an input data space associated with the first self-organizing artificial neuron, wherein at least one of i) selection of initial interconnections to retain as neuron input connections to the first self-organizing artificial neuron and ii) selection of a transformation function to be applied by the first self-organizing artificial neuron is based at least in part on the determined entropy;

wherein the plurality of artificial neurons includes one or more self-organizing artificial neurons each self-organizing artificial neuron independently determines the initial interconnections associated therewith to retain as neuron input connections and the initial interconnections associated therewith to release in response to training data being applied to the one or more learning array input nodes, the retained interconnections forming a trained configuration of the self-organizing learning array;

wherein the first self-organizing artificial neuron determines the entropy of the input data space in response to training data being applied to the one or more learning array input nodes.

2. The apparatus of claim 1 wherein the number of retained interconnections is less than the number of initial interconnections in quantity.

3. The apparatus of claim 1 wherein determining which initial interconnections are retained at the corresponding self-organizing artificial neuron is based at least in part on showing favoritism to interconnections from neighboring artificial neurons.

4. The apparatus of claim 1 wherein determining which initial interconnections are retained is based at least in part on reducing entropy in an input data space associated with the corresponding self-organizing artificial neuron.

5. The apparatus of claim 1, the plurality of artificial neurons including first self-organizing artificial neuron comprising:

an input data multiplexer selectively connecting and disconnecting one or more initial interconnections providing input data signals to the first self-organizing artificial neuron;

wherein the first self-organizing artificial neuron selects the input data signals for connection and disconnection in response to training data being applied to the one or more learning array input nodes.

6. The apparatus of claim 5, the first self-organizing artificial neuron further comprising:

a memory for storing a value for an entropy-based threshold associated with the first self-organizing artificial neuron;

wherein the first self-organizing artificial neuron determines the value for the entropy-based threshold in response to training data being applied to the one or more learning array input nodes.

7. The apparatus of claim 5, the first self-organizing artificial neuron further comprising:

an input control multiplexer selectively connecting and disconnecting one or more initial interconnections providing input control signals to the first self-organizing artificial neuron;

wherein the first self-organizing artificial neuron selects the input control signals for connection and disconnection in response to training data being applied to the one or more learning array input nodes.

8. The apparatus of claim 1, the plurality of artificial neurons including first self-organizing artificial neuron comprising:

a neuronal processor selectively applying a transformation function to input data signals provided to the first self-organizing artificial neuron by one or more initial interconnections associated with the initial configuration of the self-organizing learning array or one or more retained interconnections associated with the trained configuration of the self-organizing learning array, thus contributing to partitioning of an input data space associated with the first self-organizing artificial neuron;

wherein the first self-organizing artificial neuron selects the transformation function to be applied in response to training data being applied to the one or more learning array input nodes.

9. The apparatus of claim 8, the first self-organizing artificial neuron further comprising:

a memory for storing a value for an entropy-based threshold associated with the first self-organizing artificial neuron;

wherein the first self-organizing artificial neuron determines the value for the entropy-based threshold in response to training data being applied to the one or more learning array input nodes.

10. The apparatus of claim 8, the first self-organizing artificial neuron further comprising:

an input control multiplexer selectively connecting and disconnecting one or more initial interconnections providing input control signals to the first self-organizing artificial neuron;

wherein the first self-organizing artificial neuron selects the input control signals for connection and disconnection in response to training data being applied to the one or more learning array input nodes.

11. The apparatus of claim 8, the neuronal processor comprising:

a reduced instruction set processor selectively applying the transformation function to input data signals provided to the first self-organizing artificial neuron by one or more initial interconnection associated with the initial configuration of the self-organizing learning array or one or more retained interconnections associated with the trained configuration of the self-organizing learning array, thus contributing to partitioning the input data space associated with the first self-organizing artificial neuron.

12. The apparatus of claim 1 the first self-organizing artificial neuron further comprising:

a memory for storing a value for an entropy-based threshold associated with the first self-organizing artificial neuron;

wherein the first self-organizing artificial neuron determines the value for the entropy-based threshold in response to training data being applied to the one or more learning array input nodes.

13. The apparatus of claim 1 the first self-organizing artificial neuron further comprising:

an input control multiplexer selectively connecting and disconnecting of one or more initial interconnections providing input control signals to the first self-organizing artificial neuron;

wherein the first self-organizing artificial neuron selects the input control signals for connection and disconnection in response to training data being applied to the one or more learning array input nodes.

14. The apparatus of claim 1 the entropy-based evaluator comprising:

a lookup table storing a plurality of lookup values associated with entropy of the input data space;

a threshold storage storing a threshold value associated with entropy of the input data space;

an entropy calculating unit determining an entropy value based at least in part on one or more lookup values retrieved from the lookup table; and

a comparator unit comparing the determined entropy value and the threshold value to determine an entropy result;

wherein the first self-organizing artificial neuron selects the transformation function to be applied based at least in part on the entropy result.

15. The apparatus of claim 1 the entropy-based evaluator comprising:

a lookup table storing a plurality of lookup values associated with entropy of the input data space;

a maximum entropy storage storing a maximum entropy value associated with entropy of the input data space;

an entropy calculating unit determining an entropy value based at least in part on one or more lookup values retrieved from the lookup table; and

a comparator unit comparing the determined entropy value and the maximum entropy value to determine an entropy result;

wherein the first self-organizing artificial neuron selects the transformation function to be applied based at least in part on the entropy result.

16. A self-organizing artificial neuron, comprising:

one or more neuron data inputs adapted to receive corresponding input data signals;

an input data multiplexer selectively connecting and disconnecting one or more of the input data signals;

a neuronal processor selectively applying a transformation function to one or more connected input data signals, thus contributing to partitioning of an input data space associated with the self-organizing artificial neuron; and

an entropy-based evaluator determining entropy of the input data space associated with the self-organizing artificial neuron, wherein at least one of i) selection of the input data signals for connection and disconnection and ii) selection of the transformation function to be applied by the self-organizing artificial neuron is based at least in part on the determined entropy;

wherein the self-organizing artificial neuron selects the input data signals for, connection and disconnection in response to training data being operatively communicated to the self-organizing artificial neuron;

wherein the self-organizing artificial neuron determines the transformation function to be applied in response to training data being operatively communicated to the self-organizing artificial neuron;

wherein the self-organizing artificial neuron determines the entropy of the input data space in response to training data being operatively communicated to the self-organizing artificial neuron.

17. The artificial neuron of claim 16, further comprising:

a memory for storing a value for an entropy-based threshold associated with the self-organizing artificial neuron;

wherein the self-organizing artificial neuron determines the value for the entropy-based threshold in response to training data being operatively communicated to the self-organizing artificial neuron.

18. The artificial neuron of claim 16, further comprising:

one or more neuron control inputs adapted to receive corresponding input control signals; and

an input control multiplexer selectively connecting and disconnecting one or more of the received input control signals; and

wherein the self-organizing artificial neuron selects the input control signals for connection and disconnection in response to training data being operatively communicated to the self-organizing artificial neuron.

19. A self-organizing artificial neuron, comprising:

one or more neuron data inputs adapted to receive corresponding input data signals;

an input data multiplexer selectively connecting and disconnecting one or more of the input data signals; and

an entropy-based evaluator determining entropy of an input data space associated with the self-organizing artificial neuron, wherein at least one of i) selection of the input data signals for connection and disconnection and ii) selection of a transformation function to be applied by the self-organizing artificial neuron is based at least in part on the determined entropy;

wherein the self-organizing artificial neuron selects the input data signals for connection and disconnection in response to training data being operatively communicated to the self-organizing artificial neuron;

wherein the self-organizing artificial neuron determines the entropy of the input data space in response to training data being operatively communicated to the self-organizing artificial neuron.

20. The artificial neuron of claim 19, further comprising:

a memory for storing a value for an entropy-based threshold associated with the self-organizing artificial neuron;

wherein the self-organizing artificial neuron determines the value for the entropy-based threshold in response to training data being operatively communicated to the self-organizing artificial neuron.

21. The artificial neuron of claim 19, further comprising:

one or more neuron control inputs adapted to receive corresponding input control signals; and

an input control multiplexer selectively connecting and disconnecting one or more of the received input control signals; and

wherein the self-organizing artificial neuron selects the input control signals for connection and disconnection in response to training data being operatively communicated to the self-organizing artificial neuron.

22. A self-organizing artificial neuron, comprising:

one or more neuron data inputs adapted to receive corresponding input data signals;

a neuronal processor selectively applying a transformation function to one or more input data signals, thus contributing to partitioning of an input data space associated with the self-organizing artificial neuron; and

an entropy-based evaluator determining entropy of the input data space associated with the self-organizing artificial neuron, wherein at least one of i) selection of the input data signals for connection and disconnection at the self-organizing artificial neuron and ii) selection of the transformation function to be applied by the self-organizing artificial neuron is based at least in part on the determined entropy;

wherein the self-organizing artificial neuron determines the transformation function to be applied in response to training data being operatively communicated to the self-organizing artificial neuron;

wherein the self-organizing artificial neuron determines the entropy of the input data space in response to training data being operatively communicated to the self-organizing artificial neuron.

23. The artificial neuron of claim 22, further comprising:

a memory for storing a value for an entropy-based threshold associated with the self-organizing artificial neuron;

wherein the self-organizing artificial neuron determines the value for the entropy-based threshold in response to training data being operatively communicated to the self-organizing artificial neuron.

24. The artificial neuron of claim 22, further comprising:

one or more neuron control inputs adapted to receive corresponding input control signals; and

an input control multiplexer selectively connecting and disconnecting of one or more of the received input control signals; and

wherein the self-organizing artificial neuron selects the input control signals for connection and disconnection in response to training data being operatively communicated to the self-organizing artificial neuron.

25. A method for training a self-organizing apparatus, comprising:

a) arranging a plurality of artificial neurons in multiple layers, the plurality of artificial neurons including one or more self-organizing artificial neuron;

b) interconnecting the plurality of artificial neurons with a plurality of initial interconnections to form an initial configuration of a self-organizing learning array, where less than total interconnection is undertaken between adjacent layers;

c) connecting each of one or more learning array input nodes to at least one artificial neuron;

d) applying training data to the one or more learning array input nodes;

e) at each self-organizing artificial neurons, independently determining initial interconnections associated therewith to retain as neuron input connections and initial interconnections associated therewith to release in response to the training data applied in d);

f) forming a trained configuration of the self-organizing learning array based at least in part on the retained interconnnections; and

g) at a first self-organizing artificial neuron, independently determining entropy of an input data space associated with the first self-organizing artificial neuron based at least in part on the training data applied in d), wherein at least one of i) selecting initial interconnections to retain in e) and ii) selecting a transformation function to be applied by the first self-organizing artificial neuron is based at least in part on the determined entropy.

26. The method of claim 25, further comprising:

h) at a first self-organizing artificial neuron, independently and selectively connecting and disconnecting one or more initial interconnections providing input data signals to the first self-organizing artificial neuron, selection of the input data signals for connecting and disconnecting being based at least in part on the training data applied in d).

27. The method of claim 26, further comprising:

i) at the first self-organizing artificial neuron, independently determining a value for an entropy-based threshold associated with the first self-organizing artificial neuron based at least in part on the training data applied in d); and

j) at the first self-organizing artificial neuron, storing the value for an entropy-based threshold in a memory.

28. The method of claim 26, further comprising:

i) at the first self-organizing artificial neuron, independently and selectively connecting and disconnecting one or more initial interconnections providing input control signals to the first self-organizing artificial neuron, selection of the input control signals for connecting and disconnecting being based at least in part on the training data applied in d).

29. The method of claim 25, further comprising:

h) at a first self-organizing artificial neuron, independently and selectively applying a transformation function to input data signals provided to the first self-organizing artificial neuron by one or more initial interconnections associated with the initial configuration of the self-organizing learning array or one or more retained interconnections associated with the trained configuration of the self-organizing learning array to at least partially partition an input data space associated with the first self-organizing artificial neuron, selection of the transformation function to be applied being based at least in part on the training data applied in d).

30. The method of claim 29, further comprising:

i) at the first self-organizing artificial neuron, independently determining a value for an entropy-based threshold associated with the first self-organizing artificial neuron based at least in part on the training data applied in d); and

j) at the first self-organizing artificial neuron, storing the value for an entropy-based threshold in a memory.

31. The method of claim 29, further comprising:

i) at the first self-organizing artificial neuron, independently and selectively connecting and disconnecting one or more initial interconnections providing input control signals to the first self-organizing artificial neuron, selection of the input control signals for connecting and disconnecting being based at least in part on the training data applied in d).

32. The method of claim 25, further comprising:

h) at the first self-organizing artificial neuron, independently determining a value for an entropy-based threshold associated with the first self-organizing artificial neuron based at least in part on the training data applied in d); and

i) at the first self-organizing artificial neuron, storing the value for an entropy-based threshold in a memory.

33. The method of claim 25, further comprising:

h) at the first self-organizing artificial neuron, independently and selectively connecting and disconnecting one or more initial interconnections providing input control signals to the first self-organizing artificial neuron, selection of the input control signals for connecting and disconnecting being based at least in part on the training data applied in d).

34. The method of claim 25, further comprising:

h) at the first self-organizing artificial neuron, independently determining an entropy value based at least in part on one or more stored lookup values; and

i) at the first self-organizing artificial neuron, independently comparing the determined entropy value to a stored threshold value to determine an entropy result, wherein selection of the transformation function in g) is based at least in part on the entropy result.

35. The method of claim 25, further comprising:

h) at the first self-organizing artificial neuron, independently determining an entropy value based at least in part on one or more stored lookup values; and

i) at the first self-organizing artificial neuron, independently comparing the determined entropy value to a stored maximum entropy value to determine an entropy result, wherein selection of the transformation function in g) is based at least in part on the entropy result.

36. A method for training a self-organizing artificial neuron, comprising:

a) receiving one or more input data signals at corresponding neuron data inputs of the self-organizing artificial neuron;

b) at the self-organizing artificial neuron, independently and selectively connecting and disconnecting one or more of the input data signals, selection of the input data signals for connecting and disconnecting being based at least in part on training data operatively communicated to the self-organizing artificial neuron;

c) at the self-organizing artificial neuron, independently and selectively applying a transformation function to one or more connected input data signals to at least partially partition an input data space associated with the self-organizing artificial neuron, selection of the transformation function to be applied being based at least in part on training data operatively communicated to the self-organizing artificial neuron; and

d) at the self-organizing artificial neuron, independently determining entropy of the input data space associated with the self-organizing artificial neuron based at least in part on training data operatively communicated to the self-organizing artificial neuron, wherein at least one of i) selecting the input data signals for connecting and disconnecting in b) and ii) selecting the transformation function to be applied in c) is based at least in part on the determined entropy.

37. The method of claim 36, further comprising:

e) at the self-organizing artificial neuron, independently determining a value for an entropy-based threshold associated with the self-organizing artificial neuron based at least in part on training data operatively communicated to the self-organizing artificial neuron; and

f) at the self-organizing artificial neuron, independently storing the value for the entropy-based threshold in a memory.

38. The method of claim 36, further comprising:

e)-receiving one or more input control signals at corresponding neuron control inputs of the self-organizing artificial neuron; and

f) at the self-organizing artificial neuron, independently and selectively connecting and disconnecting one or more of the received input control signals, selection of the received input control signals for connecting and disconnecting being based at least in part on training data operatively communicated to the self-organizing artificial neuron.

39. A method for training a self-organizing artificial neuron, comprising:

a) receiving one or more input data signals at corresponding neuron data inputs of the self-organizing artificial neuron;

b) at the self-organizing artificial neuron, independently and selectively connecting and disconnecting one or more of the input data signals, selection of the input data signals for connecting and disconnecting being based at least in part on training data operatively communicated to the self-organizing artificial neuron; and

c) at the self-organizing artificial neuron, independently determining entropy of the input data space associated with the self-organizing artificial neuron based at least in part on training data operatively communicated to the self-organizing artificial neuron, wherein at least one of i) selecting the input data signals for connecting and disconnecting in b) and ii) selecting a transformation function to be applied by the self-organizing artificial neuron is based at least in part on the determined entropy.

40. The method of claim 39, further comprising:

d) at the self-organizing artificial neuron, independently determining a value for an entropy-based threshold associated with the self-organizing artificial neuron based at least in part on training data operatively communicated to the self-organizing artificial neuron; and

e) at the self-organizing artificial neuron, independently storing the value for the entropy-based threshold in a memory.

41. The method of claim 39, further comprising:

d) receiving one or more input control signals at corresponding neuron control inputs of the self-organizing artificial neuron; and

e) at the self-organizing artificial neuron, independently and selectively connecting and disconnecting one or more of the received input control signals, selection of the received input control signals for connecting and disconnecting being based at least in part on training data operatively communicated to the self-organizing artificial neuron.

42. A method for training a self-organizing artificial neuron, comprising:

a) receiving one or more input data signals at corresponding neuron data inputs of the self-organizing artificial neuron;

b) at the self-organizing artificial neuron, independently and selectively applying a transformation function to one or more input data signals to at least partially partition an input data space associated with the self-organizing artificial neuron, selection of the transformation function to be applied being based at least in part on training data operatively communicated to the self-organizing artificial neuron; and

c) at the self-organizing artificial neuron, independently determining entropy of the input data space associated with the self-organizing artificial neuron based at least in part on training data operatively communicated to the self-organizing artificial neuron, wherein at least one of i) selecting input data signals for connecting and disconnecting at the self-organizing artificial neuron and ii) selecting the transformation function to be applied in b) is based at least in part on the determined entropy.

43. The method of claim 42, further comprising:

d) at the self-organizing artificial neuron, independently determining a value for an entropy-based threshold associated with the self-organizing artificial neuron based at least in part on training data operatively communicated to the self-organizing artificial neuron; and

e) at the self-organizing artificial neuron, independently storing the value for the entropy-based threshold in a memory.

44. The method of claim 42, further comprising:

d) receiving one or more input control signals at corresponding neuron control inputs of the self-organizing artificial neuron; and

e) at the self-organizing artificial neuron, independently and selectively connecting and disconnecting one or more of the received input control signals, selection of the received input control signals for connecting and disconnecting being based at least in part on training data operatively communicated to the self-organizing artificial neuron.