Concurrent learning and performance information processing system

• US 5,835,902 A
• Filed: 11/02/1994
• Issued: 11/10/1998
• Est. Priority Date: 11/02/1994
• Status: Expired due to Term

• Current Assignee: Metricly, Inc. (Virtana Corp.)
• Original Assignee: Robert J. Jannarone
• Inventors: Jannarone, Robert J.
• Primary Examiner(s): MacDonald, Allen R.
• Assistant Examiner(s): Smith, Jeffrey S.
• Application Number:
  US08/333,204
• Time in Patent Office: 1,469 Days
• Field of Search: 395/23, 395/24, 706/25, 706/26
• US Class Current: 706/26
• CPC Class Codes: G06F 18/2113 by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation
• CPC Class Codes: G06N 3/063 using electronic means
• CPC Class Codes: G06N 3/10 Interfaces, programming languages or software development kits, e.g. for simulating neural networks
• CPC Class Codes: G06V 10/771 Feature selection, e.g. selecting representative features from a multi-dimensional feature space

Abstract

The present invention provides a system for learning from and responding to regularly arriving information at once by quickly combining prior information with concurrent trial information to produce useful learned information. At the beginning of each time trial a vector of measurement values and a vector of measurement plausibility values are supplied to the system, and a learning weight is either supplied to or generated by the system. The system then performs the following operations during each time trial: converting the measurement values to feature values; converting the measurement plausibility values to feature viability values; using each viability value to determine missing value status of each feature value; using non-missing feature values to update parameter learning; imputing each missing feature value from non-missing feature values and/or prior learning; converting imputed feature values to output imputed measurement values; and supplying a variety of feature value and feature function monitoring and interpretation statistics. A parallel embodiment of the system performs all such operations concurrently, through the coordinated use of parallel feature processors and a joint access memory, which contains connection weights and provision for connecting feature processors pairwise. The parallel version also performs feature function monitoring, interpretation and refinement operations promptly and in concert with concurrent operation, through the use of a parallel subsystem that receives interpretable concurrent connection weights from a parallel port. A nonparallel embodiment of the system uses a single processor to perform the above operations however, more slowly than the parallel embodiment, yet faster than available alternatives.

Claims

1. A parallel processing system for computing output values from input values received during a time trial, comprising:

a plurality of processing units, each of said processing units operative to receive, during a time trial and in parallel, individual input values from an input vector; and

a plurality of interconnected conductors, operative to connect each of said processing units to every other processing unit of said system and operative to transfer weighted values among said processing units;

each of said processing units operative to provide, during said time trial, an output value based on said weighted values and each of said processing units operative to update, during said time trial, said weighted values based on said input values; and

said processing units operative to be uniquely paired with each other processing unit, each processing unit of paired processing units operative to send and receive learning and prediction information to the processing unit paired therewith independent of a separate processing unit.

2. The apparatus of claim 1 further comprising a plurality of switching junctions located along said interconnected conductors, said switching junctions being operable for uniquely pairing each of said processing units to every other processor of said system.

3. The apparatus of claim 2 wherein each of said switching junctions selectively connects each of said processing units to only one other of said processing units during a time interval, thereby forming multiple paired sets of said processing units, for communicating said weighted values during a time interval.

4. The apparatus of claim 3 further comprising a plurality of memory elements, each of said memory elements being individually coupled to a separate switching junction and each of said memory elements containing a weighted memory value.

5. The apparatus of claim 3 wherein said switching junctions successively connect different sets of multiple paired sets of said processing units during multiple time intervals.

6. The apparatus of claim 5 wherein said switching junctions successively connect all possible combinations of said different sets of multiple paired sets of said processing units in a minimum number of steps.

7. The apparatus of claim 6 further comprising a plurality of memory elements, each of said memory elements being individually coupled to a separate switching junction and each of said memory elements containing a weighted memory value.

8. The apparatus of claim 3 wherein said conductors comprise a first conductor layer and a second conductor layer, said first and said second conductor layers operable for connection at said switching junctions.

9. The apparatus of claim 8 wherein said switching junctions are located within a semi-conductor layer, said switching junctions located between said first and second conductor layers.

10. The apparatus of claim 9 further comprising a plurality of memory elements, each of said memory elements being individually coupled to a separate switching junction and each of said memory elements containing a weighted memory value.

11. The apparatus of claim 10 wherein each of said switching junctions selectively connects each of said processing units to only one other of said processing units, thereby forming multiple paired sets of said processing units, thereby forming multiple paired sets of said processing units, for communicating weight values during a time interval.

12. The apparatus of claim 1 further comprising a transducer input processor for receiving external measurement vector and for converting said external measurement vector to said input vector, said transducer input processor operative to store a time-ordered preselected number of said input vectors.

13. A processing system for computing output values from input values received during a time trial, comprising:

a processing unit operative to receive, sequentially, input data values from an input vector;

a connection weight matrix that is the inverse of a sample covariance matrix, said connection weight matrix being the inverse of a sample covariance matrix at the beginning of the time trial and the inverse of the updated sample covariance matrix at the end of the time trial;

a memory unit is connected to said processing unit containing elements of said connection weight matrix is stored in sequential order as a data string;

said processing unit operative to provide, during said time trial, an expected output value, said expected output value being calculated as a function of the relative weights of each of said elements of said connection weight matrix and said processing unit operative to update, during said time trial, said connection weight values as a function of the relationship among the input values; and

said connection weight matrix being the inverse of a sample covariance matrix at the beginning of the time trial and the inverse of the updated sample covariance matrix based on the input values at the end of the trial.

14. The apparatus of claim 13 wherein said processing unit is operative to access, in sequential order, each element of said connection weight matrix.

15. The apparatus of claim 14 wherein said processing unit is operative to perform all imputing operations based on the currently accessed connection weight when said currently accessed connection weight is encountered in said string.

16. The apparatus of claim 15 wherein said processing unit is operative to perform all updating operations based on the currently accessed connection weight when said currently accessed connection weight is encountered in said data string.

17. A computer system for identifying a statistical relationship among multiple input values contained in input data vectors m IN! provided during multiple time trials, comprising:

a processing unit operative to receive an input vector during a time trial;

a memory unit containing connection weight elements representative of relationships among current input m IN! elements based on prior m IN! vectors received; and

said processing unit operative to update said connection weight elements based on non-missing values of said input data vector received and operative to update said connection weight elements based on a component learning weight, l C!(ƒ

), said l C!(ƒ

) being a distinct learning weight for each element m IN!(ƒ

) and said l C!(ƒ

) determining the amount of adjustment to said connection weight elements that said m IN!(ƒ

) causes relative to prior measurement vector elements m IN!(ƒ

) received.

18. A parallel processing system for computing output values from an input data vector received during a time trial, comprising:

a plurality of processing units connected in parallel for processing separate values of said input data vector received during said trial; and

a memory unit connected to each of said processors and said memory unit containing a connection weight matrix, said connection weight matrix being an inverse of a matrix of elements;

each of said processing units operative to provide an output value, during said trial, for a missing input data value by implementing mathematical regression analysis using said connection weight elements and operative to directly update, during said trial, each of said connection weight elements to reflect the relationship of input data vector elements from the current trial.

19. An apparatus for providing communication paths among each of a plurality of processing units, comprising:

a first processing unit, a second processing unit and a third processing unit;

a first conductor path connected to said first processing unit;

a second conductor path connected to said second processing unit;

a third conductor path connected to said third processing unit;

a first and a second switching junction, each of said switching junctions connected at different points along said first conductor;

said second conductor path extending from said second processing unit to said first junction, said first junction operative for connecting said first processing unit to said second processing unit via said second conductor path and said first conductor path;

said third conductor path extending from said third processing unit to said second switching junction, said second switching junction operative for connecting said first processing unit to said third processing unit via said third conductor path and said first conductor path; and

a third junction connected to said third conductor path and said second conductor path, said third junction operative for connecting said third processing unit to said second processing unit via said third conductor path and said second conductor path.

20. The apparatus of claim 19 further comprising a plurality of memory elements, one of said memory elements being located at each of said switching junctions, said memory elements being accessible by said processing units connected at said first, second, and third switching junctions.

21. The apparatus of claim 20 wherein said memory elements contain weight values.

22. The apparatus of claim 21 further comprising a control processing unit operative to control the switching of said switching junction.

23. The apparatus of claim 22 wherein said control unit is operative to provide a first signal to said switching junctions for selectively connecting each processing unit to selected ones of said memory elements.

24. The apparatus of claim 23 wherein said control unit is operative to provide a second control signal to each of said switching junctions for connecting each said processing unit to the other processing unit connected at said switching junction.

25. The apparatus of claim 24 wherein said control unit is operative to provide a third control signal to said switching junctions for selectively connecting one of said processing units connected at said switching junctions to selected ones of said memory elements.

26. In a neural network type computer system for identifying statistical relationships among multiple variables by analyzing input data input vectors m IN! provided during multiple time trials, updating the connection weight elements of the neural network type computer system and providing an output vector reflecting an expected output, m OUT!(ƒ

), for a missing data value of m IN!(ƒ

), comprising the steps of;

receiving at a processor an input data vector during a time trial;

imputing, during said time trial, an output value for a missing input value of said data vector based on connection weight elements for a neural network type computer forming the inverse, ω

IN! of a covariance matrix; and

updating, during said current time trial, said connection weight elements, based on non-missing values of said input measurement vector received, thereby forming an updated inverse covariance matrix, ω

OUT!.

27. The method of claim 26 further comprising updating said connection weight elements based on a component learning weight, l C!(ƒ

), said l C!(ƒ

) being a distinct learning weight for each measurement vector received and said l C!(ƒ

) determining the amount of adjustment that said input data vector causes relative to prior input data vectors received, during said step of updating.

28. The method of claim 26 further comprising updating said connection weight matrix based on the prior mean vector of all prior data vectors including said input data vector received.

29. The method of claim 27 further comprising updating said connection weight matrix based on the prior mean vector of all prior measurements vectors received.

30. The method of claim 27 further comprising:

receiving a global learning weight, l, used in determining the amount of adjustment that said input data vector has during said step of updating relative to prior data vectors received;

receiving a learning history parameter vector λ

IN! that is an indicator of the prior learning weight and missing value history of each input data vector;

receiving a viability vectorv that indicates the extent to which an input data vector is missing; and

calculating each said component learning weight l C!(ƒ

) by the equation l C!(ƒ

)=lν

(ƒ

)λ

IN!.

31. The method of claim 29 further comprising the step of updating said learning history parameter vector elements based on component learning weight elements l C!(ƒ

) and a prior learning history parameter λ

IN! wherein

space="preserve" listing-type="equation">λ

OUT!I(F)=λ

IN!(A)(1+l)/(1+l C!(ƒ

)).

32. The method of claim 29 further comprising:

updating said connection weight matrix based on a prior mean vector μ

IN! of all prior measurements vectors received, wherein said prior mean vector, μ

IN!, equals μ

OUT! from the previous measurement trial and wherein each element of m OUT! is computed as

space="preserve" listing-type="equation">μ

OUT!(ƒ

)=(l C!(ƒ

)m IN!(ƒ

)+μ

IN!(.function.))/(1+l C!(ƒ

)), and

wherein μ

IN!(ƒ

) equals one for the first measurement trial.

33. The method of claim 32 further comprising the step of updating said connection weight elements utilizing an intermediate imputed vector, e IN!.

34. The method of claim 33 wherein:

each element of e IN! is computed as,

space="preserve" listing-type="equation">e IN!(ƒ

)=ν

(ƒ

)(m IN!(ƒ

)-μ

IN (ƒ

))/(1+l C!(ƒ

)).

35.

35. The method of claim 34 further comprising updating said connection weight matrix according to the process of calculating,

36. The method of claim 34 further comprising said step of imputing missing values m IN!(ƒ

) wherein

space="preserve" listing-type="equation">m OUT!(ƒ

)=μ

IN!(ƒ

)+e IN!(ƒ

)(2-ν

(ƒ

))+x(ƒ

)(ν

(ƒ

)-1)/ω

(ƒ

,ƒ

).

37. The method of claim 34 further comprising the step of calculating a variance vector ν

D,OUT!, wherein, based on a prior mean vector μ

IN!, of all prior data vectors received, wherein said prior mean μ

IN!, equals μ

OUT! from the previous measurement trial and wherein the elements of ν

D,OUT! are computed as

space="preserve" listing-type="equation">ν

D,OUT!(ƒ

)=le IN!(ƒ

).sup.2 +ν

D,IN!(ƒ

)/(1+l),

wherein

ν

D,IN! equals ν

ID,OUT! from the previous trial and wherein ν

D,IN! equals one for the first trial.

38. The method of claim 37 further comprising the step of computing a standardized squared deviance values, d 1!(ƒ

), from non-missing learned means, wherein

space="preserve" listing-type="equation">d 1!(ƒ

)=(m IN!μ

OUT!(ƒ

)).sup.2 /ν

D,OUT!(ƒ

).

39.

39. The method of claim 34 further comprising the step of calculating a regressed measurement value, m(ƒ

), wherein

space="preserve" listing-type="equation">m(ƒ

)=μ

IN!(ƒ

)-x(ƒ

)/ω

(ƒ

,.function.)+e IN!(ƒ

).

40. The method of claim 39 further comprising the step of calculating a variance vector, ν

D,OUT!(ƒ

), wherein, based on a prior mean vector, μ

IN!(ƒ

), of all prior measurements vectors received, wherein said prior mean, μ

IN!(ƒ

), equals μ

OUT!(ƒ

) from the previous measurement trial and wherein

space="preserve" listing-type="equation">ν

D,OUT!(ƒ

)=le IN!(ƒ

).sup.2 +ν

D,IN!(ƒ

)/(1+l),

wherein

ν

D,IN!(ƒ

) equals ν

D,OUT!(ƒ

) from the previous trial and wherein ν

D,IN!(ƒ

) equals for the first trial.

41. The method of claim 39 further comprising the step of computing a standardized squared deviance values, d 2!(ƒ

), between regressed values d 2!(ƒ

), wherein

space="preserve" listing-type="equation">d 2!(ƒ

)=(m IN!-m(ƒ

)).sup.2 /ν

D,OUT!(ƒ

).

42. The method of claim 35 further comprising the step of accessing a set of processor pairs for computing x,

said processors pairs being connected to switching junctions, each of said switching junctions connecting only one pair of processors and being operable for uniquely pairing each of said processor ƒ

to every other processor g of said system during a time interval i, said switching junctions each connected to one element, ω

IN!(ƒ

,g) of said connection weight matrix ω

IN!, comprising the steps of;

(a) accessing multiple sets of processors during a time interval, each of said processors being paired with only one other processor during said time interval,

(b) retrieving, by each processor, the connection weight element ω

IN!(ƒ

,g) located at said switching junction;

(c) transferring an e IN!, (ƒ

) to processor (g) and e IN!(g) to processor (ƒ

) connected at said switching junction;

(d) computing x(ƒ

)=x+e IN!(g)ω

IN!(ƒ

,g) within each processor ƒ and

x(g);

+e IN!(ƒ

)ω

IN!(ƒ

, g) within processor g; and

(e) repeating steps (a) through (d) until all processors have been paired with every other processor of said system.

43. The method of claim 35 further comprising the step of accessing a set of processor pairs for updating elements of ω

IN!,

said processors pairs being connected to switching junctions, each of said switching junctions connecting only one pair of processors and being operable for uniquely pairing each of said processor ƒ

to every other processor g of said system during selected time intervals, said switching junctions each connected to one element ω

IN!(ƒ

,g) of said connection weight matrix ω

IN!, comprising the steps of;

(a) accessing multiple sets of processors during a time interval, each of said processors being paired with only one other processor during said time interval,

(b) retrieving, by one of said processors, the connection weight element ω

IN! located at said switching junction;

(c) transferring an e IN!, (ƒ

) to processor (g) and e IN!(g) to processor (ƒ

) connected at said switching junction;

(d) computing x(ƒ

)=x+e IN!(g)ω

IN!(ƒ

,g) within each processor ƒ and

x(g);

+e IN!(f)ω

IN!(ƒ

,g) within processor g; and

(e) repeating steps (a) through (d) until all processors have been paired with every other processor of said system.

44. The method of claim 35 further comprising the step of computing x in a single processor computer system,

said single processor storing the elements of ω

IN! in a data structure as a consecutive string of ω

IN! elements, comprising the steps;

(a) accessing the connection weight element ω

IN! of data string;

(b) updating x corresponding to the row value of the ω

IN! element from the connection weight matrix;

(c) accessing next ω

IN! in string;

(d) if the current ω

IN! element is not on the main diagonal of the covariance matrix and not the last ω

IN! updating x corresponding to the column value of the ω

IN! element from the connection weight matrix; and

(d) repeat step (a) through (d).

45. The method of claim 35 further comprising the step of updating ω

IN! in a single processor computer system,

said single processor storing the elements of ω

IN! in a data structure as a consecutive string of ω

IN! elements, comprising the steps;

(a) accessing the first connection weight element ω

IN! of data string;

(b) updating ω

IN!

(c) access next ω

IN!

(d) if the current ω

IN! element is on the main diagonal of the covariance matrix and not the last ω

IN! proceed to step (b).

46. In a neural network type computer system for analyzing input data, a method of identifying statistical relationships among multiple variables by analyzing input data vectors m IN! provided during multiple time trials, comprising the steps of:

receiving at said processing unit an input data vector from an input data device during a time trial;

retrieving connection weight elements of a connection weight matrix ω

IN! for a neural network type computer system, being the inverse of a covariance matrix; and

updating said connection weight matrix elements based on non-missing values of said input data vector received and updating said connection weight elements based on a component learning weight, l C!(ƒ

), said l C!(ƒ

) being a distinct learning weight for each measurement vector received and said l C!(ƒ

) determining the amount of adjustment that said measurement vector causes relative to prior measurement vectors received, thereby forming an updated inverse covariance matrix, ω

OUT!.

47. The method of claim 46 further comprising updating said connection weight matrix based on the prior mean vector of all prior measurements vectors including said input data vector received.

48. The method of claim 47 further comprising updating said connection weight matrix based on the prior mean vector of all prior data vectors received.

49. The method of claim 46 further comprising:

receiving a global learning weight l used in determining the amount adjustment that said data vector has during said step of updating relative to prior data vectors received;

receiving a learning history parameter λ

OUT!(F) that is an indicator of the prior learning weight of each input data vector;

receiving a viability vector ν

(ƒ

) that indicates the extent to which an input measurement vector is missing; and

calculating said component learning weight, l C!(ƒ

), by the equation l C!(ƒ

)=lν

(ƒ

)λ

OUT!(F).

50. The method of claim 49 further comprising the steps of updating said learning history parameter vector elements based on component learning weight elements l C!(ƒ

) and a prior learning history parameter λ

IN! wherein said prior learning history parameter λ

IN! equals λ

OUT! from a previous trial and wherein

space="preserve" listing-type="equation">λ

OUT!(F)=λ

IN!(A)(1+l)/(1+l C!(ƒ

)).

51. The method of claim 49 further comprising:

updating said connection weight matrix based on a prior mean vector μ

IN! of all prior measurements vectors received, wherein said prior mean μ

IN! equals μ

OUT! from the previous measurement trial and wherein each element of μ

OUT! is computed as

space="preserve" listing-type="equation">μ

OUT!(ƒ

)=(l C!(ƒ

)m IN!(ƒ

)+μ

IN!(.function.))/(1+l C!(ƒ

)), and

wherein μ

IN! equals zero before the first time trial.

52. The method of claim 51 further comprising the step of updating said connection weight elements utilizing an intermediate imputed vector, e IN!.

53. The method of claim 52 wherein each element of e IN!, is computed as

54. The method of claim 53 further comprising updating said connection weight matrix according to the process of calculating,

55. The method of claim 54 further comprising the step of accessing a set of processor pairs for computing x,

said processors pairs being connected to switching junctions, each of said switching junctions connecting only one pair of processors and being operable for uniquely pairing each of said processor ƒ

to every other processor g of said system during a time interval i, said switching junctions each connected to element, ω

IN!(ƒ

,g), of said connection weight matrix ω

IN!, comprising the steps of;

(a) accessing multiple sets of processors during a time interval, each of said processors being paired with only one other processor during said time interval,

(b) retrieving, by each processor, the connection weight element ω

IN!(ƒ

,g) located at said switching junction;

(c) transferring an e IN!, connected at said switching junction;

(d) computing x(ƒ

)=x+e IN!(g)ω

IN!(ƒ

,g) within each processor ƒ and

x(g);

+e IN!(ƒ

)ω

IN!(ƒ

, g) within processor g; and

(e) repeating steps (a) through (d) until all processors have been paired with every other processor of said system.

56. The method of claim 54 further comprising the step of accessing a set of processor pairs for updating elements of ω

IN!,

said processors pairs being connected to switching junctions, each of said switching junctions connecting only one pair of processors and being operable for uniquely pairing each of said processors (ƒ

) to every other processor (g) of said system during selected time intervals, said switching junctions each connected to element, ω

IN!(ƒ

, g), of said connection weight matrix ω

IN!, comprising the steps of;

(a) accessing multiple sets of processors during a time interval, each of said processors being paired with only one other processor during said time interval,

(b) retrieving, by one of said processors, the connection weight element ω

IN!(ƒ

f, g) located at said switching junction;

(c) updating said connection weight element, ω

IN!(ƒ

, g);

(d) transferring said connection weight element ω

IN!(ƒ

, g); and

(e) repeating steps (a) through (d) until all processors have been paired with every other processor of said system.

57. The method of claim 54 further comprising the step of computing x in a single processor computer system,

said single processor storing the elements of ω

IN! in a data structure as a consecutive string of ω

IN! elements, comprising the steps;

(a) accessing a connection weight element of ω

IN! of said data string;

(b) updating x(ƒ

) corresponding to the row value of the ω

IN! element from the connection weight matrix;

(c) accessing next ω

IN! element in string;

(d) if the current ω

IN! element is not on the main diagonal of the covariance matrix and not the last ω

IN! updating x(g) corresponding to the column value of the ω

IN! element from the connection weight matrix; and

(e) repeating step (a) through (c).

58. The method of claim 54 further comprising the step of updating ω

IN! in a single processor computer system,

said single processor storing the elements of ω

IN! in a data structure as a consecutive string of ω

IN! elements, comprising the steps;

(a) accessing the first connection weight element ω

IN! of data string;

(b) updating ω

IN!

(c) accessing the next ω

IN! element

(d) if the current ω

IN! element is on the main diagonal of the covariance matrix and not the last ω

IN! proceed to step (b).

59. An information processing system for computing output values from an input measurement vector and for evaluating input and output measurements of said system, comprising:

a first subsystem of processing units operative to receive initial input measurement values and operative to convert, according to input conversion functions, said input measurement values to input feature values for use by said first subsystem to impute output feature values from non-missing input feature values and/or input learned regression parameters;

said first subsystem operative to convert, said output feature values to final output measurement values;

said first subsystem containing a memory unit of connection weights upon which learning and output performance are based; and

a second subsystem of processing units, connected to said first system, for receiving output data from said first subsystem for display and for evaluation.

60. The information processing system of claim 59 wherein said second subsystem further comprises a first controller operative to control the learning function of said first system.

61. The information processing system of claim 59 wherein said second subsystem further comprises a second controller operative to control the input conversion functions and output conversion functions.

62. The information processing system of claim 59 wherein said second subsystem is operative to receive said connection weight elements from said memory.

63. The information processing system of claim 62 wherein said second subsystem is operative to receive said connection weights and transfer said adjusted weights to for said display and evaluation.

64. The information processing system of claim 59 wherein said second subystem is operative to disable the learning function of said first subsystem when abnormal deviation of input values occur.

65. The information processing system of claim 64 wherein said second subsystem calculates the first-order difference between a current measurement value and the corresponding measurement value stored from the previous trial to identify sudden change.