Neural network learning system inferring an input-output relationship from a set of given input and output samples

• US 5,479,576 A
• Filed: 02/23/1995
• Issued: 12/26/1995
• Est. Priority Date: 01/30/1992
• Status: Expired due to Term

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Fukumizu, Kenji, Watanabe, Sumio
• Primary Examiner(s): MacDonald, Allen R.
• Assistant Examiner(s): Dorvil, Richemond
• Application Number:
  US08/393,024
• Time in Patent Office: 306 Days
• Field of Search: 395/21, 395/23, 395/24, 364/130, 364/275
• US Class Current: 706/25
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/08 Learning methods

Abstract

A neural network learning system in which an input-output relationship is inferred. The system includes a probability density part for determining a probability density on a sum space of an input space and an output space from a set of given input and output samples by learning, the probability density on the sum space being defined to have a parameter, and an inference part for inferring a probability density function based on the probability density from the probability density part, so that an input-output relationship of the samples is inferred from the probability density function having a parameter value determined by learning, the learning of the parameter being repeated until the value of a predefined parameter differential function using a prescribed maximum likelihood method is smaller than a prescribed reference value.

Claims

1. A neural network learning system for inferring an input-output relationship from a set of given input and output samples, comprising:

a) probability density means for determining a probability density on a sum space of an input space and an output space from the set of given input and output samples by learning a value of a parameter through a prescribed maximum likelihood method;

b) inference means for determining a probability density function based on the probability density from said probability density means, to infer the input-output relationship of the samples from the probability density function;

wherein the learning a value of the parameter is repeated by said inference means until a value of a predefined parameter differential function using the prescribed maximum likelihood method is smaller than a prescribed value, thereby determining said parameter value;

c) conditional probability distribution means for computing a conditional probability distribution from either the given input samples or the given output samples in accordance with the probability density function determined by said inference means; and

d) output means for obtaining an inference value based on the conditional probability distribution from said conditional probability distribution means and for outputting said inference value.

2. A neural network learning system according to claim 1, wherein said probability density is derived from a linear combination of exponential functions exp (-φ

(w, x, y)) with respect to the given input and output samples, where w is the probability density function parameter, x is an input vector, y is an output vector and φ

is an arbitrary function indicating a relationship between w and x and y.

3. A neural network learning system according to claim 1, wherein the input and output samples are distributed on the sum space in accordance with a normal distribution, and said probability density is derived from a linear combination of normal distributions with respect to the input and output samples, said normal distributions defined by Gaussian type exponential functions.

4. A neural network learning system according to claim 1, wherein said conditional probability distribution computed in accordance with the probability density function is a probability function P(w;

y|x), where y|x indicates a relationship between y and x, and said output means obtains an average of outputs y in accordance with said function P(w;

y|x) for a given input x as said inference value so that said average of the outputs y is output by said output means.

5. A neural network learning system according to claim 1, wherein said conditional probability distribution computed in accordance with the probability density function is a probability function P(w;

y|x), where y|x indicates a relationship between y and x, and said output means obtains and outputs a set of outputs y of said function P(w;

y|x) for a given input x together with a set of output probabilities P(w;

y|x) corresponding to the respective outputs y.

6. A neural network learning system according to claim 1, wherein said conditional probability distribution computed in accordance with the probability density function is a function P(w;

x|y), where x|y indicates a relationship between x and y, and said output means obtains an average of inputs x in accordance with said function P(w;

x|y) for a given output y as said inference value so that said average of the inputs x is output by said output means.

7. A neural network learning system according to claim 1, wherein a probability of occurrence for a given input is obtained based on the probability density of said probability density means, so that said output means determines and outputs the inference value for said given input based on said probability of occurrence.

8. A neural network learning system according to claim 1, wherein a probability of occurrence for a given output is determined based on the probability density of said probability density means, so that said output means determines and outputs the inference value for the given output based on said probability of occurrence.

9. A neural network learning system according to claim 1, wherein said probability density is derived from the linear combination of exponential functions exp (-φ

(w, x, y)) with respect to the given input and output samples, where w is a parameter, x is an input vector, y is an output vector and φ

is an arbitrary function indicating a relationship between w, x, and y.

10. A neural network learning system according to claim 1, wherein the input and output samples are distributed on the sum space in accordance with a normal distribution, and said probability density is derived from a linear combination of normal distributions with respect to the input and output samples, said normal distributions defined by Gaussian type exponential functions.

15. A neural network learning system for inferring an input-output relationship from a set of given input and output samples, comprising:

a) probability density means for determining a probability density on a sum space of an input space and an output space from the set of given input and output samples by learning a value of a parameter through a prescribed maximum likelihood method;

b) inference means for determining a probability density function based on the probability density from said probability density means, to infer the input-output relationship of the samples from the probability density function;

wherein the learning a value of the parameter is repeated by said inference means until a value of a predefined parameter differential function using the prescribed maximum likelihood method is smaller than a prescribed value, thereby determining said parameter value;

c) conditional probability distribution means for computing a conditional probability distribution from either the given input samples or the given output samples in accordance with the probability density function determined by said inference means; and

d) output means for obtaining an inference value based on the conditional probability distribution from said conditional probability distribution means and for outputting said inference value;

wherein;

1) said conditional probability distribution computed in accordance with the probability density function is an input distribution function P(w;

x|y),

2) x|y indicates a relationship between x and y, and

3) said output means determines and outputs a set of inputs x in accordance with said function P(w;

x|y) for a given output y, and a set of values of said function P(w;

x|y) corresponding to the respective inputs x.

16. A neural network learning system for inferring an input-output relationship from a set of given input and output samples, comprising:

a) probability density means for determining a probability density on a sum space of an input space and an output space from the set of given input and output samples by learning a value of a parameter through a prescribed maximum likelihood method;

b) inference means for determining a probability density function based on the probability density from said probability density means, to infer the input-output relationship of the samples from the probability density function;

wherein the learning a value of the parameter is repeated by said inference means until a value of a predefined parameter differential function using the prescribed maximum likelihood method is smaller than a prescribed value, thereby determining said parameter value;

c) conditional probability distribution means for computing a conditional probability distribution from either the given input samples or the given output samples in accordance with the probability density function determined by said inference means; and

d) output means for obtaining an inference value based on the conditional probability distribution from said conditional probability distribution means and for outputting said inference value;

wherein;

1) a set of outputs and a probability of occurrence for each one of the outputs are determined for a given input, a probability density to determine the probability of occurrence being approximated by a linear combination of predetermined exponential functions according to a prescribed probability distribution function and being learned repeatedly using the prescribed maximum likelihood method; and

2) when the probability of occurrence for the given input is obtained, a determination is made whether the given input is known or unknown based on said probability of occurrence for the given input.

17. A neural network learning system for inferring an input-output relationship from a set of given input and output samples, comprising:

a) probability density means for determining a probability density on a sum space of an input space and an output space from the set of given input and output samples by learning a value of a parameter through a prescribed maximum likelihood method;

b) inference means for determining a probability density function based on the probability density from said probability density means, to infer the input-output relationship of the samples from the probability density function;

wherein the learning a value of the parameter is repeated by said inference means until a value of a predefined parameter differential function using the prescribed maximum likelihood method is smaller than a prescribed value, thereby determining said parameter value;

c) conditional probability distribution means for computing a conditional probability distribution from either the given input samples or the given output samples in accordance with the probability density function determined by said inference means; and

d) output means for obtaining an inference value based on the conditional probability distribution from said conditional probability distribution means and for outputting said inference value;

wherein an inference value for each of a set of outputs and a variance of the inference values are determined by said learning performed based on the probability density on the sum space for the given input and output samples.

18. A neural network learning system according to claim 17, wherein said probability density is defined to have a first parameter w1 used to determine the inference value and a second parameter w2 used to determine the variance of the inference value, and wherein the learning of each of the parameters w1 and w2 is repeated until the sum of squares of values of a predefined parameter differential function using a prescribed maximum likelihood method with respect to the first and second parameters is smaller than a prescribed reference value so as to determine values of the first and second parameters, thereby the inference value and the variance thereof are determined.

19. A neural network learning system according to claim 18, wherein said probability density is derived from the linear combination of exponential functions exp (-(y-φ

(w2, x))/σ

(w1, x))²) with respect to the given inputs x and outputs y where φ

is a function indicating a relationship between x and y.

20. A neural network learning system according to claim 19, wherein a function σ

(w1, x) of said exponential function is derived from a linear combination of Gaussian type exponential functions.

21. A neural network learning system according to claim 18, wherein a tolerance of the inference value at a given critical factor that satisfies a required condition is output based on an output of an average function, defined to have the second parameter w2, and an output of a variance function, defined to have the first parameter w1.

23. A neural network learning system according to claim 20, wherein said parameter computing means determines the values of the parameters for the probability distributions, each parameter having an average of each probability distribution, a variance thereof and a coefficient of the linear combinations, the values of said parameters being determined by said parameter computing means based on the statistical quantities of the data in each of the clusters, said parameter learning means supplying said values of said parameters to said output probability means.

24. A neural network learning system according to claim 23, wherein a number of clusters in said clustering means accords with the number of the linear combinations of probability distributions used in the output probability means.

25. A neural network learning system according to claim 24, wherein said clustering means carries out the classifying of the given data into clusters using a prescribed non-hierarchical classifying method, and the number of linear combinations of probability distributions used in said output probability means is predetermined as being the same as the number of clusters from said clustering means.

26. A neural network learning system according to claim 24, wherein said clustering means carries out the classifying of the given data into clusters using a hierarchical classifying method, and the number of clusters in said clustering means is determined at the end of said classifying, the number of linear combinations of probability distributions used in said output probability means being determined as being the same as said number of clusters.

22. A neural network learning system in which an input-output relationship for a set of input and output samples is predetermined, said system comprising:

a) output probability means for determining output probabilities for given inputs in accordance with probability distributions on a sum space of an input space and an output space; and

b) parameter learning means for carrying out a learning of parameters, the parameters respectively defining the probability distributions of said output probability means, wherein said parameter learning means includes;

1) clustering means for classifying the given data on the sum space into a set of clusters to determine statistical quantities of data in each of the clusters, and

2) parameter computing means for determining values of the parameters for the probability distributions of the output probability means based on the statistical quantities of the data from the clustering means.

11. A data analyzing device used in the neural network learning system according to claim 22 for performing a clustering of a set of given data, said device comprising:

classifying means for classifying data, stored in a first storage part, into a set of clusters in accordance with a prescribed probability distribution with respect to data included in each cluster so as to allocate the given data to the clusters;

initializing means for determining initial values of parameters from the data in the first storage part, each of said parameters including an average, a standard deviation, and a coefficient to define the probability distribution of the data for each cluster, and said initial values of the parameters being stored in a second storage part; and

parameter updating means for updating said parameters in said second storage part for each cluster each time the given data is allocated by said classifying means to the clusters,

wherein the data is repeatedly classified by said classifying means into a set of clusters in accordance with a probability distribution based on said parameters in said second storage part each time said parameters are updated by said parameter updating means.

12. A data analyzing device according to claim 11, further comprising convergence discriminating means for detecting whether or not a prescribed convergence criterion is satisfied by the parameters being updated, wherein said convergence discriminating means instructs said parameter updating means to stop the updating of the parameters when the convergence criterion is satisfied.

13. A data analyzing device according to claim 12, wherein said convergence discriminating means computes a change in the parameters from the sum of squares of weighted errors of the parameters previously stored in the second storage part to the sum thereof obtained from the parameters after the updating is performed, and instructs said parameter updating means to stop the updating of the parameters when it is detected that the change is smaller than a prescribed threshold value.

14. A data analyzing device according to claim 12, wherein said convergence discriminating means increments a count each time the updating of the parameters is performed, and instructs said parameter updating means to stop the updating of the parameters when it is detected that the count is greater than a prescribed value.

27. A neural network learning system according to claim 22, wherein said parameter learning means comprises:

initializing means for obtaining initial values of the parameters from said clustering means and storing the initial values of the parameters in a storage part of the output probability means; and

updating means for updating the parameters by performing an inference process using a prescribed maximum likelihood method, starting from the initial values of the parameters stored in the storage part, so that optimal values of the parameters are determined.