Method for editing an input quantity for a neural network

• US 5,706,401 A
• Filed: 08/19/1996
• Issued: 01/06/1998
• Est. Priority Date: 08/21/1995
• Status: Expired due to Fees

• Current Assignee: Siemens AG
• Original Assignee: Siemens AG
• Inventors: Tresp, Volker
• Primary Examiner(s): Hafiz, Tariq R.
• Assistant Examiner(s): Rhodes, Jason W.
• Application Number:
  US08/699,558
• Time in Patent Office: 505 Days
• Field of Search: 395/21, 395/2.41, 395/22, 395/23, 395/24, 395/11, 395/932
• US Class Current: 706/25
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology

Abstract

In a method for supplementing missing data in a time series used as an input to a neural network or for improving noise-infested data supplied to a neural network, error distribution densities for the missing values are calculated on the basis of the known measured values from the time series and their known or predetermined error distribution density, and samples are taken from this error distribution density according to the Monte Carlo method. These each lead to an estimated or predicted value whose average is introduced for the value to be predicted. The method can be employed for the operation as well as for the training of the neural network, and is suitable for use in all known fields of utilization of neural networks.

Claims

1. A method for training a computerized neural network, comprising the steps of:

a) in a computer, forming a time series of a set of measured values of a variable dynamic input quantity by determining respective values of the input quantity at discrete points in time;

b) in said computer, identifying a statistical noise distribution of an uncorrelated noise of finite variance that has a chronological average of zero and is superimposed on the measured values;

c) in said computer, generating a substitute value for any missing measured values in the time series by, for each missing value, calculating a statistical missing value noise distribution according to said known noise distribution from at least one of said measured values neighboring the missing value in the time series and calculating said substitute value from a plurality of Monte Carlo samples of the missing value obtained according to the missing value noise distribution and replacing said missing value with said substitute value;

d) supplying said time series, with the substitute value generated in step (c) replacing any missing value, from said computer to a computerized neural network as input quantities; and

e) training said neural network using said time series and a behavior of a technical system represented by the neural network.

2. A method according to claim 1, wherein step (c) comprises obtaining said plurality of Monte Carlo samples of the missing value according to the missing value noise distribution and determining values thereof by calculating an arithmetic average of all of said samples.

3. A method according to claim 1, wherein a value for a first of two missing and immediately neighboring measured values of the time series is generated first and a value for a second of said two missing and immediately neighboring measured values is generated thereafter using the value generated first.

4. A method according to claim 1 comprising repeating steps (a) through (b) multiple times.

5. A method according to claim 1, wherein the step of training said neural network includes, this, in a back-propagation learning step, determining a learning step width for the input quantities of the neural network normed to one by dividing the plurality of Monte Carlo samples by 0.1.

6. A method according to claim 1, where in step (b) comprises using a Gaussian distribution as said statistical noise distribution.

7. A method according to claim 1 wherein step (a) comprises forming said a time series with form:

space="preserve" listing-type="equation">y.sub.t =f(y.sub.t-1, y.sub.t-2, . . . , y.sub.t-N)+ε

.sub.t

wherein ε

_t is said statistical noise distribution, y are measured values of the time series, y_t is said missing value for which said value is to be generated by the neural network, the function f is internally available to the neural network, and wherein the statistical error distribution density is determined in step (b) as

space="preserve" listing-type="equation">P.sub.ε

(y.sub.t-1 -f(y.sub.t-1, y.sub.t-2, . . . , y.sub.t-N))=P (y.sub.t |y.sub.t-1,y.sub.t-2, . . . , y.sub.t-N)

from which Monte Carlo samples y_t-k^l. . . y_t-k^s are taken and said value of the missing value to be generated by the neural network with said samples is calculated;

##EQU9## wherein y_t-k are the missing measured value in the time series, k≦

N, m represents all known measured values of the time series, and S is the number of samples.

8. A method according to claim 7, wherein said function f is modeled by the neural network.

9. A method according to claim 7, wherein the function f is stored in a memory accessible by the neural network.

10. A method according to claim 7 comprising the additional step of training said neural network using said time series and a behavior of a technical system represented by the neural network, including training the neural network with at least one generated value according to a learning function:

##EQU10## wherein w represents neuron weighting, L is a logarithmic probability function, η

is a learning factor, and wherein ##EQU11## with NN_w values of the function from the neural network, and employing measured values of the time series for y_l^s and, when a value is not present,

space="preserve" listing-type="equation">P.sup.M (y.sup.l |y.sup.m)

Monte Carlo samples are obtained from the probability distribution density.

11. A method according to claim 7 comprising the additional step of training said neural network using said time series and a behavior of a technical system represented by the neural network, including training the neural network with at least one generated value according to a learning function:

##EQU12## wherein w represents neuron weighting, L is a logarithmic probability function, η

is a learning factor, and wherein ##EQU13## wherein NN_w are values of the function from the neural network, y^s₁ are measured values of the time series and wherein

space="preserve" listing-type="equation">P.sup.M (y.sup.l |y.sup.m)

Monte Carlo samples are obtained from the probability distribution density, with m representing all known measured values of the time series.

12. A method according to claims 11 comprising training the neural network with a learning rule:

##EQU14## wherein w represents neuron weighting, L is a logarithmic probability function, η

is a learning factor, and wherein ##EQU15##

13. A method according to claim 1, wherein the statistical noise distribution of the measured values is unknown and the measured values are superimposed with further noise having a statistical noise distribution which is known, wherein step (a) comprises forming said time series with a form:

space="preserve" listing-type="equation">z.sub.t =y.sub.t +δ

=f(y.sub.t-1,y.sub.t-2, . . . , y.sub.t-N)+ε

.sub.t

wherein ε

_t is the unknown statistical noise distribution, δ

_t ;

is the known statistical noise distribution, y;

are the measured values of the time series, y_t ;

is the value to be generated by the neural network, and wherein the statistical error distribution density is determined as;

space="preserve" listing-type="equation">P.sub.ε

(y.sub.t-1 -f(y.sub.t-1, y.sub.t-2, . . . , y.sub.t-N))=P (y.sub.t |y.sub.t-1,y.sub.t-2, . . . , y.sub.t-N)

and the overall probability density over the time series is determined with;

##EQU16## and wherein step (c) comprises generating said value for said missing value by the neural network derived from at least one edited value as and obtaining Monte Carlo samples for

space="preserve" listing-type="equation">P (y.sub.t-1, . . . , y.sub.t-N |z).