Blind signal processing system employing information maximization to recover unknown signals through unsupervised minimization of output redundancy

• US 5,706,402 A
• Filed: 11/29/1994
• Issued: 01/06/1998
• Est. Priority Date: 11/29/1994
• Status: Expired due to Fees

• Current Assignee: Salk Institute for Biological Studies
• Original Assignee: Salk Institute for Biological Studies
• Inventors: Bell, Anthony J.
• Primary Examiner(s): Davis, George B.
• Application Number:
  US08/346,535
• Time in Patent Office: 1,134 Days
• Field of Search: 395/23, 395/20
• US Class Current: 706/22
• CPC Class Codes: G06F 18/21342 using statistical independence, i.e. minimising mutual information or maximising non-gaussianity
• CPC Class Codes: G06N 3/088 Non-supervised learning, e.g. competitive learning
• CPC Class Codes: H03H 2021/0034 Blind source separation
• CPC Class Codes: H04L 2025/03464 Neural networks
• CPC Class Codes: H04L 25/03165 using neural networks

Abstract

A neural network system and unsupervised learning process for separating unknown source signals from their received mixtures by solving the Independent Components Analysis (ICA) problem. The unsupervised learning procedure solves the general blind signal processing problem by maximizing joint output entropy through gradient ascent to minimize mutual information in the outputs. The neural network system can separate a multiplicity of unknown source signals from measured mixture signals where the mixture characteristics and the original source signals are both unknown. The system can be easily adapted to solve the related blind deconvolution problem that extracts an unknown source signal from the output of an unknown reverberating channel.

Claims

1. A method performed in a neural network having input means for receiving a plurality J of input signals (X_j) and output means for producing a plurality I of output signals (U_i) each said output signal U_i representing a combination of said input signals (X_j) weighted by a plurality I of bias weights (W_i0) and a plurality I² of scaling weights (W_ij) such that (U_i)=(W_ij)(X_j)+(W_i0), said method minimizing the information redundancy among said output signals (U_j), wherein 0<

i≦

I>

1 and 0<

j≦

J>

1 are integers, said method comprising;

(a) selecting initial values for said bias weights (W_i0) and said scaling weights (W_ij);

(b) producing a plurality I of training signals (Y_i) responsive to a transformation of said input signals (X_j) such that Y_i =g(U_i), wherein g(x) is a nonlinear function and the Jacobian of said transformation is J=det(∂

Y_i /∂

X_j) when J=I; and

(c) adjusting said bias weights (W_i0) and said scaling weights (W_ij) responsive to one or more samples of said training signals (Y_i) such that each said bias weight Wi_i0 is changed proportionately to a corresponding bias measure Δ

W_i0 accumulated over said one or more samples and each said scaling weight W_ij is changed proportionately to a corresponding scaling measure Δ

W_ij =ε

·

∂

(ln|J|)/∂

W_ij accumulated over said one or more samples, wherein ε

>

0 is a learning rate.

2. The method of claim 1 wherein said nonlinear function g(x) is a nonlinear function selected from a group consisting essentially of the solutions to the equation ##EQU22## and said Δ

W_i0=ε

·

(-r|Y_i |^r-1 sgn(Y_i)) accumulated over said one or more samples and each said scaling weight W_ij is changed proportionately to a corresponding scaling measure Δ

W_ij =ε

·

((cof(W_ij)/det(W_ij))·

rX_j |Y_i|^r-1 sgn(Y_i)) accumulated over said one or more samples.

3. The method of claim 1 wherein said nonlinear function g(x) is a nonlinear function selected from a group consisting essentially of g₁ (x)=tanh(x) and g₂ (x)=(1-e^-x)^-1 and said Δ

W_i0 selected from the group consisting essentially of Δ

₁ W_i0 =ε

·

(-2Y_i) and Δ

₂ W_i0 =ε

·

(1-2Y_i) accumulated over said one or more samples and each said scaling weight W_ij is changed proportionately to the a corresponding scaling measure Δ

W_ij selected from the group consisting essentially of Δ

₁ W_ij =ε

·

((cof(W_ij)/det(W_ij))-2X_j Y_i) and Δ

₂ W_ij =ε

·

((cof(W_ij)/det(W_ij))+X_j (1-2Y_i)) accumulated over said one or more samples.

4. A neural-network implemented method for recovering one or more of a plurality I of independent source signals (S_i) from a plurality J>

I of sensor signals (X_j) each including a combination of at least some of said source signals (S_i) wherein 0<

i<

I>

1 and 0<

j≦

J>

I are integers, said method comprising;

(a) selecting a plurality I of bias weights (W_i0) and a plurality I² of scaling weights (W_ij);

(b) adjusting said bias weights (W_i0) and said scaling weights (W_ij) by repeatedly performing the steps of;

(b.1) producing a plurality I of estimation signals (U_i) responsive to said sensor signals (X_j) such that (U_i)=(W_ij)(X_j)+(W_i0),

(b.2) producing a plurality I of training signals (Y_i) responsive to a transformation of said sensor signals (X_j) such that Y_i =g(U_i), wherein g(x) is a nonlinear function and the Jacobian of said transformation is J=det(∂

Y_i /∂

X_j) when J=I, and

(b.3) adjusting each said bias weight W_i0 and each said scaling weight W_ij responsive to one or more samples of said training signals (Y_i) such that said each bias weight W_i0 is changed proportionately to a bias measure Δ

W_i0 accumulated over said one or more samples and said each scaling weight W_ij is changed proportionately to a corresponding scaling measure Δ

W_ij =ε

·

∂

(ln|J|)/∂

W_ij accumulated over said one or more samples, wherein ε

>

0 is a learning rate; and

(c) producing said estimation signals (U_i) to represent said one or more recovered source signals (S_i).

5. The method of claim 4 wherein said nonlinear function g(x) is a nonlinear function selected from a group consisting essentially of the solutions to the equation ##EQU23## and said Δ

W_i0 =ε

·

(-rX_j |Y_i |^r-1 sgn(Y_i)) accumulated over said one or more samples and each said scaling weight W_ij is changed proportionately to a corresponding scaling measure Δ

W_ij =ε

·

((cof(W_ij)/det(W_ij))-rX_j |Y_i |^r-1 sgn(Y_i)) accumulated over said one or more samples.

6. The method of claim 4 wherein said nonlinear function g(x) is a nonlinear function selected from a group consisting essentially of g₁ (x)=tanh(x) and g₂ (x)=(1-e^-x)^-1 and said adjusting comprises:

(c) adjusting said bias weights (W_i0) and said scaling weights (W_ij) responsive to one or more samples of said training signals (Y_i) such that each said bias weight W_i0 is changed proportionately to a corresponding bias measure Δ

W_i0 selected from the group consisting essentially of Δ

₁ W_i0 =ε

·

(-2Y_i) and Δ

₂ W_i0 =ε

·

(1-2Y_i) accumulated over said one or more samples and each said scaling weight W_ij is changed proportionately to the a corresponding scaling measure Δ

W_ij selected from the group consisting essentially of Δ

_l W_ij =Δ

·

((cof(W_ij)/det(W_ij))-2X_j Y_i) and Δ

₂ W_ij =ε

·

((cof(W_ij)/det (W_ij))+X_j (1-2Y_i)) accumulated over said one or more samples.

7. A method implemented in a transversal filter having an input for receiving a sensor signal X that includes a combination of multipath reverberations of a source signal S and having a plurality I of delay line tap output signals (T_i) distributed at intervals of one or more time delays τ

, said source signal S and said sensor signal X varying with time over a plurality J≧

I of said time delay intervals τ

such that said sensor signal X has a value X_j at time τ

(j-1) and each said delay line tap output signal T_i has a value X_j+1-i representing said sensor signal value X_j delayed by a time interval τ

(i-1), wherein τ

>

0 is a predetermined constant and 0<

i≦

I>

1 and 0<

j≦

J≧

I are integers, said method recovering said source signal S from said sensor signal X and comprising;

(a) selecting a plurality I of filter weights (W_i);

(b) adjusting said filter weights (W_i) by repeatedly performing the steps of

(b.1) producing a plurality K=I of weighted tap output signals (V_k) by combining said delay line tap output signals (T_i) such that (V_k)=(F_ki) (T_i), wherein 0<

k≦

K=I>

1 are integers, and wherein F_ki =W_k+1-i when 1≦

k+1-i≦

I and F_ki =0 otherwise,

(b.2) summing a plurality K=I of said weighted tap signals (V_k) to produce an estimation signal ##EQU24## wherein said estimation signal U has a value U_j at time τ

(j-1), (b.3) producing a plurality J of training signals (Y_j) responsive to a transformation of said sensor signal values (X_j) such that Y_j =g(U_j) wherein g(x) is a nonlinear function and the Jacobian of said transformation is J=det(∂

Y_i /∂

X_j) when J=I, and

(b.4) adjusting each said filter weight W_i responsive to one or more samples of said training signals (Y_j) such that said each filter weight W_i is changed proportionately to a corresponding leading measure Δ

W₁ accumulated over said one or more samples when i=1 and a corresponding scaling measure Δ

W_i =ε

·

∂

(ln|J|)/∂

W_i accumulated over said one or more samples otherwise; and

(c) producing said estimation signal U to represent said recovered source signal S.

8. The method of claim 7 wherein said nonlinear function g(x) is a nonlinear function selected from a group consisting essentially of g₁ (x)=tanh(x) and g₂ (X)=(1-e^-x)^-1 and said Δ

W₁ selected from the group consisting essentially of ##EQU25## accumulated over said one or more samples when i=1 and a corresponding scaling measure Δ

W_i selected from the group consisting essentially of ##EQU26## accumulated over said one or more samples otherwise.

9. The method of claim 7 wherein said nonlinear function g(x) is a nonlinear function selected from a group consisting essentially of the solutions to the equation ##EQU27## accumulated over said one or more samples when i=1 and a corresponding scaling measure ##EQU28## accumulated over said one or more samples otherwise.

10. A neural network for recovering a plurality of source signals from a plurality of mixtures of said source signals, said neural network comprising:

input means for receiving a plurality J of input signals (X_j) each including a combination of at least some of a plurality I of independent source signals (S_i), wherein 0<

i≦

I>

1 and 0<

j≦

J≧

I are integers;

weight means coupled to said input means for storing a plurality I of bias weights (W_i0) and a plurality I² of scaling weights (W_ij);

output means coupled to said weight means for producing a plurality I of output signals (U_i) responsive to said input signals (X_j) such that (U_i)=(W_ij) (X_j)+(W_i0);

training means coupled to said output means for producing a plurality I of training signals (Y_i) responsive to a transformation of said input signals (X_j) such that Y_i =g(U_i),

wherein g(x) is a nonlinear function and the Jacobian of said transformation is J=det(∂

Y_i /∂

X_j) when J=I;

adjusting means coupled to said training means and said weight means for adjusting said bias weights (W_i0) and said scaling weights (W_ij) responsive to one or more samples of said training signals (Y_i) such that each said bias weight W_i0 is changed proportionately to a corresponding bias measure Δ

W_i0 accumulated over said one or more samples and each said scaling weight W_ij is changed proportionately to a corresponding scaling measure Δ

W_ij =ε

·

∂

(ln|J|)/∂

W_ij accumulated over said one or more samples, wherein ε

>

0 is a learning rate.

11. The neural network of claim 10 wherein said nonlinear function g(x) is a nonlinear function selected from a group consisting essentially of the solutions to the equation ##EQU29## and said bias measure Δ

W_i0 =ε

·

(-r|Y_i |^r-1 sgn(Y_i)) and said scaling measure Δ

W_ij =ε

·

((cof(W_ij)/det(W_ij))-rX_j |Y_i |^r-1 sgn(Y_i)).

12. The neural network of claim 10 wherein said nonlinear function g(x) is a nonlinear function selected from a group consisting essentially of g₁ (x)=tanh(x) and g₂ (x)=(1-e^-x)^-1 and said bias measure Δ

W_i0 is selected from a group consisting essentially of Δ

₁ W_i0 =-2Y_i and Δ

₂ W_i0 =1-2Y_i and said scaling measure Δ

W_ij is selected from a group consisting essentially of Δ

W₁ W_ij =(cof(W_ij)/det(W_ij))-X_j 2Y_i and Δ

₂ W_ij =(cof(W_ij)/det(W_ij))+X_j (1-2Y_i).

13. A system for adaptively cancelling one or more interferer signals (S_n) comprising:

input means for receiving a plurality J of input signals (X_j) each including a combination of at least some of a plurality I of independent source signals (S_i) that includes said one or more interferer signals (S_n), wherein 0<

i≦

I>

1, 0<

j≦

J≧

I and 0<

n≦

N≧

1 are integers;

weight means coupled to said input means for storing a plurality I of bias weights (W_i0) and a plurality I² of scaling weights (W_ij);

output means coupled to said weight means for producing a plurality I of output signals (U_i) responsive to said input signals (X_j) such that (U_i)=(W_ij) (X_j)+(W_i0);

training means coupled to said output means for producing a plurality I of training signals (Y_i) responsive to a transformation of said input signals (X_j) such that Y_i =g(U_i), wherein g(x) is a nonlinear function and the Jacobian of said transformation is J=det(∂

Y_i /∂

X_j);

adjusting means coupled to said training means and said weight means for adjusting said bias weights (W_i0) and said scaling weights (W_ij) responsive to one or more samples of said training signals (Y_i) such that each said bias weight W_i0 is changed proportionately to a corresponding bias measure Δ

W_i0 accumulated over said one or more samples and each said scaling weight W_ij is changed proportionately to a corresponding scaling measure Δ

W_ij =ε

·

∂

(ln|J|)/∂

W_ij accumulated over said one or more samples, wherein ε

>

0 is a learning rate; and

feedback means coupled to said output means and said input means for selecting one or more said output signals (U_n) representing said one or more interferer signals (S_n) for combination with said input signals (X_j), thereby cancelling said interferer signals (S_n).

14. The system of claim 13 wherein said nonlinear function g(x) is a nonlinear function selected from a group consisting essentially of the solutions to the equation ##EQU30## and said bias measure Δ

W_i0 =ε

·

(-r|Y_i |^r-1 sgn(Y_i)) and said scaling measure Δ

W_ij =ε

·

((cof(W_ij)/det(W_ij))-rX_j |Y_i |^r-1 sgn(Y_i)).

15. The system of claim 13 wherein said nonlinear function g(x) is a nonlinear function selected from a group consisting essentially of g₁ (x)=tanh(x) and g₂ (x)=(1-e^-x)^-1 and said bias measure Δ

W_i0 is selected from a group consisting essentially of Δ

₁ W_i0 =-2Y_i and Δ

₂ W_i0 =1-2Y_i and said scaling measure Δ

W_ij is selected from a group consisting essentially of Δ

₁ W_ij =(cof(W_ij)/det(W_ij))-X_j 2Y_i and Δ

₂ W_ij =(cof(W_ij)/det(W_ij))+X_j (1-2Y_i).