Method and apparatus for implementation of neural networks for face recognition

• US 5,699,449 A
• Filed: 11/14/1994
• Issued: 12/16/1997
• Est. Priority Date: 11/14/1994
• Status: Expired due to Term

• Current Assignee: University of Connecticut (State of Connecticut)
• Original Assignee: University of Connecticut (State of Connecticut)
• Inventors: Javidi, Bahram
• Primary Examiner(s): Boudreau, Leo
• Assistant Examiner(s): Davis, Monica S.
• Application Number:
  US08/339,118
• Time in Patent Office: 1,128 Days
• Field of Search: 382/118, 382/156, 382/278, 382/280, 359/561, 359/559, 395/25, 364/822
• US Class Current: 382/156
• CPC Class Codes: G06N 3/0675 using electro-optical, acousto-optical or opto-electronic means
• CPC Class Codes: G06V 10/92 using spatial domain filters, e.g. joint transform correlators
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G07C 9/253 visually

Abstract

A method and apparatus for implementation of neural networks for face recognition is presented. A nonlinear filter or a nonlinear joint transform correlator (JTC) employs a supervised perceptron learning algorithm in a two-layer neural network for real-time face recognition. The nonlinear filter is generally implemented electronically, while the nonlinear joint transform correlator is generally implemented optically. The system implements perception learning to train with a sequence of facial images and then classifies a distorted input image in real-time. Computer simulations and optical experimental results show that the system can identify the input with the probability of error less than 3%. By using time multiplexing of the input image under investigation, that is, using more than one input image, the probability of error for classification can be reduced to zero.

Claims

1. A real-time object recognition system comprising:

image detector means for capturing an input image of an object to be recognized;

means for storing a plurality of reference images, each of said reference images being indicative of a cluster of images of an object on record; and

a two-layer neural network comprising,

a first layer for nonlinear joint correlation of said input image with each corresponding said reference image to generate a corresponding correlation signal, and

a second layer having nonlinearity for combining said correlation signals to provide a combined correlation signal, wherein said combined correlation signal above a threshold is indicative that the object to be recognized is the object on record.

2. The system of claim 1 wherein said first layer of said two-layer neural network comprises a joint transform correlator.

3. The system of claim 2 wherein said joint transform correlator is a nonlinear joint transform correlator.

4. The system of claim 3 wherein said joint transform correlator comprises:

a first spatial light modulator receptive to said input image and said reference images, said first spatial light modulator positioned at an input plane of said joint transform correlator, said first spatial light modulator for projecting said input image and for projecting said reference images;

a first Fourier transform lens positioned for jointly Fourier transforming said input image and said reference images to project corresponding joint Fourier transformed images;

a second spatial light modulator receptive to said joint Fourier transformed images, said second light modulator positioned at a Fourier plane of said joint transform correlator, said second spatial light modulator for projecting a joint power spectrum for each of said joint transformed images;

a second Fourier transform lens positioned for Fourier transforming said joint power spectrums to project corresponding correlation images indicative of a correlation between said input image and a corresponding said reference image; and

detector means for detecting said correlation images to provide said correlation signals, said detector means positioned at an output plane of said joint transform correlator.

5. The system of claim 4, wherein said first spatial light modulator comprises a liquid crystal television.

6. The system of claim 4, wherein said first spatial light modulator comprises an electronically addressed spatial light modulator.

7. The system of claim 4, wherein said first spatial light modulator comprises an optically addressed spatial light modulator.

8. The system of claim 4, wherein said second spatial light modulator means comprises a liquid crystal light valve.

9. The system of claim 4, wherein said second spatial light modulator comprises an electronically addressed spatial light modulator.

10. The system of claim 4, wherein said second spatial light modulator comprises an optically addressed spatial light modulator.

11. The system of claim 4 wherein said joint power spectrum is a nonlinear joint power spectrum comprising an expression:

##EQU5## where, α and

β

are spatial frequency coordinates,

R(α

,β

)exp{jΦ

_R (α

,β

! is the Fourier transform of a reference image function r(x,y),

S(α

,β

)exp jΦ

_s (α

,β

)! is the Fourier transform of an input image function s(x,y),

G(ω

) is the Fourier transform of the nonlinearity, and

x_o is the refraction between the input function and the reference function.

12. The system of claim 1, wherein during a training mode of operation:

said image detector means further comprises means for capturing a plurality of training images for the object on record,

said training images being combined in accordance with a learning algorithm to generate a plurality of corresponding said reference images.

13. The system of claim 12 wherein said learning algorithm is a perception learning algorithm.

14. The system of claim 13 wherein said perceptron learning algorithm comprises:

space="preserve" listing-type="equation">w(t+1)=w(t)+μ

(t)x(t),

where;

w=(w₁,w₂ . . . ,w_N)^T is a weight vector at time t,

x(t) is an input pattern at time t, and

μ

t(t) is a multiplier that is given by;

##EQU6##

15. The system of claim 1 wherein:

said image detector means further comprises means for capturing a plurality of input images of the object to be recognized,

said plurality of input images being time multiplexed; and

whereby each said correlation signal is indicative of a correlation between said time multiplexed input images and a corresponding said reference image.

16. The system of claim 3 wherein said first layer of said two-layer neural network comprises:

signal processing means having memory means, said memory means including said means for storing said plurality of reference images; and

a joint transform correlator comprising;

(1) spatial light modulator means,

(2) a Fourier transform lens,

(3) detector means, and

(4) switch means for switching between a first position wherein said signal processing means is connected to said camera means and a second position wherein said signal processing means is connected to said detector means,

(A) with said switch means in said first position,

(a) said signal processing means is receptive to an input facial image signal indicative of said input image, said input facial image signal is stored in said memory means,

(b) said spatial light modulator (SLM) means is receptive to said input image signal and reference image signals indicative of said reference images, said spatial light modulator positioned at an input plane of said joint transform correlator, said spatial light modulator for projecting a SLM input image in response to said input facial image signal and for projecting SLM reference images in response to said reference image signals,

(c) said Fourier transform lens is positioned for jointly Fourier transforming said input image and each corresponding said reference image to project corresponding joint transformed images, and

(d) said detector means for detecting said joint transformed images, said detector means is positioned at a Fourier plane of said joint transform correlator, said detector means generates a joint power spectrum signal for each of said joint transformed images,

(B) with said switch means in said second position,

(a) said signal processing means is receptive to said joint power spectrum signals which are stored in said memory means,

(b) said spatial light modulator means is receptive to said joint power spectrum signals, said spatial light modulator for projecting a joint power spectrum image in response to each of said joint power spectrum signals,

(c) said Fourier transform lens transforms said joint power spectrum images to project corresponding correlation images indicative of a correlation between said input image and a corresponding said reference image, and

(d) said detector means for detecting said correlation images to provide said correlation signals, said detector means positioned at an output plane of said joint transform correlator, wherein said output plane of said joint transform correlator with said switch means in said second position is said Fourier plane of said joint transform correlator in said first position.

17. The system of claim 16, wherein said spatial light modulator means comprises a liquid crystal television.

18. The system of claim 16, wherein said spatial light modulator comprises an electronically addressed spatial light modulator.

19. The system of claim 16, wherein said spatial light modulator comprises an optically addressed spatial light modulator.

20. The system of claim 16 wherein said joint power spectrum is a nonlinear joint power spectrum comprising an expression:

##EQU7## where, α and

β

are spatial frequency coordinates,

R(α

,β

)exp{jΦ

_R (α

,β

! is the Fourier transform of a reference image function r(x,y),

S(α

,β

)exp jΦ

_s (α

,β

)! is the Fourier transform of an input image function s(x,y),

G(ω

) is the Fourier transform of the nonlinearity, and

x_o is the refraction between the input function and the reference function.

21. The system of claim 1 wherein said means for storing a plurality of reference images comprises an optical film having said reference images thereon.

22. The system of claim 1 wherein said means for storing a plurality of reference images comprises an optical memory having said reference images thereon.

23. The system of claim 21 wherein said optic film is disposed on a card.

24. The system of claim 1 wherein said means for storing a plurality of reference images comprises recording said reference images in a thick optical material, the thickness of said optical material being at least several times the wavelength of light from which said reference images were recorded.

25. The system of claim 1 wherein said two-layer neural network comprises:

signal processing means for processing an input image signal indicative of said input image and reference image signals indicative of said reference images, said signal processing means including memory means for storing signals including signals defining an executable algorithm of a nonlinear filter to generate said correlation signals at said first layer of said two-layer neural network, said signal processing means for combining said correlation signals to provide said combined correlation signal at said second layer of said two-layer neural network.

26. The system of claim 25 wherein said algorithm of said nonlinear filter comprises the following expression:

##EQU8## where, α and

β

are spatial frequency coordinates,

R(α

,β

)exp{jΦ

_R (α

,β

! is the Fourier transform of a reference image function r(x,y),

S(α

,β

)exp jΦ

_s (α

,β

)! is the Fourier transform of an input image function s(x,y),

G(ω

) is the Fourier transform of the nonlinearity, and

x_o is the refraction between the input function and the reference function.

27. The system of claim 25, wherein during a training mode of operation:

said image detector means further comprises means for capturing a plurality of training images for the object on record and generating a corresponding plurality of training image signals indicative thereof; and

said memory means for storing signals defining an executable learning algorithm for combining said training image signals to generate a plurality of corresponding said reference image signals, said executable learning algorithm is a perceptron learning algorithm.

28. The system of claim 27 wherein said algorithm comprises a perceptron learning algorithm comprising the following expression:

space="preserve" listing-type="equation">w(t+1)=w(t)+μ

(t)x(t),

where;

w=(w₁,w₂ . . . ,w_N)^T is a weight vector at time t,

x(t) is an input pattern at time t, and

μ

(t) is a multiplier that is given by;

##EQU9##

29. The system of claim 25 wherein:

said image detector means further comprises means for capturing a plurality of input images of the object to be recognized and generates a corresponding plurality of image signals indicative thereof, and

said plurality of image signals being time multiplexed to generate said input image signal.

30. The system of claim 1 wherein said second layer of said two-layer neural network comprises:

said second layer for summing said correlation signals and binarizing said summed correlation signals to provide said combined correlation signal.

31. The system of claim 1 wherein said second layer having nonlinearity comprises said second layer having a multiple level nonlinearity for indicating a degree of confidence for recognition.

59. The system of claim 1 wherein said object is a person and said input image is an input facial image.

32. A method for real-time object recognition comprising the steps of:

capturing an input image of an object to be recognized;

storing a plurality of reference images, each of said reference images being indicative of a cluster of images of an object on record; and

nonlinearly jointly correlating said input image with each corresponding said reference image at a first layer of a two-layer neural network to generate a corresponding correlation signal, and

combining said correlation signals at a second layer having nonlinearity of said two-layer neural network to provide a combined correlation signal, wherein said combined correlation signal above at least one threshold is indicative that the object to be recognized is the object on record.

33. The method of claim 32 wherein said step of jointly correlating comprises the steps of:

projecting said input image;

projecting said reference images;

jointly Fourier transforming said input image and said reference images to project corresponding joint transformed images;

projecting a joint power spectrum for each of said joint transformed images;

Fourier transforming said joint power spectrums to project corresponding correlation images indicative of a correlation between said input image and a corresponding said reference image; and

detecting said correlation images to provide said correlation signals.

34. The method of claim 33 wherein said joint power spectrum is a nonlinear joint power spectrum comprising an expression:

##EQU10## where, α and

β

are spatial frequency coordinates,

R(α

,β

)exp{j Φ

_R (α

,β

! is the Fourier transform of a reference image function r(x,y),

S(α

,β

)exp jΦ

_s (α

,β

)! is the Fourier transform of an input image function s(x,y),

G(ω

) is the Fourier transform of the nonlinearity, and

x_o is the refraction between the input facial function and the reference function.

35. The method of claim 32, wherein during a training mode of operation:

said step of capturing comprises capturing a plurality of training images for the object on record; and

combining said training images in accordance with a learning algorithm to generate a plurality of corresponding said reference images.

36. The method of claim 35 wherein said learning algorithm is a perception learning algorithm.

37. The method of claim 36 wherein said perceptron learning algorithm comprises:

space="preserve" listing-type="equation">w(t+1)=w(t)+μ

(t)x(t),

where;

w=(w₁,w₂ . . . ,w_N)^T is a weight vector at time t,

x(t) is an input pattern at time t, and

μ

(t) is a multiplier that is given by;

##EQU11##

38. The method of claim 32 wherein:

said step of capturing further comprises capturing a plurality of input images of the object to be recognized;

time multiplexing said plurality of input images; and

whereby said correlation signals are indicative of a correlation between said time multiplexed input images and a corresponding said reference image.

39. The method of claim 32 wherein said step of jointly correlating comprises the steps of:

switching between a first position and a second position,

(A) in said first position,

(a) projecting a spatial light modulator input image,

(b) projecting spatial light modulator reference images,

(c) jointly Fourier transforming said input image and each corresponding said reference image to project corresponding joint transformed images,

(d) detecting said joint transformed images,

(e) generating a joint power spectrum signal for each of said joint transformed images,

(B) in said second position,

(a) projecting a joint power spectrum image in response to each of said joint power spectrum signals,

(b) Fourier transforming said joint power spectrum images,

(c) projecting corresponding correlation images indicative of a correlation between said input image and a corresponding said reference image, and

(d) detecting said correlation images with nonlinearity,

(e) generating said correlation signals.

40. The method of claim 39 wherein said joint power spectrum signal is a nonlinear joint power spectrum signal comprising an expression:

##EQU12## where, α and

β

are spatial frequency coordinates,

R(α

,β

)exp{jΦ

_R (α

,β

! is the Fourier transform of a reference image function r(x,y),

S(α

,β

)exp jΦ

_s (α

,β

)! is the Fourier transform of an input image function s(x,y),

G(ω

) is the Fourier transform of the nonlinearity, and

x_o is the refraction between the input function and the reference function.

41. The method of claim 32 wherein said step of jointly correlating comprises jointly correlating using a nonlinear filter.

42. The method of claim 41 wherein said nonlinear filter comprises the following expression:

##EQU13## where, α and

β

are spatial frequency coordinates,

R(α

,β

)exp{jΦ

_R (α

,β

! is the Fourier transform of a reference image function r(x,y),

S(α

,β

)exp jΦ

_s (α

,β

)! is the Fourier transform of an input image function s(x,y),

G(ω

) is the Fourier transform of the nonlinearity, and

x_o is the refraction between the input function and the reference function.

43. The method of claim 32 wherein wherein said step of jointly correlating comprises jointly correlating using a joint transform correlator.

44. The method of claim 32 wherein wherein said step of jointly correlating comprises jointly correcting using a nonlinear joint transform correlator.

45. The method of claim 32 wherein said step of combining at said second layer comprises:

summing said correlation signals which have passed through a nonlinearity; and

binarizing said summed correlation signals to provide said combined correlation signal.

46. The method of claim 32 wherein said second layer having nonlinearity comprises said second layer having a multiple level nonlinearity for indicating a degree of confidence for recognition.

60. The method of claim 32 wherein said object is a person and said input image is an input facial image.

47. A real-time object recognition system comprising:

image detector means for capturing an input image of an object to be recognized;

a plurality of reference images, each of said reference images being indicative of a cluster of images of an object on record, said reference images stored on an optical film deposited on a card; and

a two-layer neural network comprising,

(1) a first layer comprising,

(a) a light source for illuminating each of said reference images to project corresponding reference images,

(b) a spatial light modulator receptive to said input image, said spatial light modulator in response to illumination by said light source projecting said input image, and

(c) detector means for detecting combined corresponding said reference images and said input image to provide a corresponding nonlinear joint correlation signal, each of said joint correlation signals being indicative of a joint correlation between said input image and each corresponding said reference image, and

(2) a second layer having nonlinearity for combining said correlation signals to provide a combined correlation signal.

48. The system of claim 47 wherein said spatial light modulator comprises a liquid crystal television.

49. The system of claim 47 wherein said spatial light modulator comprises an electronically addressed spatial light modulator.

52. The system of claim 47 wherein:

said image detector means further comprises means for capturing a plurality of input images of the object to be recognized,

said plurality of input images being time multiplexed; and

whereby each said correlation signal is indicative of a correlation between said time multiplexed input images and a corresponding said reference image.

61. The system of claim 47 wherein said object is a person and said input image is an input facial image.

50. The system of 47 wherein said spatial light modulator comprises an optically addressed spatial light modulator.

51. The system of 47 wherein said spatial light modulator comprises a liquid crystal light value.

53. A method for real-time object recognition comprising the steps of:

capturing an input image of an object to be recognized;

storing a plurality of reference images each of said reference images being indicative of a cluster of images of an object on record, storing said reference images on an optical film deposited on a card;

at a first layer of a two-layer neural network,

(a) illuminating each of said reference images to project corresponding reference images,

(b) illuminating a spatial light modulator receptive to said input image to project said input image,

(c) detecting combined corresponding said reference images and said input image to provide a corresponding nonlinear joint correlation signal, each of said joint correlation signals being indicative of a joint correlation between said input image and each corresponding said reference image; and

combining at a second layer having nonlinearity of said two-layer neural network said correlation signals to provide a combined correlation signal, wherein said combined correlation signal above at least one threshold is indicative that the object to be recognized is the object on record.

54. The method of claim 53 wherein said spatial light modulator comprises a liquid crystal television.

55. The method of claim 53 wherein said spatial light modulator comprises an electronically addressed spatial light modulator.

58. The method of claim 53 wherein:

said step of capturing further comprises capturing a plurality of input images of the object to be recognized;

time multiplexing said plurality of input images; and

whereby each said correlation signal is indicative of a correlation between said time multiplexed input images and a corresponding said reference image.

62. The method of claim 53 wherein said object is a person and said input image is an input facial image.

56. The method of 53 wherein said spatial light modulator comprises an optically addressed spatial light modulator.

57. The method of 53 wherein said spatial light modulator comprises a liquid crystal light value.