Infrared and visible fusion face recognition system

• US 7,602,942 B2
• Filed: 11/12/2004
• Issued: 10/13/2009
• Est. Priority Date: 11/12/2004
• Status: Active Grant

• Current Assignee: Honeywell International Inc.
• Original Assignee: Honeywell International Inc.
• Inventors: Bazakos, Michael E., Morellas, Vassilios, Ma, Yunqian
• Primary Examiner(s): Bali, Vikkram
• Assistant Examiner(s): YEH, EUENG NAN
• Application Number:
  US10/987,806
• Publication Number:
  US 20060102843A1
• Time in Patent Office: 1,796 Days
• Field of Search: 382/103, 382/115, 382/117, 702/3, 209/577
• US Class Current: 382/103
• CPC Class Codes: G01J 3/36 Investigating two or more bands of a spectrum by separate detectors
• CPC Class Codes: G06V 40/166 using acquisition arrangements

Abstract

A face detection and recognition system having several arrays imaging a scene in the infrared and visible spectrums. The system may use weighted subtracting and thresholding to distinguish human skin in a sensed image. A feature selector may locate a face in the image. The image may be cropped with a frame or border incorporating essentially only the face. The border may be superimposed on images from an infrared imaging array and the visible imaging array. Sub-images containing the face may be extracted from within the border on the infrared and visible images, respectively, and compared with a database of face information to attain recognition of the face. Confidence levels of recognition for infrared and visible imaged faces may be established. A resultant confidence level of recognition may be determined from these confidence levels. Infrared lighting may be used as needed to illuminate the scene.

Claims

1. A detection system comprising:

a first detector array for providing a first image having a field-of-view of a scene in a lower reflective infrared band;

a second detector array for providing a second image having the field-of-view of the scene in an upper reflective infrared band;

a third detector array for providing a third image having the field-of-view of the scene in a visible band;

a subtractive mechanism, connected to the first, second and third detector arrays, for determining the differences of intensities of elements of the first and second images and forming a fourth image of elements having the differences of intensities and having the field-of-view;

a thresholder, connected to the subtractive mechanism, for nulling the elements of the fourth image having differences of intensities that do not meet a certain threshold intensity value;

a feature selector, connected to the thresholder, for finding facial features in the fourth image and detecting a face with the features;

a framer, connected to the feature selector, for placing a frame around the face in the fourth image;

a first integrator, connected to the framer, for inserting the frame that surrounds the face in the first image; and

a second integrator, connected to the framer, for inserting the frame that surrounds the face in the third image.

2. The system of claim 1, further comprising:

a first extractor, connected to the first integrator, for extracting a first sub-image from within the frame containing the face in the first image;

a second extractor, connected to the second integrator, for extracting a second sub-image from within the frame containing the face in the third image;

a first recognition system, connected to the first extractor, for comparing the face of the first sub-image with faces from a first database; and

a second recognition system, connected to the second extractor, for comparing the face of the second sub-image with faces from a second database.

3. The system of claim 2, wherein:

the first recognition system is for providing an approximate match of the face of the first sub-image with face-related information from the first database; and

the second recognition system is for providing an approximate match of the face of the second sub-image with face-related information from the second database.

4. The system of claim 3, further comprising:

a first evaluator, connected to the first extractor, for evaluating the match of the face of the first sub-image with face-related information from the first database; and

a second evaluator, connected to the second extractor, for evaluating the match of the face of the second sub-image with face-related information from the second database.

5. The system of claim 4, wherein:

the first evaluator is for providing a first confidence level of recognition of the face of the first sub-image; and

the second evaluator is for providing a second confidence level of recognition of the face of the second sub-image.

6. The system of claim 5, further comprising:

a confidence levels combiner, connected to the first and second evaluator, for combining the first confidence level and the second confidence level; and

wherein the confidence levels combiner is for outputting a resultant confidence level of recognition of the face of the first and second sub-images.

7. The system of claim 6, further comprising a light source for providing infrared illumination for the first, second and third detector arrays, as needed.

8. The system of claim 7, wherein:

the lower reflective infrared band is approximately between 800 and 1400 nanometers; and

the upper reflective infrared band is approximately between 1400 and 2200 nanometers.

9. A human detection system comprising:

a multi-band camera having first, second and third outputs;

a subtractor connected to the first and second outputs;

a thresholder connected to the subtractor;

a feature selector connected to the thresholder;

a framer connected to the thresholder;

a first pixel integrator connected to the framer and the first output;

a second pixel integrator connected to the framer and the third output;

a first extractor connected to the first integrator;

a second extractor connected to the second integrator;

a first recognition system connected to the first extractor; and

a second recognition system connected to the second extractor.

10. The system of claim 9, further comprising:

a first evaluator connected to the first recognition system; and

a second evaluator connected to the second recognition system.

11. The system of claim 10, wherein:

the first output of the multi-band camera has a bandwidth in the lower reflective region of the infrared spectrum;

the second output of the multi-band camera has a bandwidth in the upper reflective region of the infrared spectrum; and

the third output of the multi-band camera has a bandwidth in the visible region of the spectrum.

12. The system of claim 11, wherein the subtractor may remove pixels from an image which are not representative of skin.

13. The system of claim 12, wherein:

the feature selector seeks out features of a face in the image;

the feature selector may retain only the face in the image; and

the framer may output a framed image of the face.

14. The system of claim 13, wherein:

the first pixel integrator may match the pixels of the framed image with an image from the first output of the multi-band camera;

the first pixel integrator may superimpose the frame on the image from the first output of the multi-band camera;

the second pixel integrator may match the pixels of the framed image with an image from the third output of the multi-band camera; and

the second pixel integrator may superimpose the frame on the image from the third output of the multi-band camera.

15. The system of claim 14, wherein:

the first extractor may extract a portion of the image from the first output within the frame; and

the second extractor may extract a portion of the image from the third output within the frame.

16. The system of claim 15, wherein:

the portion of the image from the first output may be sent to the first recognition system;

the first recognition system may perform a match of the portion of the image from the first output with a similar image from a data system;

the portion of the image from the third output may be sent to the second recognition system;

the second recognition system may perform a match of the portion of the image from the third output with a similar image from a data system.

17. The system of claim 16, wherein:

the first evaluator may provide a first confidence level of recognition of the match of the portion of the image from the first output with a similar image from a data system; and

the second evaluator may provide a second confidence level of recognition of the match of the portion of the image from the third output with a similar image from a data system.

18. The system of claim 17, further comprising a confidence level integrator for providing a resultant confidence level based on the first and second confidence levels of recognition.

19. The system of claim 18, further comprising a light source for providing infrared lighting as needed.

20. A detection system comprising:

a first array of detector pixels having a field of view of a scene;

a second array of detector pixels having the field of view;

a third array of detector pixels having the field of view;

a subtractor connected to the first and second arrays;

a comparator connected to the subtractor; and

wherein;

the first array of detector pixels has a sensitivity to light in a lower portion of a reflective infrared spectrum;

the second array of detector pixels has a sensitivity to light in an upper portion of a reflective infrared spectrum;

the third array of detector pixels has a sensitivity to light in a visible spectrum;

a pixel trio includes a first pixel at a location of the first array of detector pixels which has a one-to-one correlation with a second pixel of the pixel trio at a corresponding location of the second array of detector pixels, and which has a one-to-one correlation with a third pixel of the pixel trio at a corresponding location of the third array of detector pixels;

the subtractor is for detecting differences of magnitudes of output signals from the first and second pixels of each pixel trio and outputting a magnitude difference signal of the first and second pixels for each pixel trio; and

the comparator is for comparing the magnitude difference signals to a certain threshold value and selecting the pixel trios with the first and second pixels having magnitude difference values on one side of the threshold value.

21. The system of claim 20, wherein:

the pixel trios with the first and second pixels having a magnitude difference value on one side of the threshold value reveal low infrared light reflectance areas in the scene; and

the low reflectance areas may form an image having the field of view and a pixel layout like that of the first, second and third arrays of detector pixels.

22. The system of claim 21, further comprising:

a face feature selector; and

wherein the feature selector may locate features of a face in the image and detect the face.

23. The system of claim 22, further comprising:

a framer; and

wherein the framer may place a frame around the face in the image;

pixels within the frame may be matched with pixels at corresponding locations in an image from the first array of detector pixels and an image from the third array of pixels.

24. The system of claim 23, further comprising:

an extractor; and

wherein the extractor may extract a first sub-image containing the face within the frame in the image from the first array of detector pixels and a second sub-image containing the face within the frame in the image from the third array of detector pixels.

25. The system of claim 24, further comprising:

a first face recognition system; and

a second face recognition system; and

wherein;

the first face recognition system may match the first sub-image of the face with another sub-image of an identified face; and

the second face recognition system may match the second sub-image of the face with another sub-image of an identified face.

26. The system of claim 25, further comprising:

an evaluator; and

wherein;

the evaluator may indicate a first confidence level of recognition of the face in the first sub-image; and

the evaluator may indicate a second confidence level of recognition of the face in the second sub-image.

27. The system of claim 26, further comprising:

a resultant confidence indicator; and

wherein the resultant confidence indicator may provide a combined confidence level of recognition of the face in the first and second sub-images.

28. The system of claim 27, further comprising a database of sub-images of identified faces, connected to the first and second face recognition systems.

29. The system of claim 28, further comprising a light source for providing infrared light on the scene.

30. The system of claim 29, further comprising a processor for regulating the source for providing light on the scene as needed.

31. A method for detecting a face in a scene, comprising:

capturing a first image the scene with a first detector array sensitive to a first band of reflective infrared light;

capturing a second image of the scene with a second detector array sensitive to a second band of reflective infrared light; and

capturing a third image of the scene with a third detector array sensitive to a band of visible light; and

wherein;

the first image comprises m×

n pixels;

the second image comprises m×

n pixels;

the third image comprises m×

n pixels;

the first, second and third images have the same field of view of the scene;

each pixel P_(i,j)1 of the first image is located at an ith row and a jth column;

each pixel P_(i,j)2 of the second image is located at an ith row and a jth column;

each pixel P_(i,j)3 of the third image is located at an ith row and a jth column;

1≦

i≦

m;

1≦

j≦

n;

each pixel P_(i,j)1 of the first image has a counterpart pixel P_(i,j)2 in the second image in a one-to-one mapping of the first image to the second image;

each pixel P_(i,j)2 of the second image has a counterpart pixel P_(i,j)3 in the third image in a one-to-one mapping of the second image to the third image;

a difference intensity pixel Δ

I_P(i,j) may be determined from an intensity of each pixel I_P(i,j)1 of the first image and each counterpart pixel I_P(i,j) of the second image, for all of the pixels of the first and second images;

each difference intensity pixel Δ

I_P(i,j) has an intensity value that may be measured against an intensity threshold value and has its value nullified if it does not meet a certain range of intensity values relative to the intensity threshold value, and becomes a thresholded difference intensity pixel TΔ

I_P(i,j);

each thresholded difference intensity pixel together with each other thresholded difference intensity pixel forms a fourth image having a one-to-one mapping of pixels to the first and second images; and

features of a face are sought, and if found, a face is detected in the fourth image and a frame is scribed as frame pixels in the fourth image around the face.

32. The method of claim 31, further comprising:

copying the frame pixels from the fourth image to the first image and the third image on a one-to-one mapping basis;

extracting the pixels within the frame in the first image to form a fifth image having the face extracted from the first image; and

extracting the pixels within the frame in the third image to form a sixth image having the face extracted from the third image.

33. The method of claim 32, further comprising:

comparing the face in the fifth image with faces from a first database;

comparing the face in the sixth image with faces from a second database;

matching the face in the fifth image with a face from the first database; and

matching the face in the sixth image with a face from the second database.

34. The method of claim 33, further comprising:

evaluating the matching of the face in the fifth image with the face from the first database;

determining a first confidence level of recognition of the face in the fifth image relative to the face from the first database;

evaluating the matching of the face in the sixth image with the face from the second database; and

determining a second confidence level of recognition of the face in the sixth image relative to the face from the second database.

35. The method of claim 34, further comprising determining an overall confidence of recognition of the face in the scene from the first and second confidence levels of recognition.

36. The method of claim 35, further comprising shining infrared light on the scene as needed for obtaining adequate first and second images of the scene.

37. The method of claim 36, wherein:

the first band of reflective infrared light is approximately between 800 and 1400 nanometers; and

the second band of reflective infrared light is approximately between 1400 and 2200 nanometers.

38. The method of claim 37, wherein the first and second databases are the same database.

39. An apparatus for detecting faces, comprising:

means for obtaining a first image with a first field of view of a scene at a lower reflective infrared band of light;

means for obtaining a second image with the first field of view of the scene at an upper reflective infrared band of light;

means for obtaining a third image with the first field of view of the scene at a visible band of light;

means for subtractively combining the first and second images into a fourth image;

means for thresholding the pixels of the fourth image to result in a fifth image having only the pixels meeting a set threshold value;

means for finding features of a face in the fifth image;

means for inserting a frame around the face based on the features;

means for transferring the frame to be around the face in the first image; and

means for transferring the frame to be around the face in the third image.

40. The means of claim 39, further comprising:

means for extracting a first sub-image of the face within the frame in the first image;

means for extracting a second sub-image of the face within the frame in the third image;

means for possibly at least approximately matching the face in the first sub-image with at least one other face in a first database; and

means for possibly at least approximately matching the face in the second sub-image with at least one other face in a second database.

41. The means of claim 40, further comprising:

means for evaluating the matching the face in the first sub-image with the at least one other face in the first database;

means for evaluating the matching the face in the second sub-image with the at least one other face in the second database;

means for determining a first confidence level of recognition of the face in the first sub-image; and

means for determining a second confidence level of recognition of the face in the second sub-image.

42. The means of claim 41, further comprising means for determining an overall confidence level of recognition from the first and second levels of recognition.

43. The means of claim 42, further comprising means for providing infrared lighting on the scene as needed for obtaining the first and second images.

44. The means of claim 43, wherein:

the lower reflective infrared band of light is approximately between 800 and 1400 nanometers; and

the upper reflective infrared band of light approximately between 1400 and 2200 nanometers.

45. The means of claim 44, wherein the first and second databases are the same database.