Real-time facial recognition and verification system

• US 20020136448A1
• Filed: 08/16/2001
• Published: 09/26/2002
• Est. Priority Date: 07/20/1998
• Status: Active Grant

• Current Assignee: L-1 Identity Solutions Operating Co. (Advent International Corporation)
• Original Assignee: LAU Technologies
• Inventors: Saulnier, Michael S., Cusack, Francis J. Jr., Ehn, Dennis C., Kuzeja, Thomas M., Bortolussi, Jay F.
• Granted Patent: US 6,681,032 B2
• Time in Patent Office: Days
• US Class Current: 382/164
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/169 Holistic features and representations, i.e. based on the facial image taken as a whole
• CPC Class Codes: G06V 40/19 Sensors therefor

Abstract

A system and method for acquiring, processing, and comparing an image with a stored image to determine if a match exists. In particular, the system refines the image data associated with an object based on pre-stored color values, such as flesh tone color. The system includes a storage element for storing flesh tone colors of a plurality of people, and a defining stage for localizing a region of interest in the image. A combination stage combines the unrefined region of interest with one or more pre-stored flesh tone colors to refine the region of interest based on color. This flesh tone color matching ensures that at least a portion of the image corresponding to the unrefined region of interest having flesh tone color is incorporated into the refined region of interest. Hence, the system can localize the head, based on the flesh tone color of the skin of the face in a rapid manner. According to one practice, the refined region of interest is smaller than or about equal to the unrefined region of interest.

Claims

1. A system for refining an object within an image based on color, the system comprising

a storage element for storing flesh tone colors of a plurality of people,

defining means for defining an unrefined region of interest corresponding to at least part of the object in the image, the unrefined region of interest including flesh tone colors, and

combination means for combining the unrefined region of interest with one or more of the flesh tone colors stored in the storage element to refine the region of interest to ensure that at least a portion of the image corresponding to the unrefined region of interest having flesh tone color is incorporated into the refined region of interest.

2. A system in accordance with claim 1, wherein the refined region of interest being smaller than or about equal to the unrefined region of interest.

3. A system in accordance with claim 1, further comprising an image acquisition element for acquiring the image.

4. A system in accordance with claim 1, further comprising a motion detector for detecting motion of the image within a field of view.

5. A system in accordance with claim 1, wherein the storage element comprises a look-up-table.

6. A system in accordance with claim 1, wherein the storage element comprises a memory element for storing a color histogram constructed from the plurality of people, the histogram being representative of the distribution of colors that constitute flesh tone.

7. A system in accordance with claim 1, further comprising blob means for connecting together selected pixels of the object in the image to form a selected number of blobs therein.

8. A system in accordance with claim 1, further comprising

a motion detector for detecting motion of the image within a field of view, the detected motion signifying the presence of the object within the field of view, and

blob means for connecting together a selected number of pixels of the object in the detected image to form a selected number of blobs therein, wherein one of the blobs corresponds to the unrefined region of interest.

9. A system in accordance with claim 8, wherein said unrefined region of interest corresponds to a head of a person, and the object corresponds to an eye of the person.

10. A system in accordance with claim 8, wherein the combination means is adapted to combine one of the blobs with the flesh tone colors to construct the refined region of interest.

11. A system in accordance with claim 4, wherein the motion detector means comprises

differencing means for subtracting selected pixel values associated with generally spatially adjacent images and for generating a difference value therefrom, and

threshold means for comparing the difference value with a threshold value to determine motion within the field of view.

12. A system in accordance with claim 1, further comprising

first histogram means for sampling the flesh tone colors of the plurality of people and for generating a first flesh tone color histogram, and

first transform means for transforming the first color histogram into ST color space.

13. A system in accordance with claim 8, further comprising normalizing means for normalizing the first color histogram.

14. A system in accordance with claim 12, further comprising conversion means for converting the histogram to a normalized function according to Bayes Rule.

15. A system in accordance with claim 12, wherein the flesh tone colors correspond to a face of a person within an image, the system further comprising second histogram means for generating a second color histogram not associated with the face within the image.

16. A system in accordance with claim 15, further comprising second transform means for transforming the second color histogram into ST color space.

17. A system in accordance with claim 1, further comprising

histogram means for generating a histogram of colors corresponding to at least one of a face of a person in the image and a non-face portion of the image, and

transforming means for transforming the histogram into ST color space.

18. A system in accordance with claim 1, further comprising histogram means for generating a histogram of colors corresponding to the object in the image.

19. A system in accordance with claim 1, further comprising means for adjusting the flesh tone colors stored within the storage element.

20. A system in accordance with claim 1, wherein the object corresponds to a face of a person within the image, the system further comprising erosion means for applying an erosion operation to the face to separate pixels corresponding to hair from pixels corresponding to face.

21. A system in accordance with claim 1, further comprising erosion means for applying an erosion operation to reduce the size of an object within the image, thereby reducing the size of the unrefined region of interest.

22. A system in accordance with claim 20, further comprising dilation means to expand one of the region of interests to obtain the object within the image.

24. A method in accordance with claim 23, further comprising the step of detecting motion of the image within a field of view.

25. A method in accordance with claim 23, further comprising the step of generating a color histogram from the plurality of people, the histogram being representative of the distribution of colors that constitute flesh tone.

26. A method in accordance with claim 23, further comprising the step of connecting together selected pixels of the object in the image to form a selected number of blobs therein.

27. A method in accordance with claim 23, further comprising the steps of

detecting motion of the image within a field of view, and

connecting together a selected number of pixels of the object in the detected image to form a selected number of blobs therein, wherein one of the blobs corresponds to the unrefined region of interest.

28. A method in accordance with claim 27, wherein the step of combining further comprises the step of generating the refined region of interest by combining one of the blobs with one or more of the flesh tone colors.

29. A method in accordance with claim 24, wherein the step of detecting motion comprises the steps of

subtracting selected pixel values associated with generally spatially adjacent images captured by the image acquisition element and for generating a difference value therefrom, and

comparing the difference value with a threshold value to determine whether motion is detected within the field of view.

30. A method in accordance with claim 23, further comprising the steps of

sampling the flesh tone colors of the plurality of people,

generating a first flesh tone color histogram from the sampling of flesh tone colors, and

transforming the first color histogram into ST color space.

31. A method in accordance with claim 30, further comprising the step of normalizing the first color histogram according to Bayes Rule.

32. A method in accordance with claim 23, further comprising the step of generating a color histogram of the object in the image.

33. A method in accordance with claim 23, further comprising the steps of

generating a color histogram of the object in the image, and

transforming the second color histogram into ST color space.

34. A method in accordance with claim 23, further comprising the steps of

generating a histogram of colors corresponding to at least one of a face of a person in the image and a non-face portion of the image, and

transforming the histogram into ST color space.

35. A method in accordance with claim 23, further comprising the step of adjusting the flesh tone colors stored within the storage element.

36. A method in accordance with claim 23, further comprising the step of applying an erosion operation to a face within the image to separate pixels corresponding to hair from pixels corresponding to the face.

37. A method in accordance with claim 23, further comprising the step of applying a dilation operation to expand one of the region of interests to obtain the face.

39. A facial recognition system in accordance with claim 38, further comprising a detection module for detecting a feature of the object.

40. A facial recognition system in accordance with claim 38, wherein said detection module comprises

a motion detector for detecting motion of the image within a field of view, and

blob means for connecting together a selected number of pixels of the object in the detected image to form a selected number of blobs therein, wherein one of the blobs corresponds to the unrefined region of interest.

41. A facial recognition system in accordance with claim 40, wherein said detector module further comprises selector means for selecting at least a portion of the object in the image.

42. A facial recognition system in accordance with claim 38, wherein the combination means further comprises means for constructing the region of interest by combining one of the blobs with one or more of the flesh tone colors.

43. A facial recognition system in accordance with claim 39, wherein said detection module comprises

means for generating one or more eigenheads corresponding to a set of eigenvectors generated from a reference set of images in a multi-dimensional image space,

means for scaling said eigenhead to a size generally about that of the object in the image, and

comparison means for comparing the eigenhead to the refined region of interest corresponding to the object in the image to determine whether there is a match.

44. A facial recognition system in accordance with claim 43, wherein said eigenhead is a low resolution eigenhead.

45. A facial recognition system in accordance with claim 38, further comprising location means for locating a feature of the object.

46. A facial recognition system in accordance with claim 45, wherein said location means comprises

means for representing a feature of the region of interest in the object as a plurality of eigenvectors in a multi-dimensional image space, and

correlation means for correlating the region of interest with the eigenvectors to locate the feature within the object.

47. A facial recognition system in accordance with claim 46, further comprising

a compression module for generating a set of eigenvectors of a training set of people in the multi-dimensional image space, and

projection means for projecting the feature onto the multi-dimensional image space to generate a weighted vector that represents the feature.

48. A facial recognition system in accordance with claim 47, further comprising discrimination means for comparing the weighted vector corresponding to the feature with a pre-stored vector to determine whether there is a match.

49. A facial recognition system in accordance with claim 40, wherein the motion detector comprises

differencing means for subtracting selected pixel values associated with generally spatially adjacent images captured by the image acquisition element and for generating a difference value therefrom, and

threshold means for comparing the difference value with a threshold value to determine motion within the field of view.

50. A facial recognition system in accordance with claim 38, further comprising

first histogram means for sampling the flesh tone colors of the plurality of people and for generating a first flesh tone color histogram, and

first transform means for transforming the first color histogram into ST color space.

51. A facial recognition system in accordance with claim 50, further comprising normalizing means for normalizing the first color histogram according to Bayes Rule.

52. A facial recognition system in accordance with claim 50, wherein the flesh tone colors correspond to a face of a person within an image, the system further comprising

second histogram means for generating a second color histogram not associated with the face within the image, and

second transform means for transforming the second color histogram into ST color space.

53. A facial recognition system in accordance with claim 38, further comprising

histogram means for generating a histogram of flesh tone colors corresponding to at least one of a face of a person in the image and a non-face portion of the image, and

transform means for transforming the histogram into ST color space.

54. A facial recognition system in accordance with claim 38, further comprising histogram means for generating a histogram of flesh tone colors corresponding to the object in the image.

55. A facial recognition system in accordance with claim 38, wherein the object corresponds to a face of a person within the image, the system further comprising erosion means for applying an erosion operation to the face to separate pixels corresponding to hair from pixels corresponding to face.

56. A facial recognition system in accordance with claim 38, further comprising erosion means for applying an erosion operation to the image to reduce the size of an object within the image, thereby reducing the size of the unrefined region of interest.

57. A facial recognition system in accordance with claim 38, further comprising dilation means for applying a dilation operation to expand one of the region of interests to obtain a face image corresponding to the object within the image.

58. A facial recognition system in accordance with claim 46, wherein said correlation means further comprises

means for storing a center-weighted windowing function,

means for placing the windowing function over the region of interest, and

means for analyzing the region of interest with the windowing function to locate the feature of the object.

59. A facial recognition system in accordance with claim 46, wherein said correlation means further comprises means for analyzing the region of interest with a center-weighted windowing function to locate the feature of the object.

60. A facial recognition system in accordance with claim 45, further comprising second location means for determining the location of the feature of the object when the first location module is unable to locate the feature.

61. A facial recognition system in accordance with claim 60, wherein said second location means comprises

means for representing the region of interest in the object as a plurality of eigenvectors in a multi-dimensional image space, and

correlation means for correlating the region of interest with the eigenvectors.

62. A facial recognition system in accordance with claim 61, wherein said second location module further comprises

second means for representing a feature of the region of interest in the object as a plurality of eigenvectors in a multi-dimensional image space, and

correlation means for correlating the region of interest with the eigenvectors to locate the feature in the image.

63. A facial recognition system in accordance with claim 38, further comprising means for adjusting one of the contrast and brightness of the image.

64. A facial recognition system in accordance with claim 38, further comprising means for correlating the image with a windowing function to generate a correlation map.

65. A facial recognition system in accordance with claim 38, further comprising means for determining a standard deviation and a mean of pixels that constitute the image.

23. A method of refining an object within an image based on color, the method comprising the steps of

storing flesh tone colors of a plurality of people in a storage element,

defining an unrefined region of interest corresponding to at least part of the object in the image, the unrefined region of interest including flesh tone colors, and

combining the unrefined region of interest with the flesh tone colors stored in the storage element to refine the region of interest to ensure that at least a portion of the image corresponding to the unrefined region of interest having flesh tone color is incorporated into the refined region of interest, the refined region of interest being smaller than or about equal to the unrefined region of interest.

38. A facial recognition and identification system for identifying an object in an image, comprising

an image acquisition element for acquiring the image,

defining means for defining an unrefined region of interest corresponding to at least part of the object in the image, the unrefined region of interest including flesh tone colors,

combination means for combining the unrefined region of interest with the flesh tone colors to refine the region of interest to ensure at least a portion of the image corresponding to the unrefined region of interest that includes flesh tone color are incorporated into the refined region of interest, the refined region of interest being smaller than or about equal to the unrefined region of interest, said refined region of interest corresponding at least in part to the object, and

a recognition module for determining whether the acquired object matches a pre-stored object,

whereby said system recognizes the object when the object matches the pre-stored object.