Real-time face tracking in a digital image acquisition device

• US 8,050,465 B2
• Filed: 07/03/2008
• Issued: 11/01/2011
• Est. Priority Date: 08/11/2006
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: DigitalOptics Corporation Europe Limited (Adeia Inc.)
• Inventors: Ianculescu, Mihai, Steinberg, Eran, Petrescu, Stefan, Gangea, Mihnea, Bigioi, Petronel, Corcoran, Peter
• Primary Examiner(s): Alavi; Amir
• Application Number:
  US12/167,500
• Publication Number:
  US 20080267461A1
• Time in Patent Office: 1,216 Days
• Field of Search: 382/103, 382115-118, 382/164, 382/173, 382/181, 382/190, 382/218, 382/224, 382/254, 382/284, 382/298, 382/299, 382/305, 358/538
• US Class Current: 382/118
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 18/2148 characterised by the process organisation or structure, e.g. boosting cascade
• CPC Class Codes: G06V 30/19147 Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 30/2504 Coarse or fine approaches, e.g. resolution of ambiguities or multiscale approaches
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: H04N 1/2112 using still video cameras
• CPC Class Codes: H04N 1/32112 in a separate computer file, document page or paper sheet, e.g. a fax cover sheet H04N1/32122 takes precedence
• CPC Class Codes: H04N 1/32128 attached to the image data, e.g. file header, transmitted message header, information on the same page or in the same computer file as the image for information embedded in the image H04N1/32144
• CPC Class Codes: H04N 2101/00 Still video cameras
• CPC Class Codes: H04N 2201/0084 Digital still camera
• CPC Class Codes: H04N 2201/3205 of identification information, e.g. name or ID code H04N2201/3209 takes precedence
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body

Abstract

A database includes an identifier and associated parameters for each of a number of faces to be recognized. A new acquired image from an image stream is received potentially including one or more face regions. Face detection is applied to at least a portion of the acquired image to provide a set of candidate face regions each having a given size and a respective location. Using the database, face recognition is selectively applied to at least one of the candidate face regions to provide an identifier for a face recognized in a candidate face region. A portion of the image is stored including the recognized face in association with at least one image of the image stream.

Claims

1. A method of recognizing faces in an image stream with a digital image acquisition device, comprising:

using a processor to perform the following;

providing a database comprising an identifier and associated parameters for each of a number of one or more faces to be recognized;

receiving a new acquired image from an image stream;

obtaining a location of one or more tracked face regions within said image;

determining if said one or more tracked face regions is optimized for accurate face recognition;

selectively applying face recognition using said database to at least one of said one or more tracked face regions to determine a first identifier for said at least one of said one or more tracked face regions; and

repeating the method one or more times to determine additional identifiers associated with the one or more tracked face regions.

2. The method of claim 1, further comprising identifying one or more of said tracked face regions based on at least a first identifier and a second identical identifier determined for the same face region.

3. The method of claim 1, further comprising failing to identify one or more tracked face regions based on at least a first identifier and a second different identifier determined for the same face region.

4. The method of claim 1 where said determining if said tracked face region is optimized for accurate face recognition includes determining if a face is substantially frontal in aspect, or if the vertical or horizontal rotation, or both, of the face lie within a predetermined range of angles, or if the mean and variance of luminance for a face region lie within predetermined ranges, or combinations thereof.

5. The method of claim 4 wherein the parameters associated with a particular identifier are functions of the vertical or horizontal rotation, or both, or are functions of the mean or variance, or both, of luminance of a tracked face region, or combinations thereof.

6. A method of tracking faces in an image stream with a digital image acquisition device, comprising:

using a processor to perform the following;

receiving an acquired image from an image stream including one or more face regions;

calculating a corresponding integral image for at least a portion of said image;

applying face detection to at least a portion of said integral image to provide a set of one or more candidate face regions each having a given size and a respective location;

selectively applying face recognition using said database to at least one of said one or more candidate face regions to provide an identifier for a face recognized in a candidate face region; and

storing a portion of said image including the recognized face in association with at least one image of said image stream.

7. The method of claim 6, further comprising sub-sampling said acquired image at a specified resolution to provide a sub-sampled image.

8. The method of claim 7, wherein said calculating said corresponding integral image comprises utilizing said sub-sampled image.

9. The method of claim 6, further comprising merging said set of candidate face regions with one or more previously detected face regions to provide a merged set of candidate face regions of different sizes or locations or both.

10. The method of claim 9, wherein said merging comprises merging a set of candidate face regions for an acquired image with said set of confirmed face regions for a previously acquired image.

11. The method of claim 9, wherein said regions of said acquired image corresponding to a region of said merged set of candidate face regions comprise regions surrounding respective regions of said merged set of candidate face regions.

12. The apparatus of claim 9, wherein said regions of said acquired image corresponding to a region of said merged set of candidate face regions comprise regions surrounding respective regions of said merged set of candidate face regions.

13. The method of claim 6, further comprising, for at least one candidate face region associated with a previous image, enhancing the contrast of the luminance characteristics of the corresponding region of said acquired image.

14. The method of claim 6, wherein said face detection is performed with relaxed face detection parameters.

15. The method of claim 6, wherein said selectively applying face recognition includes selecting only one or more candidate face regions having a frontal alignment for face recognition.

16. The method of claim 6, further comprising selecting only one or more candidate face regions including two eye regions and a mouth region within a given area of the candidate face region.

17. The method of claim 6, further comprising:

providing a level of confidence for each candidate face region based on said face detection;

designating a candidate face region with a first level of confidence as a face region; and

applying recognition only to candidate face regions having a second confidence level higher than said first level of confidence.

18. The method of claim 6, further comprising:

applying an Active Appearance Model (AAM) to each candidate face region; and

applying face recognition only to one or more candidate face regions having AAM parameters indicating said candidate face region has a vertical and a horizontal rotation within a pre-determined range.

19. The method of claim 6, comprising:

calculating an average luminance for said each candidate face region; and

applying face recognition only to one or more candidate face region having average luminance within a pre-determined range.

20. The method of claim 6, wherein said new acquired image comprises a low resolution preview image and wherein said portion of said image including a recognized face is stored in association with a main high resolution image.

21. The method of claim 6, further comprising tracking candidate face regions of different sizes from a plurality of images of said image stream.

22. The method of claim 6, wherein said applying fixed size face detection comprises applying a cascade of Haar classifiers of a fixed size to said integral image.

23. The method of claim 6, further comprising responsive to said acquired image being captured with a flash, analyzing one or more regions of said acquired image corresponding to said tracked candidate face regions for red-eye defect.

24. An image acquisition and processing apparatus including a lens and an image sensor for acquiring digital images, and one or more processors and one or more digital storage media having digitally-encoded instructions embedded therein for programming the one or more processors to perform a method of recognizing faces in an image stream, the method comprising:

providing a database comprising an identifier and associated parameters for each of a number of one or more faces to be recognized;

receiving a new image from an image stream;

obtaining a location of one or more tracked face region within said image;

determining if said tracked face region is optimized for accurate face recognition;

selectively applying face recognition to at least one of said one or more tracked face regions to determine a first identifier for said face region; and

repeating the method one or more times to determine additional identifiers associated with the one or more tracked face regions.

25. The apparatus of claim 24, wherein the method further comprises identifying one or more of said tracked face regions based on at least a first identifier and a second identical identifier determined for the same face region.

26. The apparatus of claim 24, wherein the method further comprises failing to identify one or more tracked face regions based on at least a first identifier and a second different identifier determined for the same face region.

27. The apparatus of claim 24 where said determining if said tracked face region is optimized for accurate face recognition includes determining if a face is substantially frontal in aspect, or if the vertical or horizontal rotation, or both, of the face lie within a predetermined range of angles, or if the mean and variance of luminance for a face region lie within predetermined ranges, or combinations thereof.

28. The apparatus of claim 27 wherein the parameters associated with a particular identifier are functions of the vertical or horizontal rotation, or both, or are functions of the mean or variance, or both, of luminance of a tracked face region, or combinations thereof.

29. An image acquisition and processing apparatus including a lens and an image sensor for acquiring digital images, and one or more processors and one or more digital storage media having digitally-encoded instructions embedded therein for programming the one or more processors to perform a method of recognizing faces in an image stream, the method comprising:

receiving an acquired image from an image stream including one or more face regions;

calculating a corresponding integral image for at least a portion of said image;

applying face detection to at least a portion of said integral image to provide a set of one or more candidate face regions each having a given size and a respective location;

selectively applying face recognition using said database to at least one of said one or more candidate face regions to provide an identifier for a face recognized in a candidate face region; and

storing a portion of said image including the recognized face in association with at least one image of said image stream.

30. The apparatus of claim 29, wherein the method further comprises sub-sampling said acquired image at a specified resolution to provide a sub-sampled image.

31. The apparatus of claim 30, wherein said calculating said corresponding integral image comprises utilizing said sub-sampled image.

32. The apparatus of claim 29, wherein the method further comprises merging said set of candidate face regions with one or more previously detected face regions to provide a merged set of candidate face regions of different sizes or locations or both.

33. The apparatus of claim 32, wherein said merging comprises merging a set of candidate face regions for an acquired image with said set of confirmed face regions for a previously acquired image.

34. The apparatus of claim 29, wherein the method further comprises, for at least one candidate face region associated with a previous image, enhancing the contrast of the luminance characteristics of the corresponding region of said acquired image.

35. The apparatus of claim 29, wherein said face detection is performed with relaxed face detection parameters.

36. The apparatus of claim 29, wherein said selectively applying face recognition includes selecting only one or more candidate face regions having a frontal alignment for face recognition.

37. The apparatus of claim 29, wherein the method further comprises selecting only one or more candidate face regions including two eye regions and a mouth region within a given area of the candidate face region.

38. The apparatus of claim 29, wherein the method further comprises:

providing a level of confidence for each candidate face region based on said face detection;

designating a candidate face region with a first level of confidence as a face region; and

applying recognition only to candidate face regions having a second confidence level higher than said first level of confidence.

39. The apparatus of claim 29, wherein the method further comprises:

applying an Active Appearance Model (AAM) to each candidate face region; and

applying face recognition only to one or more candidate face regions having AAM parameters indicating said candidate face region has a vertical and a horizontal rotation within a pre-determined range.

40. The apparatus of claim 29, wherein the method further comprises:

calculating an average luminance for said each candidate face region; and

applying face recognition only to one or more candidate face region having average luminance within a pre-determined range.

41. The apparatus of claim 29, wherein said new acquired image comprises a low resolution preview image and wherein said portion of said image including a recognized face is stored in association with a main high resolution image.

42. The apparatus of claim 29, wherein the method further comprises tracking candidate face regions of different sizes from a plurality of images of said image stream.

43. The apparatus of claim 29, wherein said applying fixed size face detection comprises applying a cascade of Haar classifiers of a fixed size to said integral image.

44. The apparatus of claim 29, wherein the method further comprises responsive to said acquired image being captured with a flash, analyzing one or more regions of said acquired image corresponding to said tracked candidate face regions for red-eye defect.

45. One or more non-transitory, computer readable media having digitally-encoded instructions embedded therein for programming one or more processors to perform a method of recognizing faces in an image stream, the method comprising:

providing a database comprising an identifier and associated parameters for each of a number of one or more faces to be recognized;

receiving a new image from an image stream;

obtaining a location of one or more tracked face region within said image;

determining if said tracked face region is optimized for accurate face recognition;

selectively applying face recognition to at least one of said one or more tracked face regions to determine a first identifier for said face region; and

repeating the method one or more times to determine additional identifiers associated with the one or more tracked face regions.

46. The one or more processor readable storage devices of claim 45, wherein the method further comprises identifying one or more of said tracked face regions based on at least a first identifier and a second identical identifier determined for the same face region.

47. The one or more processor readable storage devices of claim 45, wherein the method further comprises failing to identify one or more tracked face regions based on at least a first identifier and a second different identifier determined for the same face region.

48. The one or more processor readable storage devices of claim 45 where said determining if said tracked face region is optimized for accurate face recognition includes determining if a face is substantially frontal in aspect, or if the vertical or horizontal rotation, or both, of the face lie within a predetermined range of angles, or if the mean and variance of luminance for a face region lie within predetermined ranges, or combinations thereof.

49. The one or more processor readable storage devices of claim 48 wherein the parameters associated with a particular identifier are functions of the vertical or horizontal rotation, or both, or are functions of the mean or variance, or both, of luminance of a tracked face region, or combinations thereof.

50. One or more non-transitory, computer readable media having digitally-encoded instructions embedded therein for programming one or more processors to perform a method of tracking and recognizing faces in an image stream, the method comprising:

receiving an acquired image from an image stream including one or more face regions;

calculating a corresponding integral image for at least a portion of said image;

applying face detection to at least a portion of said integral image to provide a set of one or more candidate face regions each having a given size and a respective location;

selectively applying face recognition using said database to at least one of said one or more candidate face regions to provide an identifier for a face recognized in a candidate face region; and

storing a portion of said image including the recognized face in association with at least one image of said image stream.

51. The one or more processor readable storage devices of claim 50, wherein the method further comprises sub-sampling said acquired image at a specified resolution to provide a sub-sampled image.

52. The one or more processor readable storage devices of claim 51, wherein said calculating said corresponding integral image comprises utilizing said sub-sampled image.

53. The one or more processor readable storage devices of claim 50, wherein the method further comprises merging said set of candidate face regions with one or more previously detected face regions to provide a merged set of candidate face regions of different sizes or locations or both.

54. The one or more processor readable storage devices of claim 53, wherein said merging comprises merging a set of candidate face regions for an acquired image with said set of confirmed face regions for a previously acquired image.

55. The one or more processor readable storage devices of claim 53, wherein said regions of said acquired image corresponding to a region of said merged set of candidate face regions comprise regions surrounding respective regions of said merged set of candidate face regions.

56. The one or more processor readable storage devices of claim 50, wherein the method further comprises, for at least one candidate face region associated with a previous image, enhancing the contrast of the luminance characteristics of the corresponding region of said acquired image.

57. The one or more processor readable storage devices of claim 50, wherein said face detection is performed with relaxed face detection parameters.

58. The one or more processor readable storage devices of claim 50, wherein said selectively applying face recognition includes selecting only one or more candidate face regions having a frontal alignment for face recognition.

59. The one or more processor readable storage devices of claim 50, wherein the method further comprises selecting only one or more candidate face regions including two eye regions and a mouth region within a given area of the candidate face region.

60. The one or more processor readable storage devices of claim 50, wherein the method further comprises:

providing a level of confidence for each candidate face region based on said face detection;

designating a candidate face region with a first level of confidence as a face region; and

applying recognition only to candidate face regions having a second confidence level higher than said first level of confidence.

61. The one or more processor readable storage devices of claim 50, wherein the method further comprises:

applying an Active Appearance Model (AAM) to each candidate face region; and

applying face recognition only to one or more candidate face regions having AAM parameters indicating said candidate face region has a vertical and a horizontal rotation within a pre-determined range.

62. The one or more processor readable storage devices of claim 50, wherein the method further comprises:

calculating an average luminance for said each candidate face region; and

applying face recognition only to one or more candidate face region having average luminance within a pre-determined range.

63. The one or more processor readable storage devices of claim 50, wherein said new acquired image comprises a low resolution preview image and wherein said portion of said image including a recognized face is stored in association with a main high resolution image.

64. The one or more processor readable storage devices of claim 50, wherein the method further comprises tracking candidate face regions of different sizes from a plurality of images of said image stream.

65. The one or more processor readable storage devices of claim 50, wherein said applying fixed size face detection comprises applying a cascade of Haar classifiers of a fixed size to said integral image.

66. The one or more processor readable storage devices of claim 50, wherein the method further comprises responsive to said acquired image being captured with a flash, analyzing one or more regions of said acquired image corresponding to said tracked candidate face regions for red-eye defect.