Real-time face tracking in a digital image acquisition device

• US 7,403,643 B2
• Filed: 05/24/2007
• Issued: 07/22/2008
• Est. Priority Date: 08/11/2006
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Vision Limited (Adeia Inc.)
• Inventors: Steinberg, Eran, Petrescu, Stefan, Gangea, Mihnea, Bigioi, Petronel, Corcoran, Peter, Ianculescu, Mihai
• Primary Examiner(s): Alavi; Amir
• Application Number:
  US11/753,397
• Publication Number:
  US 20080037838A1
• Time in Patent Office: 425 Days
• Field of Search: 382/103, 382/115, 382/117, 382/118, 382/164, 382/173, 382/181, 382/190, 382/218, 382/224, 382/254, 382/284, 382/298, 382/299, 382/305, 358/538
• US Class Current: 382/118
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 18/2148 characterised by the process organisation or structure, e.g. boosting cascade
• CPC Class Codes: G06V 30/19147 Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 30/2504 Coarse or fine approaches, e.g. resolution of ambiguities or multiscale approaches
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: H04N 1/2112 using still video cameras
• CPC Class Codes: H04N 1/32112 in a separate computer file, document page or paper sheet, e.g. a fax cover sheet H04N1/32122 takes precedence
• CPC Class Codes: H04N 1/32128 attached to the image data, e.g. file header, transmitted message header, information on the same page or in the same computer file as the image for information embedded in the image H04N1/32144
• CPC Class Codes: H04N 2101/00 Still video cameras
• CPC Class Codes: H04N 2201/0084 Digital still camera
• CPC Class Codes: H04N 2201/3205 of identification information, e.g. name or ID code H04N2201/3209 takes precedence
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body

Abstract

A database includes an identifier and associated parameters for each of a number of faces to be recognized. A new acquired image from an image stream is received potentially including one or more face regions. Face detection is applied to at least a portion of the acquired image to provide a set of candidate face regions each having a given size and a respective location. Using the database, face recognition is selectively applied to at least one of the candidate face regions to provide an identifier for a face recognized in a candidate face region. A portion of the image is stored including the recognized face in association with at least one image of the image stream.

Claims

1. A method of recognizing faces in an image stream using a digital image acquisition device, comprising:

a. providing a database comprising an identifier and associated parameters for each of a number of one or more faces to be recognized;

b. receiving a new acquired image from an image stream;

c. applying face detection to at least a portion of said acquired image to provide a set of one or more candidate face regions, each candidate face region having a given size and a respective location;

d. selectively applying face recognition using said database to at least one of said candidate face regions to provide an identifier for a face recognized in a candidate face region; and

e. storing a portion of said image including the recognized face in association with at least one image of said image stream.

2. A method as claimed in claim 1, wherein said selectively applying face recognition includes selecting only one or more candidate face regions having a frontal alignment for face recognition.

3. A method as claimed in claim 2 comprising selecting only one or more candidate face regions including two eye regions and a mouth region within a given area of the candidate face region.

4. A method as claimed in claim 2, comprising:

providing a level of confidence for each candidate face region based on said face detection;

designating a candidate face region with a first level of confidence as a face region; and

applying recognition only to candidate face regions having a second confidence level higher than said first level of confidence.

5. A method as claimed in claim 2, comprising:

applying an Active Appearance Model (AAM) to each candidate face region; and

applying face recognition only to one or more candidate face regions having AAM parameters indicating said candidate face region has a vertical and a horizontal rotation within a pre-determined range.

6. A method as claimed in claim 2, wherein said face detection comprises:

applying a plurality of chain classifiers to said main image, one of said classifiers comprising a frontal classifier chain; and

applying face recognition only to candidate face regions detected with said frontal classifier chain.

7. A method as claimed in claim 2, comprising:

calculating an average luminance for said each candidate face region; and

applying face recognition only to one or more candidate face region having average luminance within a pre-determined range.

8. A method as claimed in claim 1, wherein said new acquired image is a low resolution preview image and wherein said portion of said image including a recognized face is stored in association with a main high resolution image.

9. An image processing apparatus including one or more processors and one or more digital storage media having digitally-encoded instructions embedded therein for programming the one or more processors to perform an iterative method of recognizing faces in an image stream, the method comprising:

a. providing a database comprising an identifier and associated parameters for each of a number of one or more faces to be recognized;

b. receiving a new acquired image from an image stream;

c. applying face detection to at least a portion of said acquired image to provide a set of one or more candidate face regions, each candidate face region having a given size and a respective location;

d. selectively applying face recognition using said database to at least one of said candidate face regions to provide an identifier for a face recognized in a candidate face region; and

e. storing a portion of said image including the recognized face in association with at least one image of said image stream.

10. An apparatus as claimed in claim 9, wherein said selectively applying face recognition includes selecting only one or more candidate face regions having a frontal alignment for face recognition.

11. An apparatus as claimed in claim 10, wherein the method further comprises selecting only one or more candidate face regions including two eye regions and a mouth region within a given area of the candidate face region.

12. An apparatus as claimed in claim 10, wherein the method further comprises:

providing a level of confidence for each candidate face region based on said face detection;

designating a candidate face region with a first level of confidence as a face region; and

applying recognition only to candidate face regions having a second confidence level higher than said first level of confidence.

13. An apparatus as claimed in claim 10, wherein the method further comprises:

applying an Active Appearance Model (AAM) to each candidate face region; and

applying face recognition only to one or more candidate face regions having AAM parameters indicating said candidate face region has a vertical and a horizontal rotation within a pre-determined range.

14. An apparatus as claimed in claim 10 wherein said face detection comprises:

applying a plurality of chain classifiers to said main image, one of said classifiers comprising a frontal classifier chain; and

applying face recognition only to candidate face regions detected with said frontal classifier chain.

15. An apparatus as claimed in claim 10, wherein the method further comprises:

calculating an average luminance for said each candidate face region; and

applying face recognition only to one or more candidate face region having average luminance within a pre-determined range.

16. An apparatus as claimed in claim 9, wherein said new acquired image is a low resolution preview image and wherein said portion of said image including a recognized face is stored in association with a main high resolution image.

17. A method of tracking faces in an image stream with a digital image acquisition device, comprising:

a. receiving an acquired image from an image stream including one or more face regions;

b. sub-sampling said acquired image at a specified resolution to provide a sub-sampled image;

c. calculating a corresponding integral image for a least a portion of said sub-sampled image;

d. applying face detection to at least a portion of said integral image to provide a set of one or more candidate face regions each having a given size and a respective location;

e. merging said set of candidate face regions with one or more previously detected face regions to provide a set of candidate face regions of different sizes or locations or both.

18. A method as claimed in claim 17, further comprising tracking candidate face regions of different sizes from a plurality of images of said image stream.

19. A method as claimed in claim 17, further comprising, for each region of said acquired image corresponding to a region of said merged set of candidate face regions:

calculating an integral image; and

applying variable-sized face detection to each merged region integral image to provide a set of confirmed face regions and a set of rejected face regions.

20. A method as claimed in claim 19, further comprising:

checking a rejected face region based on alternative criteria from said fixed and variable sized face detection; and

responsive to said checking, indicating that the rejected face region is a face region, and adding said previously rejected face region to said set of confirmed face regions.

21. A method as claimed in claim 20, wherein said checking comprises applying a skin prototype to a rejected face region.

22. A method as claimed in claim 19, wherein said merging comprises merging a set of candidate face regions for an acquired image with said set of confirmed face regions for a previously acquired image.

23. A method as claimed in claim 19, wherein said regions of said acquired image corresponding to a region of said merged set of candidate face regions comprise regions surrounding respective regions of said merged set of candidate face regions.

24. A method as claimed in claim 19, wherein applying variable sized face detection comprises applying a plurality of cascades of Haar classifiers of varying size to each merged region integral image.

25. A method as claimed in claim 24, comprising applying twelve or fewer sizes of cascades of Haar classifiers.

26. The method of claim 24, comprising applying 3 or 4 sizes of Haar classifiers.

27. A method as claimed in claim 26, wherein each cascade comprises 32 classifiers.

28. A method as claimed in claim 17, wherein said applying fixed size face detection comprises applying a cascade of Haar classifiers of a fixed size to said integral image.

29. A method as claimed in claim 17, further comprising responsive to said acquired image being captured with a flash, analyzing one or more regions of said acquired image corresponding to said tracked candidate face regions for red-eye defect.

30. A method as claimed in claim 29, further comprising correcting in said acquired image a red-eye defect.

31. A method as claimed in claim 29, further comprising storing with said acquired image an indication of a red-eye defect.

32. A method of detecting faces in an image stream using a digital image acquisition device, comprising:

a. receiving an acquired image from said image stream including one or more face regions;

b. sub-sampling said acquired image at a specified resolution to provide a sub-sampled image;

c. identifying one or more regions of said acquired image as candidate face regions;

d. calculating a corresponding integral image for a least one of said candidate face regions of said sub-sampled acquired image;

e. applying face detection to at least a portion of said integral image to provide a set of one or more candidate face regions each having a given size and a respective location; and

f. for at least one candidate face region associated with a previous image, enhancing the contrast of the luminance characteristics of the corresponding region of said acquired image.

33. A method as claimed in claim 32, wherein said identifying is performed on said sub-sampled image.

34. A method as claimed in claim 32, wherein said face detection is performed with relaxed face detection parameters.

35. A method as claimed in claim 32, wherein said enhancing is performed on said sub-sampled image.

36. A method as claimed in claim 32, wherein said enhancing is performed during calculation of said integral image.

37. A method as claimed in claim 32, wherein in a face detection mode of said digital image acquisition device, each new acquired image is acquired with progressively increased exposure parameters until at least one candidate face region is detected.

38. An image processing apparatus including one or more processors and one or more digital storage media having digitally-encoded instructions embedded therein for programming the one or more processors to perform an iterative method of tracking faces in an image stream with a digital image acquisition device, wherein the method comprises:

a. receiving an acquired image from an image stream including one or more face regions;

b. sub-sampling said acquired image at a specified resolution to provide a sub-sampled image;

c. calculating a corresponding integral image for a least a portion of said sub-sampled image;

d. applying face detection to at least a portion of said integral image to provide a set of one or more candidate face regions each having a given size and a respective location;

e. merging said set of candidate face regions with one or more previously detected face regions to provide a set of candidate face regions of different sizes or locations or both.

39. An apparatus as claimed in claim 38, wherein the method further comprises tracking candidate face regions of different sizes from a plurality of images of said image stream.

40. An apparatus as claimed in claim 38, wherein the method further comprises, for each region of said acquired image corresponding to a region of said merged set of candidate face regions:

calculating an integral image; and

applying variable-sized face detection to each merged region integral image to provide a set of confirmed face regions and a set of rejected face regions.

41. An apparatus as claimed in claim 40, wherein the method further comprises:

checking a rejected face region based on alternative criteria from said fixed and variable sized face detection; and

responsive to said checking indicating that the rejected face region is a face region, and adding said previously rejected face region to said set of confirmed face regions.

42. An apparatus as claimed in claim 41, wherein said checking comprises applying a skin prototype to a rejected face region.

43. An apparatus as claimed in claim 40, wherein said merging comprises merging a set of candidate face regions for an acquired image with said set of confirmed face regions for a previously acquired image.

44. An apparatus as claimed in claim 38, wherein said regions of said acquired image corresponding to a region of said merged set of candidate face regions comprise regions surrounding respective regions of said merged set of candidate face regions.

45. An apparatus as claimed in claim 38, wherein applying variable sized face detection comprises applying a plurality of cascades of Haar classifiers of varying size to each merged region integral image.

46. An apparatus as claimed in claim 45, wherein the method further comprises applying twelve or fewer sizes of cascades of Haar classifiers.

47. An apparatus as claimed in claim 45, wherein the method further comprises applying 3 or 4 sizes of Haar classifiers.

48. An apparatus as claimed in claim 47, wherein each cascade comprises 32 classifiers.

49. An apparatus as claimed in claim 38, wherein said applying fixed size face detection comprises applying a cascade of Haar classifiers of a fixed size to said integral image.

50. An apparatus as claimed in claim 38, wherein the method further comprises responsive to said acquired image being captured with a flash, analyzing regions of said acquired image corresponding to said tracked candidate face regions for red-eye defects.

51. An apparatus as claimed in claim 38, wherein the method further comprises correcting in said acquired image a red-eye defect.

52. An apparatus as claimed in claim 51, wherein the method further comprises storing with said acquired image an indication of a red-eye defect.

53. An image processing apparatus including one or more processors and one or more digital storage media having digitally-encoded instructions embedded therein for programming the one or more processors to perform an iterative method of detecting faces in an image stream using a digital image acquisition device, wherein the method comprises:

a. receiving an acquired image from said image stream including one or more face regions;

b. sub-sampling said acquired image at a specified resolution to provide a sub-sampled image;

c. identifying one or more regions of said acquired image as candidate face regions;

d. calculating a corresponding integral image for a least one of said candidate face regions of said sub-sampled acquired image;

e. applying face detection to at least a portion of said integral image to provide a set of one or more candidate face regions each having a given size and a respective location; and

f. for at least one candidate face region associated with a previous image, enhancing the contrast of the luminance characteristics of the corresponding region of said acquired image.

54. An apparatus as claimed in claim 53, wherein said identifying is performed on said sub-sampled image.

55. An apparatus as claimed in claim 53, wherein said face detection is performed with relaxed face detection parameters.

56. An apparatus as claimed in claim 53, wherein said enhancing is performed on said sub-sampled image.

57. An apparatus as claimed in claim 53, wherein said enhancing is performed during calculation of said integral image.

58. An apparatus as claimed in claim 53, wherein in a face detection mode of said digital image acquisition device, each new acquired image is acquired with progressively increased exposure parameters until at least one candidate face region is detected.