FACE TRACKING FOR CONTROLLING IMAGING PARAMETERS

• US 20090303342A1
• Filed: 06/05/2009
• Published: 12/10/2009
• Est. Priority Date: 08/11/2006
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Ireland Limited (Adeia Inc.)
• Inventors: Steinberg, Eran, Bigioi, Petronel, Corcoran, Peter
• Granted Patent: US 7,916,897 B2
• Time in Patent Office: Days
• US Class Current: 348/222.100
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20121 Active appearance model [AAM]
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 10/446 using Haar-like filters, e.g. using integral image techniques
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/173 face re-identification, e.g. recognising unknown faces across different face tracks
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/635 Region indicators; Field of view indicators
• CPC Class Codes: H04N 23/675 comprising setting of focusing regions

Abstract

A method of tracking faces in an image stream with a digital image acquisition device includes receiving images from an image stream including faces, calculating corresponding integral images, and applying different subsets of face detection rectangles to the integral images to provide sets of candidate regions. The different subsets include candidate face regions of different sizes and/or locations within the images. The different candidate face regions from different images of the image stream are each tracked.

Claims

1. A method of acquiring an improved image based on tracking a face in a preview image stream with a digital image acquisition device, comprising programming a processor to perform the following:

determining an initial location or size, or both, of a face in a first preview image of a preview image stream;

determining a subsequent location or size, or both, for the same face in a subsequent preview image;

based on the initial and subsequent locations or sizes, or combinations thereof, predicting a region of a third preview image which has just been acquired within which region the same face is expected to occur again; and

analyzing one or more characteristics of said region of said third preview image;

based on the analyzing of the one or more characteristics of said region, adjusting one or more acquisition parameters of a main acquired image.

2. The method of claim 1, wherein said one or more acquisition parameters include white balance, color balance, focus, or exposure, or combinations thereof.

3. The method of claim 1 wherein said one or more characteristics of said region include sharpness, luminance, texture, color histogram, luminance histogram, horizontal luminance profile, vertical luminance profile, horizontal chrominance profile, vertical chrominance profile, or region correlogram, or combinations thereof.

4. The method of claim 1, where the preview and main acquired images comprise different resolutions.

5. A method of tracking faces in a preview image stream with a digital image acquisition device, comprising programming a processor to perform the following:

receiving digital images from a preview image stream including faces;

calculating corresponding integral images for the digital images;

applying different subsets of face detection windows to different subsets of the integral images to provide different sets of candidate face regions of different sizes or locations or both within the digital images; and

tracking each of the different candidate face regions within further images of the image stream or a main target image with which said preview image stream is utilized, or both.

6. The method of claim 5, further comprising merging said different sets of candidate face regions to provide a merged set of candidate face regions of different sizes or locations or both.

7. The method of claim 5, further comprising merging said different sets of candidate face regions with at least one previously detected face region to provide a merged set of candidate face regions of different sizes or locations or both.

8. The method of claim 7, further comprising applying variable-sized face detection to one or more face regions of said merged set of candidate face regions to provide a set of confirmed face regions and a set of rejected face regions.

9. The method of claim 8, wherein applying variable sized face detection comprises applying a plurality of cascades of Haar classifiers of varying size to integral images of face candidate regions of said merged set.

10. The method of claim 8, wherein said applying said different subsets of face detection windows comprises applying fixed-size face detection.

11. The method of claim 10, further comprising:

checking a rejected face region based on alternative criteria from said fixed and variable sized face detection; and

responsive to said checking, indicating that the rejected face region is a face region, and adding said previously rejected face region to said set of confirmed face regions.

12. The method of claim 10, wherein said applying fixed size face detection comprises applying a cascade of Haar classifiers of a fixed size to integral images of face candidate regions of said merged set.

13. The method of claim 5, wherein said face detection windows comprise rectangles.

14. The method of claim 5, further comprising:

determining locations or sizes, or combinations thereof, of a same face within multiple preview images;

based on the determining, predicting a region of a further preview image which has just been acquired within which region the same face is expected to occur again;

analyzing one or more characteristics of said region of said third preview image; and

based on the analyzing, adjusting one or more acquisition parameters of a main acquired image.

15. A digital image acquisition device, comprising a lens, an image sensor, a processor, and a processor-readable memory having digital code embedded therein for programming the processor to perform a method of tracking faces in an image stream acquired by the device, wherein the method comprises:

determining an initial location or size, or both, of a face in a first preview image of a preview image stream;

determining a subsequent location or size, or both, for the same face in a subsequent preview image;

based on the initial and subsequent locations or sizes, or combinations thereof, predicting a region of a third preview image which has just been acquired within which region the same face is expected to occur again; and

analyzing one or more characteristics of said region of said third preview image;

based on the analyzing of the one or more characteristics of said region, adjusting one or more acquisition parameters of a main acquired image.

16. The device of claim 15, wherein said one or more acquisition parameters include white balance, color balance, focus, or exposure, or combinations thereof.

17. The device of claim 15, wherein said one or more characteristics of said region include sharpness, luminance, texture, color histogram, luminance histogram, horizontal luminance profile, vertical luminance profile, horizontal chrominance profile, vertical chrominance profile, or region correlogram, or combinations thereof.

18. The device of claim 15, where the preview and main acquired images comprise different resolutions.

19. A digital image acquisition device, comprising a lens, an image sensor, a processor, and a processor-readable memory having digital code embedded therein for programming the processor to perform a method of tracking faces in a preview image stream acquired by the device, wherein the method comprises:

receiving digital images from a preview image stream including faces;

calculating corresponding integral images for the digital images;

applying different subsets of face detection windows to different subsets of the integral images to provide different sets of candidate face regions of different sizes or locations or both within the digital images; and

tracking each of the different candidate face regions within further images of the image stream or a main target image with which the preview image stream is utilized, or both.

20. The device of claim 19, wherein the method further comprises merging said different sets of candidate face regions to provide a merged set of candidate face regions of different sizes or locations or both.

21. The device of claim 19, wherein the method further comprises merging said different sets of candidate face regions with at least one previously detected face region to provide a merged set of candidate face regions of different sizes or locations or both.

22. The device of claim 21, wherein the method further comprises applying variable-sized face detection to one or more face regions of said merged set of candidate face regions to provide a set of confirmed face regions and a set of rejected face regions.

23. The device of claim 22, wherein applying variable sized face detection comprises applying a plurality of cascades of Haar classifiers of varying size to integral images of face candidate regions of said merged set.

24. The device of claim 22, wherein said applying said different subsets of face detection windows comprises applying fixed-size face detection.

25. The device of claim 24, wherein the method further comprises:

checking a rejected face region based on alternative criteria from said fixed and variable sized face detection; and

responsive to said checking, indicating that the rejected face region is a face region, and adding said previously rejected face region to said set of confirmed face regions.

26. The device of claim 24, wherein said applying fixed size face detection comprises applying a cascade of Haar classifiers of a fixed size to integral images of face candidate regions of said merged set.

27. The device of claim 19, wherein said face detection windows comprise rectangles.

28. The device of claim 19, wherein the method further comprises:

determining locations or sizes, or combinations thereof, of a same face within multiple preview images;

based on the determining, predicting a region of a further preview image which has just been acquired within which region the same face is expected to occur again;

analyzing one or more characteristics of said region of said third preview image; and

based on the analyzing, adjusting one or more acquisition parameters of a main acquired image.

29. One or more computer-readable storage devices having computer-readable code embedded therein for programming one or more processors to perform a method of tracking faces in an image stream acquired with a digital image acquisition device, wherein the method comprises:

determining an initial location or size, or both, of a face in a first preview image of a preview image stream;

determining a subsequent location or size, or both, for the same face in a subsequent preview image;

based on the initial and subsequent locations or sizes, or combinations thereof, predicting a region of a third preview image which has just been acquired within which region the same face is expected to occur again; and

analyzing one or more characteristics of said region of said third preview image;

based on the analyzing of the one or more characteristics of said region, adjusting one or more acquisition parameters of a main acquired image.

30. The one or more computer-readable storage devices of claim 29, wherein said one or more acquisition parameters include white balance, color balance, focus, or exposure, or combinations thereof.

31. The one or more computer-readable storage devices of claim 29, wherein said one or more characteristics of said region include sharpness, luminance, texture, color histogram, luminance histogram, horizontal luminance profile, vertical luminance profile, horizontal chrominance profile, vertical chrominance profile, or region correlogram, or combinations thereof.

32. The one or more computer-readable storage devices of claim 29, where the preview and main acquired images comprise different resolutions.

33. One or more computer-readable storage devices having computer-readable code embedded therein for programming one or more processors to perform a method of tracking faces in an image stream acquired with a digital image acquisition device, wherein the method comprises:

receiving a first image from an image stream including one or more face regions;

calculating a corresponding first integral image for at least a portion of the first image or a sub-sampled version or a combination thereof;

applying a first subset of face detection windows to the first integral image to provide a first set of candidate face regions each having a given size and a respective location;

receiving a second image from the image stream including the one or more face regions, the second image comprising substantially a same scene as the first image;

calculating a corresponding second integral image for at least a portion of the second image or a sub-sampled version or a combination thereof;

applying a second subset of face detection windows to the second integral image to provide a second set of candidate face regions each having a given size and a respective location, the second subset comprising different face detection windows than the first subset, and the first and second subsets comprise candidate face regions of different sizes or locations or both; and

tracking within further images said candidate face regions of different sizes or locations, or both, of said first and second images from said image stream.

34. The one or more computer-readable storage devices of claim 33, wherein the method further comprises merging said first and second sets of candidate face regions with at least one previously detected face region to provide a merged set of candidate face regions of different sizes or locations or both.

35. The one or more computer-readable storage devices of claim 34, wherein the method further comprises applying variable-sized face detection to each face region of said merged set of candidate face regions to provide a set of confirmed face regions and a set of rejected face regions.

36. The one or more computer-readable storage devices of claim 35, wherein said applying said first and second subsets of face detection windows comprises applying fixed-size face detection.

37. The one or more computer-readable storage devices of claim 36, wherein the method further comprises:

checking a rejected face region based on alternative criteria from said fixed and variable sized face detection; and

responsive to said checking, indicating that the rejected face region is a face region, and adding said previously rejected face region to said set of confirmed face regions.

38. The one or more computer-readable storage devices of claim 37, wherein said checking comprises applying a skin prototype to a rejected face region.

39. The one or more computer-readable storage devices of claim 34, wherein said at least one previously detected face region comprises a set of confirmed face regions for one or more previously acquired images.

40. The one or more computer-readable storage devices of claim 35, wherein applying variable sized face detection comprises applying a plurality of cascades of Haar classifiers of varying size to integral images of face candidate regions of said merged set.

41. The one or more computer-readable storage devices of claim 36, wherein said applying fixed size face detection comprises applying a cascade of Haar classifiers of a fixed size to integral images of face candidate regions of said merged set.

42. The one or more computer-readable storage devices of claim 33, wherein the method further comprises responsive to the first image being captured with a flash, analyzing one or more tracked regions of the first integral image for red-eye defect.

43. The one or more computer-readable storage devices of claim 42, wherein the method further comprises correcting in said first integral image a red-eye defect.

44. The one or more computer-readable storage devices of claim 42, wherein the method further comprises storing with said first integral image an indication of a red-eye defect.

45. The one or more computer-readable storage devices of claim 33, wherein the method further comprises repeating the receiving, calculating and applying for one or more further images, including applying one or more further subsets of face detection windows to one or more further integral images to provide one or more further sets of candidate face regions each having a given size and a respective location, the one or more further subsets comprising different face detection windows than the first and second subsets, and the first, second and one more further subsets comprise candidate face regions of different sizes or locations or both.

46. The one or more computer-readable storage devices of claim 33, wherein said face detection windows comprise rectangles.

47. The one or more computer-readable storage devices of claim 33, wherein the method further comprises:

determining locations or sizes, or combinations thereof, of a same face within multiple preview images;

based on the determining, predicting a region of a further preview image which has just been acquired within which region the same face is expected to occur again;

analyzing one or more characteristics of said region of said third preview image; and

based on the analyzing, adjusting one or more acquisition parameters of a main acquired image.

48. One or more computer-readable storage devices having computer-readable code embedded therein for programming one or more processors to perform a method of tracking faces in a preview image stream acquired with a digital image acquisition device, wherein the method comprises:

receiving digital images from a preview image stream including one or more faces;

calculating corresponding integral images for the digital images;

applying different subsets of face detection windows to different subsets of the integral images to provide different sets of candidate face regions of different sizes or locations or both within the digital images; and

tracking each of the different candidate face regions within further images of the image stream or a main target image with which the preview image stream is utilized, or both.

49. The one or more computer-readable storage devices of claim 48, wherein the method further comprises merging said different sets of candidate face regions to provide a merged set of candidate face regions of different sizes or locations or both.

50. The one or more computer-readable storage devices of claim 49, wherein the method further comprises merging said different sets of candidate face regions with at least one previously detected face region to provide a merged set of candidate face regions of different sizes or locations or both.

51. The one or more computer-readable storage devices of claim 50, wherein the method further comprises applying variable-sized face detection to one or more face regions of said merged set of candidate face regions to provide a set of confirmed face regions and a set of rejected face regions.

52. The one or more computer-readable storage devices of claim 51, wherein applying variable sized face detection comprises applying a plurality of cascades of Haar classifiers of varying size to integral images of face candidate regions of said merged set.

53. The one or more computer-readable storage devices of claim 51, wherein said applying said different subsets of face detection windows comprises applying fixed-size face detection.

54. The one or more computer-readable storage devices of claim 53, wherein the method further comprises:

checking a rejected face region based on alternative criteria from said fixed and variable sized face detection; and

responsive to said checking, indicating that the rejected face region is a face region, and adding said previously rejected face region to said set of confirmed face regions.

55. The one or more computer-readable storage devices of claim 53, wherein said applying fixed size face detection comprises applying a cascade of Haar classifiers of a fixed size to integral images of face candidate regions of said merged set.

56. The one or more computer-readable storage devices of claim 48, wherein said face detection windows comprise rectangles.

57. The one or more computer-readable storage devices of claim 48, wherein the method further comprises:

determining locations or sizes, or combinations thereof, of a same face within multiple preview images;

based on the determining, predicting a region of a further preview image which has just been acquired within which region the same face is expected to occur again;

analyzing one or more characteristics of said region of said third preview image; and

based on the analyzing, adjusting one or more acquisition parameters of a main acquired image.