Face tracking for controlling imaging parameters

• US 9,398,209 B2
• Filed: 12/05/2014
• Issued: 07/19/2016
• Est. Priority Date: 08/11/2006
• Status: Active Grant

• Current Assignee: Adeia Imaging LLC (Adeia Inc.)
• Original Assignee: FotoNation Limited (Adeia Inc.)
• Inventors: Corcoran, Peter, Steinberg, Eran, Bigioi, Petronel
• Primary Examiner(s): Alavi, Amir
• Application Number:
  US14/561,905
• Publication Number:
  US 20150092078A1
• Time in Patent Office: 592 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20121 Active appearance model [AAM]
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 10/446 using Haar-like filters, e.g. using integral image techniques
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/173 face re-identification, e.g. recognising unknown faces across different face tracks
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/635 Region indicators; Field of view indicators
• CPC Class Codes: H04N 23/675 comprising setting of focusing regions

Abstract

A method of tracking faces in an image stream with a digital image acquisition device includes receiving images from an image stream including faces, calculating corresponding integral images, and applying different subsets of face detection rectangles to the integral images to provide sets of candidate regions. The different subsets include candidate face regions of different sizes and/or locations within the images. The different candidate face regions from different images of the image stream are each tracked.

Claims

1. A face detection and recognition method, comprising:

receiving a first preview image of a scene and a second preview image of nominally the same scene;

determining whether the first preview image comprises one or more face regions;

in response to determining that the first preview image comprises one or more face regions;

determining a first location of a first face region in the first preview image;

based at least in part on the first location of the first face region in the first preview image, predicting a second location in the second preview image of a second face region corresponding to the first face region in the first preview image;

based at least in part on pixel information of the second face region in the second preview image, determining one or more characteristics of the second face region in the second preview image;

based at least in part on the one or more characteristics of the second face region in the second preview image, determining one or more acquisition parameters for acquiring a main image of nominally the same scene; and

acquiring the main image of nominally the same scene using the one or more acquisition parameters;

wherein the method is performed using one or more computing devices.

2. The face detection and recognition method of claim 1, further comprising:

in response to determining that the first preview image comprises the one or more face regions;

determining a first size of a third face region in the first preview image;

based at least in part on the first size of the third face region in the first preview image, predicting a second size in the second preview image of a fourth face region corresponding to the third face region in the first preview image;

based at least in part on additional information about the fourth face region in the second preview image, determining one or more additional characteristics of the fourth face region in the second preview image;

based at least in part on the one or more additional characteristics of the fourth face region in the second preview image, adjusting the one or more acquisition parameters for acquiring the main image of nominally the same scene; and

acquiring the main image of nominally the same scene using the adjusted one or more acquisition parameters.

3. The face detection and recognition method of claim 1,

wherein the one or more characteristics of the second region include any one of:

sharpness, luminance, texture, color histogram, luminance histogram, horizontal luminance profile, vertical luminance profile, horizontal chrominance profile, vertical chrominance profile, or region correlogram;

wherein the one or more acquisition parameters for acquiring the main image include any one of;

a white balance parameter, a color balance parameter, a focus parameter, or an exposure parameter; and

wherein the one or more acquisition parameters are used to perform any one of;

auto-focus functions, auto-exposure functions, auto-white balance functions, or auto-color balance functions.

4. The face detection and recognition method of claim 1, further comprising:

in response to determining the first location of the first face region in the first preview image;

determining, based on a face detection, a first level of confidence that the first face region is present in the first preview image;

determining a second level of confidence for predicting that the second face region, corresponding to the first face region in the first preview image, is present in the second preview image;

based on the first level of confidence and the second level of confidence, determining a cumulative confidence level; and

storing a history of the face detection in the first preview image, the cumulative confidence level, the first location of the first face region, and the second location of the second face region.

5. The face detection and recognition method of claim 1,

wherein the pixel information of the second face region includes any one of:

color information of pixels includes in the second face region, or luminance information of the pixels included in the second face region.

6. The face detection and recognition method of claim 1, wherein the predicted second face region in the second preview image of the second face region, corresponding to the first face region in the first preview image, is larger than the first face region in the first preview image.

7. The face detection and recognition method of claim 1, wherein the determining whether the first preview image comprises one or more face regions comprises:

calculating an integral image for at least a portion of the first preview image;

applying a face detection to at least a portion of the integral image to provide one or more candidate face regions; and

applying one or more face recognition classifiers to the one or more candidate face regions to recognize one or more faces in the one or more candidate face regions.

8. A digital image acquisition device, comprising a lens, an image sensor, a processor, and one or more instructions which, when executed by the processor, cause the processor to perform:

receiving a first preview image of a scene and a second preview image of nominally the same scene;

determining whether the first preview image comprises one or more face regions;

in response to determining that the first preview image comprises one or more face regions;

determining a first location of a first face region in the first preview image;

based at least in part on the first location of the first face region in the first preview image, predicting a second location in the second preview image of a second face region corresponding to the first face region in the first preview image;

based at least in part on pixel information of the second face region in the second preview image, determining one or more characteristics of the second face region in the second preview image;

based at least in part on the one or more characteristics of the second face region in the second preview image, determining one or more acquisition parameters for acquiring a main image of nominally the same scene; and

acquiring the main image of nominally the same scene using the one or more acquisition parameters.

9. The digital image acquisition device of claim 8, comprising additional instructions which, when executed by the processor, cause the processor to perform:

in response to determining that the first preview image comprises the one or more face regions;

determining a first size of a third face region in the first preview image;

based at least in part on the first size of the third face region in the first preview image, predicting a second size in the second preview image of a fourth face region corresponding to the third face region in the first preview image;

based at least in part on additional information about the fourth face region in the second preview image, determining one or more additional characteristics of the fourth face region in the second preview image;

based at least in part on the one or more additional characteristics of the fourth face region in the second preview image, adjusting the one or more acquisition parameters for acquiring the main image of nominally the same scene; and

acquiring the main image of nominally the same scene using the adjusted one or more acquisition parameters.

10. The digital image acquisition device of claim 8,

wherein the one or more characteristics of the second region include any one of:

sharpness, luminance, texture, color histogram, luminance histogram, horizontal luminance profile, vertical luminance profile, horizontal chrominance profile, vertical chrominance profile, or region correlogram;

wherein the one or more acquisition parameters for acquiring the main image include any one of;

a white balance parameter, a color balance parameter, a focus parameter, or an exposure parameter; and

wherein the one or more acquisition parameters are used to perform any one of;

auto-focus functions, auto-exposure functions, auto-white balance functions, or auto-color balance functions.

11. The digital image acquisition device of claim 8, comprising additional instructions which, when executed by the processor, cause the processor to perform:

in response to determining the first location of the first face region in the first preview image;

determining, based on a face detection, a first level of confidence that the first face region is present in the first preview image;

determining a second level of confidence for predicting that the second face region, corresponding to the first face region in the first preview image, is present in the second preview image;

based on the first level of confidence and the second level of confidence, determining a cumulative confidence level; and

storing a history of the face detection in the first preview image, the cumulative confidence level, the first location of the first face region, and the second location of the second face region.

12. The digital image acquisition device of claim 8,

wherein the pixel information of the second face region includes any one of:

color information of pixels includes in the second face region, or luminance information of the pixels included in the second face region.

13. The digital image acquisition device of claim 8,

wherein the predicted second face region in the second preview image of the second face region, corresponding to the first face region in the first preview image, is larger than the first face region in the first preview image.

14. The digital image acquisition device of claim 8, comprising additional instructions which, when executed by the processor, cause the processor to perform:

calculating an integral image for at least a portion of the first preview image;

applying a face detection to at least a portion of the integral image to provide one or more candidate face regions; and

applying one or more face recognition classifiers to the one or more candidate face regions to recognize one or more faces in the one or more candidate face regions.

15. A non-transitory computer readable storage medium storing one or more instructions which, when executed by one or more processors, cause the processors to perform:

receiving a first preview image of a scene and a second preview image of nominally the same scene;

determining whether the first preview image comprises one or more face regions;

in response to determining that the first preview image comprises one or more face regions;

determining a first location of a first face region in the first preview image;

based at least in part on the first location of the first face region in the first preview image, predicting a second location in the second preview image of a second face region corresponding to the first face region in the first preview image;

based at least in part on pixel information of the second face region in the second preview image, determining one or more characteristics of the second face region in the second preview image;

based at least in part on the one or more characteristics of the second face region in the second preview image, determining one or more acquisition parameters for acquiring a main image of nominally the same scene; and

acquiring the main image of nominally the same scene using the one or more acquisition parameters.

16. The non-transitory computer readable storage medium of claim 15, comprising additional instructions which, when executed by the processor, cause the processor to perform:

in response to determining that the first preview image comprises the one or more face regions;

determining a first size of a third face region in the first preview image;

based at least in part on the first size of the third face region in the first preview image, predicting a second size in the second preview image of a fourth face region corresponding to the third face region in the first preview image;

based at least in part on additional information about the fourth face region in the second preview image, determining one or more additional characteristics of the fourth face region in the second preview image;

based at least in part on the one or more additional characteristics of the fourth face region in the second preview image, adjusting the one or more acquisition parameters for acquiring the main image of nominally the same scene; and

acquiring the main image of nominally the same scene using the adjusted one or more acquisition parameters.

17. The non-transitory computer readable storage medium of claim 15,

wherein the one or more characteristics of the second region include any one of:

sharpness, luminance, texture, color histogram, luminance histogram, horizontal luminance profile, vertical luminance profile, horizontal chrominance profile, vertical chrominance profile, or region correlogram;

wherein the one or more acquisition parameters for acquiring the main image include any one of;

a white balance parameter, a color balance parameter, a focus parameter, or an exposure parameter; and

wherein the one or more acquisition parameters are used to perform any one of;

auto-focus functions, auto-exposure functions, auto-white balance functions, or auto-color balance functions.

18. The non-transitory computer readable storage medium of claim 15, comprising additional instructions which, when executed by the processor, cause the processor to perform:

in response to determining the first location of the first face region in the first preview image;

determining, based on a face detection, a first level of confidence that the first face region is present in the first preview image;

determining a second level of confidence for predicting that the second face region, corresponding to the first face region in the first preview image, is present in the second preview image;

based on the first level of confidence and the second level of confidence, determining a cumulative confidence level; and

storing a history of the face detection in the first preview image, the cumulative confidence level, the first location of the first face region, and the second location of the second face region.

19. The non-transitory computer readable storage medium of claim 15,

wherein the pixel information of the second face region includes any one of:

color information of pixels includes in the second face region, or luminance information of the pixels included in the second face region.

20. The non-transitory computer readable storage medium of claim 15, comprising additional instructions which, when executed by the processor, cause the processor to perform:

calculating an integral image for at least a portion of the first preview image;

applying a face detection to at least a portion of the integral image to provide one or more candidate face regions; and

applying one or more face recognition classifiers to the one or more candidate face regions to recognize one or more faces in the one or more candidate face regions.