Real-time face tracking in a digital image acquisition device

• US 8,270,674 B2
• Filed: 01/03/2011
• Issued: 09/18/2012
• Est. Priority Date: 08/11/2006
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: DigitalOptics Corporation Europe Limited (Adeia Inc.)
• Inventors: Corcoran, Peter, Steinberg, Eran, Petrescu, Stefan, Drimbarean, Alexandru, Nanu, Florin, Pososin, Alexei, Bigioi, Petronel
• Primary Examiner(s): Alavi, Amir
• Application Number:
  US12/983,863
• Publication Number:
  US 20110129121A1
• Time in Patent Office: 624 Days
• Field of Search: 382/103, 382115-118, 382/164, 382/173, 382/181, 382/190, 382/203, 382/243, 382/254, 382/284, 382/299, 382/305, 358/538, 345418-420, 345/473, 345/474, 345/625, 348/14.02, 348/169, 348/349, 348/E07.077
• US Class Current: 382/103
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 18/2148 characterised by the process organisation or structure, e.g. boosting cascade
• CPC Class Codes: G06V 30/19147 Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 30/2504 Coarse or fine approaches, e.g. resolution of ambiguities or multiscale approaches
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: H04N 1/2112 using still video cameras
• CPC Class Codes: H04N 1/32112 in a separate computer file, document page or paper sheet, e.g. a fax cover sheet H04N1/32122 takes precedence
• CPC Class Codes: H04N 1/32128 attached to the image data, e.g. file header, transmitted message header, information on the same page or in the same computer file as the image for information embedded in the image H04N1/32144
• CPC Class Codes: H04N 2101/00 Still video cameras
• CPC Class Codes: H04N 2201/0084 Digital still camera
• CPC Class Codes: H04N 2201/3205 of identification information, e.g. name or ID code H04N2201/3209 takes precedence
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/635 Region indicators; Field of view indicators
• CPC Class Codes: H04N 23/695 Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects

Abstract

An image processing apparatus for tracking faces in an image stream iteratively receives an acquired image from the image stream potentially including one or more face regions. The acquired image is sub-sampled at a specified resolution to provide a sub-sampled image. An integral image is then calculated for a least a portion of the sub-sampled image. Fixed size face detection is applied to at least a portion of the integral image to provide a set of candidate face regions. Responsive to the set of candidate face regions produced and any previously detected candidate face regions, the resolution is adjusted for sub-sampling a subsequent acquired image.

Claims

1. A method of tracking faces or other objects of interest in an image stream using a digital image acquisition device determined to be moving relative to said faces or other objects of interest, comprising:

receiving first and second acquired images from said image stream including one or more face or other object regions;

detecting said one or more face or other object regions in the first acquired image;

receiving an indication of relative movement of said second acquired image relative to the first acquired image due to determined motion of the digital image acquisition device relative to said one or more face or other object regions, said second acquired image having an associated set of candidate face or other object regions, each candidate face region or other regions of interest having a given size and a respective location;

adjusting focus or other capture setting based on the relative movement to optimize focus on the one or more detected face or other object regions or to otherwise optimize a representation of the one or more detected face or other object regions within a succeeding acquired image; and

applying adjusted face or other object detection to at least a portion of said succeeding acquired image in the vicinity of said one or more candidate face or other object regions as a function of said determined motion of the digital image acquisition device relative to said one or more faces or other detected objects to remove variability of changes caused by said determined motion, to provide an updated set of candidate face or other object regions.

2. A method as claimed in claim 1, wherein said indication of relative movement comprises a size and direction of movement.

3. A method as claimed in claim 1, wherein said applying adjusted face or other object detection comprises, prior to applying face or other object detection, shifting said associated set of one or more candidate face or other object regions as a function of said movement.

4. A method as claimed in claim 1, wherein said applying adjusted face or other object detection comprises applying face or other object detection to a region of said acquired image comprising the one or more candidate regions associated with the first acquired image expanded as a function of said movement.

5. A method as claimed in claim 1, further comprising expanding said regions as a function of their original size and as a function of said movement.

6. A method as claimed in claim 1, wherein the adjusting focus or other capture setting comprises optimizing focus on a largest or longest lived face or other object, or both.

7. A method as claimed in claim 1, wherein said applying adjusted face or other object detection comprises applying adjusted sub-sampling resolution.

8. A method as claimed in claim 1, wherein the applying adjusted face or other object detection comprises analyzing one or more focus changes.

9. A method as claimed in claim 1, further comprising detecting said motion of the digital image acquisition device using a motion sensor including an accelerometer and a controlled gain amplifier connected to said accelerometer.

10. A method as claimed in claim 9, further comprising setting the gain of said amplifier relatively low for acquisition of a high resolution image and setting the gain of said amplifier relatively high during acquisition of a stream of relatively low resolution images.

11. A method as claimed in claim 1, further comprising detecting said motion of the digital image acquisition device using a motion sensor including a MEMS sensor.

12. A method as claimed in claim 1, wherein said motion of the digital image acquisition device comprises pan, tilt or zoom, or combinations thereof.

13. One or more non-transitory, processor-readable media having code embedded therein for programming a processor to perform a method of tracking faces or other objects of interest in an image stream acquired using a digital image acquisition device determined to be moving relative to said faces or other objects of interest, the method comprising:

receiving first and second acquired images from said image stream including one or more face or other object regions;

detecting said one or more face or other object regions in the first acquired image;

receiving an indication of relative movement of said second acquired image relative to the first acquired image due to determined motion of the digital image acquisition device relative to said one or more face or other object regions, said second acquired image having an associated set of candidate face or other object regions, each candidate face region or other regions of interest having a given size and a respective location;

adjusting focus or other capture setting based on the relative movement to optimize focus on the one or more detected face or other object regions or to otherwise optimize a representation of the one or more detected face or other object regions within a succeeding acquired image; and

applying adjusted face or other object detection to at least a portion of said succeeding acquired image in the vicinity of said one or more candidate face or other object regions as a function of said determined motion of the digital image acquisition device relative to said one or more faces or other detected objects to remove variability of changes caused by said determined motion, to provide an updated set of candidate face or other object regions.

14. The one or more non-transitory, processor-readable media of claim 13, wherein said indication of relative movement comprises a size and direction of movement.

15. The one or more non-transitory, processor-readable media of claim 13, wherein said applying adjusted face or other object detection comprises, prior to applying face or other object detection, shifting said associated set of one or more candidate face or other object regions as a function of said movement.

16. The one or more non-transitory, processor-readable media of claim 13, wherein said applying adjusted face or other object detection comprises applying face or other object detection to a region of said acquired image comprising the one or more candidate regions associated with the first acquired image expanded as a function of said movement.

17. The one or more non-transitory, processor-readable media of claim 13, wherein the method further comprises expanding said regions as a function of their original size and as a function of said movement.

18. The one or more non-transitory, processor-readable media of claim 13, wherein the adjusting focus or other capture setting comprises optimizing focus on a largest or longest lived face or other object, or both.

19. The one or more non-transitory, processor-readable media of claim 13, wherein said applying adjusted face or other object detection comprises applying adjusted sub-sampling resolution.

20. The one or more non-transitory, processor-readable media of claim 13, wherein the applying adjusted face or other object detection comprises analyzing one or more focus changes.

21. The one or more non-transitory, processor-readable media of claim 13, wherein the method further comprises detecting said motion of the digital image acquisition device using a motion sensor including an accelerometer and a controlled gain amplifier connected to said accelerometer.

22. The one or more non-transitory, processor-readable media of claim 21, wherein the method further comprises setting the gain of said amplifier relatively low for acquisition of a high resolution image and setting the gain of said amplifier relatively high during acquisition of a stream of relatively low resolution images.

23. The one or more non-transitory, processor-readable media of claim 13, wherein the method further comprises detecting said motion of the digital image acquisition device using a motion sensor including a MEMS sensor.

24. The one or more non-transitory, processor-readable media of claim 13, wherein said motion of the digital image acquisition device comprises pan, tilt or zoom, or combinations thereof.

25. A digital image acquisition and processing apparatus including one or more processors and one or more digital storage media having digitally-encoded instructions embedded therein for programming the one or more processors to perform an iterative method of tracking faces or other objects of interest in an image stream acquired by the apparatus determined to be moving relative to said face or other objects of interest, the method comprising:

receiving first and second acquired images from said image stream including one or more face or other object regions;

detecting said one or more face or other object regions in the first acquired image;

receiving an indication of relative movement of said second acquired image relative to the first acquired image due to determined motion of the apparatus relative to said one or more face or other object regions, said second acquired image having an associated set of candidate face or other object regions, each candidate face region or other regions of interest having a given size and a respective location;

adjusting focus or other capture setting based on the determined motion to optimize focus on the one or more detected face or other object regions or to otherwise optimize a representation of the one or more detected face or other object regions within a succeeding acquired image; and

applying adjusted face or other object detection to at least a portion of said succeeding acquired image in the vicinity of said one or more candidate face or other object regions as a function of said determined motion of the apparatus relative to said one or more faces or other detected objects to remove variability of changes caused by said determined motion, to provide an updated set of candidate face or other object regions.

26. An apparatus as claimed in claim 25, wherein said indication of relative movement comprises a size and direction of movement.

27. An apparatus as claimed in claim 25, wherein said applying adjusted face or other object detection comprises, prior to applying face or other object detection, shifting said associated set of one or more candidate face or other object regions as a function of said movement.

28. An apparatus as claimed in claim 25, wherein said applying adjusted face or other object detection comprises applying face or other object detection to a region of said acquired image comprising the one or more candidate regions associated with the first acquired image expanded as a function of said movement.

29. An apparatus as claimed in claim 28, wherein the method further comprises expanding said regions as a function of their original size and as a function of said movement.

30. An apparatus as claimed in claim 25, wherein the adjusting focus or other capture setting comprises optimizing focus on a largest or longest lived face or other object, or both.

31. An apparatus as claimed in claim 25, wherein said applying adjusted face or other object detection comprises applying adjusted sub-sampling resolution.

32. An apparatus as claimed in claim 25, wherein the applying adjusted face or other object detection comprises analyzing one or more focus changes.

33. An apparatus as claimed in claim 25, further comprising a motion sensor including an accelerometer and a controlled gain amplifier connected to said accelerometer.

34. An apparatus as claimed in claim 33, said apparatus being arranged to set the gain of said amplifier relatively low for acquisition of a high resolution image and to set the gain of said amplifier relatively high during acquisition of a stream of relatively low resolution images.

35. An apparatus as claimed in claim 25, further comprising a motion sensor including a MEMS sensor.

36. An apparatus as claimed in claim 25, wherein said motion of the apparatus comprises pan, tilt or zoom, or combinations thereof.