Face or other object detection including template matching

• US 9,639,775 B2
• Filed: 03/30/2015
• Issued: 05/02/2017
• Est. Priority Date: 12/29/2004
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Limited (Adeia Inc.)
• Inventors: Sultana, Bogdan, Petrescu, Stefan, Nicolau, Radu, Ursachi, Vlad Ionut, Bigioi, Petronel, Zaharia, Corneliu, Corcoran, Peter, Fulop, Szabolcs, Gangea, Mihnea
• Primary Examiner(s): Kholdebarin, Iman K
• Application Number:
  US14/673,246
• Publication Number:
  US 20150206030A1
• Time in Patent Office: 764 Days
• Field of Search: None
• CPC Class Codes: G06V 10/60 relating to illumination properties, e.g. using a reflectance or lighting model
• CPC Class Codes: G06V 10/75 Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
• CPC Class Codes: G06V 10/94 Hardware or software architectures specially adapted for image or video understanding
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/164 using holistic features

Abstract

A template matching module is configured to program a processor to apply multiple differently-tuned object detection classifier sets in parallel to a digital image to determine one or more of an object type, configuration, orientation, pose or illumination condition, and to dynamically switch between object detection templates to match a determined object type, configuration, orientation, pose, blur, exposure and/or directional illumination condition.

Claims

1. A digital image processing device, comprising:

an optoelectronic system;

a memory;

two or more image processing units;

a plurality of object detection templates, wherein each object detection template of the plurality of object detection templates is tuned for high detection, low detection ratios for detecting faces;

a plurality of high-quality object detection templates tuned for low detection, high rejection ratios for detecting faces;

wherein the high-quality object detection templates are different from the object detection templates;

wherein the optoelectronic system acquires a plurality of digital images;

wherein a first data processing unit (“

DPU”

) that lacks a program counter, of the two or more image processing units;

determines, for a first digital image of the plurality of digital images, a location and a boundary of one or more spatial regions where one can expect to detect a face by applying in parallel, to the first digital image, two or more object detection templates, of the plurality of object detection templates;

stores in the memory the first digital image and information about the location and the boundary of one or more spatial regions;

processes a second digital image of the plurality of digital images;

wherein, as the first DPU processes the second digital image, by applying in parallel, to the second digital image, the two or more object detection templates of the plurality of object detection templates for detecting faces, a second DPU of the two or more image processing units;

based on the location and the boundary, retrieves the one or more spatial regions of the first digital image and performs an additional processing on the one or more spatial regions of the first digital image by;

determining whether the face is depicted in the one or more spatial regions as facing a camera at a particular angle by applying, to the one or more spatial regions of the first digital image, one or more high-quality object detection templates of the plurality of high-quality object detection templates tuned for low detection, high rejection ratios for detecting faces;

in response to determining that the face is depicted in the one or more spatial regions as facing to the camera at the particular angle, sending a confirmation message that the face facing the camera at the particular angle was detected; and

wherein the additional processing on the one or more spatial regions of the first digital image by the second DPU is performed in parallel with the processing of the second digital image by the first DPU.

2. The digital image processing device of claim 1,

wherein the memory is dual-ported;

and

wherein one port of the memory is used to read one digital image data while another port of the memory is used to store another digital image data.

3. The digital image processing device of claim 1,

wherein the second DPU performs the additional processing of the one or more spatial regions while the first DPU performs a generic processing of the second digital image.

4. The digital image processing device of claim 3,

wherein the additional processing of the one or more spatial regions is delayed by a frame-processing-cycle delay;

and

wherein the frame-processing-cycle delay in the additional processing of the first digital image is compensated to achieve a pseudo-real-time processing of the first digital image.

5. The digital image processing device of claim 1,

wherein, in response to performing the additional processing on the one or more spatial regions, the first digital image is transmitted to a face tracking unit.

6. The digital image processing device of claim 1,

wherein the first DPU of the two or more image processing units stores the first digital image and the information about the location and the boundary of one or more spatial regions in one or more buffers associated with the first DPU.

7. The digital image processing device of claim 6,

wherein, upon receiving the second digital image, the first DPU retrieves the information about the location and the boundary of one or more spatial regions from the one or more buffers, and performs the additional processing on the one or more spatial regions.

8. A method for a hardware-based acceleration of object detection in digital images, the method comprising:

acquiring a plurality of digital images;

at a first data processing unit (“

DPU”

) that lack a program counter, of two or more image processing units;

determining, for a first digital image of the plurality of digital images, a location and a boundary of one or more spatial regions where one can expect to detect a face by applying in parallel, to the first digital image, two or more object detection templates of a plurality of object detection templates tuned for high detection, low rejection ratios for detecting faces;

storing in a memory the first digital image and information about the location and the boundary of one or more spatial regions;

processing a second digital image of the plurality of digital images;

at a second DPU of the two or more processing units, as the first DPU processes the second digital image by applying in parallel, to the second digital image, the two or more object detection templates of the plurality of object detection templates tuned for high detection, low rejection ratios for detecting faces;

based on the location and the boundary, retrieving the one or more spatial regions of the first digital image and performing an additional processing on the one or more spatial regions of the first digital image by;

determining whether the face is depicted in the one or more spatial regions as facing a camera at a particular angle by applying, to the one or more spatial regions of the first digital image, one or more high-quality object detection templates of a plurality of high-quality object detection templates tuned for low detection, high rejection ratios for detecting faces;

wherein the high-quality object detection templates are different from the object detection templates;

in response to determining that the face is depicted in the one or more spatial regions as facing the camera at the particular angle, sending a confirmation message that the face facing the camera at the particular angle was detected; and

wherein the additional processing on the one or more spatial regions of the first digital image by the second DPU is performed in parallel with the processing of the second digital image by the first DPU.

9. The method of claim 8,

wherein the memory is dual-ported;

and

wherein one port of the memory is used to read one digital image data while another port of the memory is used to store another digital image data.

10. The method of claim 8,

wherein the second DPU performs the additional processing of the one or more spatial regions while the first DPU performs a generic processing of the second digital image.

11. The method of claim 10,

wherein the additional processing of the one or more spatial regions is delayed by a frame-processing-cycle delay;

and

wherein the frame-processing-cycle delay in the additional processing of the first digital image is compensated to achieve a pseudo-real-time processing of the first digital image.

12. The method of claim 8,

wherein, in response to performing the additional processing on the one or more spatial regions, the first digital image is transmitted to a face tracking unit.

13. The method of claim 8,

wherein the first DPU of the two or more image processing units stores the first digital image and the information about the location and the boundary of one or more spatial regions in one or more buffers associated with the first DPU.

14. The method of claim 13,

wherein, upon receiving the second digital image, the first DPU retrieves the information about the location and the boundary of one or more spatial regions from the one or more buffers, and performs the additional processing on the one or more spatial regions.

15. A non-transitory computer-readable storage medium storing one or more instructions which, when executed by one or more processors, cause the one or more processors to perform:

acquiring a plurality of digital images;

at a first data processing unit (“

DPU”

) that lacks a program counter, of two or more image processing units;

determining, for a first digital image of the plurality of digital images, a location and a boundary of one or more spatial regions where one can expect to detect a face by applying in parallel, to the first digital image, two or more object detection templates of a plurality of object detection templates tuned for high detection, low rejection ratios for detecting faces;

storing in a memory the first digital image and information about the location and the boundary of one or more spatial regions;

processing a second digital image of the plurality of digital images;

at a second DPU of the two or more processing units, as the first DPU processes the second digital image by applying in parallel, to the second digital image, two or more object detection templates of the plurality of object detection templates tuned for high detection, low rejection ratios for detecting faces;

based on the location and the boundary, retrieving the one or more spatial regions of the first digital image and performing an additional processing on the one or more spatial regions of the first digital image by;

determining whether the face is depicted in the one or more spatial regions as facing a camera at a particular angle, by applying, to the one or more spatial regions of the first digital image, one or more high-quality object detection templates of a plurality of high-quality object detection templates tuned for low detection, high rejection ratios for detecting faces;

wherein the high-quality object detection templates are different from the object detection templates;

in response to determining that the face is depicted in the one or more spatial regions as facing the camera at the a particular angle, sending a confirmation message that the face facing the camera at the particular angle was detected; and

wherein the additional processing on the one or more spatial regions of the first digital image by the second DPU is performed in parallel with the processing of the second digital image by the first DPU.

16. The non-transitory computer-readable storage medium of claim 15, wherein the memory is dual-ported;

and

wherein one port of the memory is used to read one digital image data while another port of the memory is used to store another digital image data.

17. The non-transitory computer-readable storage medium of claim 15,

wherein the second DPU performs the additional processing of the one or more spatial regions while the first DPU performs a generic processing of the second digital image.

18. The non-transitory computer-readable storage medium of claim 17,

wherein the additional processing of the one or more spatial regions is delayed by a frame-processing-cycle delay;

and

wherein the frame-processing-cycle delay in the additional processing of the first digital image is compensated to achieve a pseudo-real-time processing of the first digital image.

19. The non-transitory computer-readable storage medium of claim 15,

wherein, in response to performing the additional processing on the one or more spatial regions, the first digital image is transmitted to a face tracking unit.

20. The non-transitory computer-readable storage medium of claim 15,

wherein the first DPU of the two or more image processing units stores the first digital image and the information about the location and the boundary of the one or more spatial region in one or more buffers associated with the first image processing unit.