Curvature-based face detector

• US 10,366,278 B2
• Filed: 05/11/2017
• Issued: 07/30/2019
• Est. Priority Date: 09/20/2016
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Eshet, Yaron
• Primary Examiner(s): Chen, Xuemei G
• Application Number:
  US15/592,228
• Publication Number:
  US 20180082109A1
• Time in Patent Office: 810 Days
• Field of Search: 1 1, 348156, 382103, 382115, 382117, 382118, 382190, 382195, 382278, 708322
• CPC Class Codes: G01B 11/24 for measuring contours or curvatures
• CPC Class Codes: G01B 11/25 by projecting a pattern, e.g. one or more lines, moiré fringes on the object G01B11/255 takes precedence ; image analysis for depth or shape recovery G06T7/50
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/64 of convexity or concavity
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: H04N 13/106 Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A method for processing data includes receiving a depth map of a scene containing at least a humanoid head, the depth map comprising a matrix of pixels having respective pixel depth values. A digital processor extracts from the depth map a curvature map of the scene. The curvature map includes respective curvature values of at least some of the pixels in the matrix. The curvature values are processed in order to identify a face in the scene.

Claims

1. A method for processing data, comprising:

receiving a depth map of a scene containing at least a humanoid head, the depth map comprising a matrix of pixels having respective pixel depth values;

using a digital processor, extracting from the depth map a curvature map of the scene, the curvature map comprising respective curvature values of at least some of the pixels in the matrix; and

processing the curvature values in order to detect and segment one or more blobs in the curvature map over which the pixels have respective curvature values that are indicative of a convex surface, to calculate a roll angle of each of the one or more blobs corresponding to an axis perpendicular to a dominant direction of a curvature orientation of the pixels in each of the one or more blobs, and to identify one of the blobs as a face in the scene by applying a face classifier filter to the one or more blobs to calculate a score for each pixel indicating a likelihood that it is a center point of the face while normalizing a rotation between the one or more blobs and the filter using the calculated roll angle.

2. The method according to claim 1, wherein processing the curvature values comprises applying a curvature filter to the curvature map in order to ascertain whether the one of the blobs is the face while correcting for the calculated roll angle.

3. The method according to claim 1, wherein processing the curvature values comprises calculating a scale of the face responsively to a size of the one of the blobs, and applying a curvature filter to the curvature map in order to ascertain whether the one of the blobs is the face while correcting for the calculated scale.

4. The method according to claim 1, wherein extracting the curvature map comprises deriving a first curvature map from the depth map at a first resolution, and wherein detecting the one or more blobs comprises finding the one or more blobs in the first curvature map, and

wherein processing the curvature values comprises deriving a second curvature map containing the one of the blobs at a second resolution, finer than the first resolution, and identifying the face using the second curvature map.

5. The method according to claim 1, wherein applying the face classifier filter comprises convolving the curvature map with a curvature filter kernel in order to find a location of the face in the scene.

6. The method according to claim 5, wherein convolving the curvature map comprises separately applying a face filter kernel and a nose filter kernel in order to compute respective candidate locations of the face, and finding the location based on the candidate locations.

7. The method according to claim 5, wherein convolving the curvature map comprises computing a log likelihood value for each of a plurality of points in the scene, and choosing the location responsively to the log likelihood value.

8. The method according to claim 1, wherein extracting the curvature map comprises applying a shape operator to the pixel depth values and finding eigenvectors and eigenvalues of the shape operator corresponding to respective curvature orientations and curvature values of the pixels in the matrix.

9. Apparatus for processing data, comprising:

an imaging assembly, which is configured to capture a depth map of a scene containing at least a humanoid head, the depth map comprising a matrix of pixels having respective pixel depth values; and

a processor, which is configured to extract from the depth map a curvature map of the scene, the curvature map comprising respective curvature values of at least some of the pixels in the matrix, and to process the curvature values in order to detect and segment one or more blobs in the curvature map over which the pixels have respective curvature values that are indicative of a convex surface, to calculate a roll angle of each of the one or more blobs corresponding to an axis perpendicular to a dominant direction of a curvature orientation of the pixels in each of the one or more blobs, and to identify one of the blobs as a face in the scene by applying a face classifier filter to the one or more blobs to calculate a score for each pixel indicating a likelihood that it is a center point of the face while normalizing a rotation between the one or more blobs and the filter using the calculated roll angle.

10. The apparatus according to claim 9, wherein processing the curvature values comprises applying a curvature filter to the curvature map in order to ascertain whether the one of the blobs is the face while correcting for the calculated roll angle.

11. The apparatus according to claim 9, wherein processing the curvature values comprises calculating a scale of the face responsively to a size of the one of the blobs, and applying a curvature filter to the curvature map in order to ascertain whether the one of the blobs is the face while correcting for the calculated scale.

12. The apparatus according to claim 9, wherein extracting the curvature map comprises deriving a first curvature map from the depth map at a first resolution, and wherein detecting the one or more blobs comprises finding the one or more blobs in the first curvature map, and

wherein processing the curvature values comprises deriving a second curvature map containing the one of the blobs at a second resolution, finer than the first resolution, and identifying the face using the second curvature map.

13. The apparatus according to claim 9, wherein applying the face classifier filter comprises convolving the curvature map with a curvature filter kernel in order to find a location of the face in the scene.

14. The apparatus according to claim 13, wherein convolving the curvature map comprises separately applying a face filter kernel and a nose filter kernel in order to compute respective candidate locations of the face, and finding the location based on the candidate locations.

15. A computer software product, comprising a non-transitory computer-readable medium in which program instructions are stored, which instructions, when read by a computer, cause the computer to receive a depth map of a scene containing at least a humanoid head, the depth map comprising a matrix of pixels having respective pixel depth values, to extract from the depth map a curvature map of the scene, the curvature map comprising respective curvature values of at least some of the pixels in the matrix, and to process the curvature values in order to detect and segment one or more blobs in the curvature map over which the pixels have respective curvature values that are indicative of a convex surface, to calculate a roll angle of each of the one or more blobs corresponding to an axis perpendicular to a dominant direction of a curvature orientation of the pixels in each of the one or more blobs, and to identify one of the blobs as a face in the scene by applying a face classifier filter to the one or more blobs to calculate a score for each pixel indicating a likelihood that it is a center point of the face while normalizing a rotation between the one or more blobs and the filter using the calculated roll angle.

16. The product according to claim 15, wherein applying the face classifier filter comprises convolving the curvature map with a curvature filter kernel in order to find a location of the face in the scene.