System and method for depth-recovery of a face in an image

• US 9,916,685 B1
• Filed: 04/28/2016
• Issued: 03/13/2018
• Est. Priority Date: 04/28/2015
• Status: Active Grant

• Current Assignee: Carnegie Mellon University
• Original Assignee: Carnegie Mellon University
• Inventors: Prabhu, Utsav, Savvides, Marios
• Primary Examiner(s): Kholdebarin, Iman K
• Assistant Examiner(s): Tran, Mai
• Application Number:
  US15/140,842
• Time in Patent Office: 684 Days
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/50 Depth or shape recovery

Abstract

A system and method for providing a fast, efficient and accurate method for recovery of depth information from a single image of a face is disclosed. The technique uses a novel thin-plate spline-based dense-correspondence method to align the face, and the representation incorporates a weighted framework, interpreting the depth recovery problem as a weighted data problem.

Claims

1. A system for deriving a 3D model of a face from a 2D image captured with an image sensor, comprising:

a computing device executing software for performing the functions of;

(a) placing a plurality of fiducial points at landmarks on said face, each of said fiducial points corresponding to a fiducial point for a mean face calculated over a database set of faces;

(b) calculating a function which captures the deviation of each fiducial point from said mean fiducial point;

(c) using a thin-plate spline function to create a test face by minimizing a function which provides a measure of the smoothness of the function calculated in step (b);

(d) selecting a set of uniformly-distributed points on said mean face; and

(e) finding a set of points on said test face corresponding to said uniformly-distributed set of points on said mean face; and

(f) for each point in said set of points on said test face, calculating a depth coordinate.

2. The system of claim 1 wherein said depth coordinates are calculated using a linear data completion algorithm.

3. The system of claim 2 wherein said linear data completion algorithm is selected from a group consisting of WGHA and WK-SVD.

4. The system of claim 1 wherein said software further performs the function of providing a textural value for each point in said set of points on said test face.

5. The system of claim 4 wherein said textural value is a grayscale value.

6. The system of claim 1 wherein said software further performs the function of calculating an observational confidence value for each point in said set of points on said test face.

7. The system of claim 6 wherein said observational confidence values are calculated using a generic depth model.

8. The system of claim 6 wherein said observational confidence values are calculated as a function of a camera projection model and a 3D rotation estimate.

9. The system of claim 8 wherein said 3D rotation estimate is calculated along the pitch and yaw axes.

10. The system of claim 1 wherein said landmarks on said face are at predefined locations and further wherein there are 79 fiducial points.