System and method of estimating 3D facial geometry

• US 9,836,846 B2
• Filed: 06/19/2014
• Issued: 12/05/2017
• Est. Priority Date: 06/19/2013
• Status: Expired due to Fees

• Current Assignee: Commonwealth Scientific & Industrial Research Organisation (Commonwealth Of Australia As Represented By Department Of Industry, Innovation, Climate Change, Science, Research And Tertiary Education)
• Original Assignee: Commonwealth Scientific & Industrial Research Organisation (Commonwealth Of Australia As Represented By Department Of Industry, Innovation, Climate Change, Science, Research And Tertiary Education)
• Inventors: Lucey, Simon
• Primary Examiner(s): Liu, Li
• Application Number:
  US14/899,700
• Publication Number:
  US 20160140719A1
• Time in Patent Office: 1,265 Days
• CPC Class Codes: G01B 11/02 for measuring length, width or thickness G01B11/08 takes precedence
• CPC Class Codes: G01B 11/24 for measuring contours or curvatures
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/20024 Filtering details
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation

Abstract

The present invention relates to image analysis. In particular, but not limited to, the invention relates to estimating 3D facial geometry. First, images are acquired 205 of an object, typically a face. Then a first three-dimensional (3D) geometry of the object is estimated 215 based upon at least the first image. A calibration image of the object and a calibration rig 120 is acquired 405. A scaling factor is determined 420 of the first 3D geometry based upon the calibration image, a known size of the calibration rig 120 and a predetermined spatial configuration. Finally, scaling the first 3D geometry using the scaling factor. The invention also concerns a system and software.

Claims

1. A method of estimating a three-dimensional geometry of an object including:

acquiring a first plurality of images of an object;

estimating a first three-dimensional (3D) geometry of the object based upon at least a subset of the first plurality of images,

wherein estimating the first 3D geometry of the object comprises;

estimating boundary edges of the object in the subset of the first plurality of images;

detecting features of the object in the subset of the first plurality of images using a correlation filter;

performing bundle adjustment across the subset of the first plurality of images for geometric consistency; and

applying a Lambertian appearance constraint on the subset of the first plurality of images;

acquiring a calibration image of both the object and a calibration rig, wherein the calibration rig has a known size, and the calibration rig and the object are in a predetermined spatial configuration;

determining a scaling factor of the first 3D geometry based upon the calibration image, the known size of the calibration rig, and the predetermined spatial configuration; and

scaling the first 3D geometry using the scaling factor.

2. The method of claim 1 wherein the object is a face of a person.

3. The method of claim 1, wherein the subset of the first plurality of images is selected based upon a pose of the object in each of the subset of the first plurality of images.

4. The method of claim 3, wherein selecting the subset of the first plurality of images comprises:

determining a pose of the object in each of the first plurality of images;

clustering the first plurality of images based upon the poses; and

selecting an image from each cluster.

5. The method of claim 4, wherein the object is a face of a person, and determining the pose of the face comprises fitting a cylindrical model to the face.

6. The method of claim 1, wherein estimating the 3D geometry of the object is based at least in part on occlusion boundary edges of the object and a geometric consistency across the first plurality of images.

7. The method of claim 1, wherein estimating boundary edges of the object comprises:

generating probability maps based upon a colour component of at least the subset of the first plurality of images.

8. The method of claim 7 wherein the colour component comprises a red-chrominance component of at least the subset of the first plurality of images.

9. The method of claim 1, wherein detecting features of the object is further based upon view specific models generated based upon training data.

10. The method of claim 1, wherein performing bundle adjustment comprises:

estimating a plurality of feature points of the object in at least the subset of the first plurality of images;

transforming the feature points to compensate for pose variations; and

aligning the transformed feature points with corresponding feature points in neighbouring images.

11. The method of claim 10, wherein the neighbouring images comprise three neighbouring images.

12. The method of claim 1, wherein determining a scaling factor of the 3D geometry comprises:

estimating a pose of the calibration rig in the calibration image; and

estimating a pose of the object in the calibration image, wherein

determining the scaling factor is further based upon the estimated poses of the calibration rig and the object from the calibration image.

13. The method of claim 1, wherein the predetermined configuration between the calibration rig and the object comprises the calibration rig touching the object.

14. The method of claim 1, further comprising rendering a virtual item on the three-dimensional object.

15. The method of claim 14, wherein the virtual item comprises eye glasses.

16. A non-transitory computer-readable medium, including computer-executable instructions stored thereon that when executed by a processor, causes the processor to perform the method of claim 1.

17. A system for estimating a three-dimensional geometry of an object, the system comprising:

a calibration rig;

a camera;

a processor coupled to the camera; and

a memory coupled to the processor, the memory including instruction code executable by the processor for;

acquiring a first plurality of images of an object using the camera;

estimating a three-dimensional (3D) geometry of the object based upon at least a subset of the first plurality of images,

wherein estimating the first 3D geometry of the object comprises;

estimating boundary edges of the object in the subset of the first plurality of images;

detecting features of the object in the subset of the first plurality of images using a correlation filter;

performing bundle adjustment across the subset of the first plurality of images for geometric consistency; and

applying a Lambertian appearance constraint on the subset of the first plurality of images;

acquiring a calibration image of both the object and the calibration rig, wherein the calibration rig and the object are in a predetermined spatial configuration; and

determining a scaling factor of the 3D geometry based upon a known size of the calibration rig and the predetermined spatial configuration.

18. The system of claim 17, wherein the calibration rig comprises an electronic device displaying a QR code.