SYSTEM AND METHOD OF ESTIMATING 3D FACIAL GEOMETRY

• US 20160140719A1
• Filed: 06/19/2014
• Published: 05/19/2016
• Est. Priority Date: 06/19/2013
• Status: Active Grant

• Current Assignee: Commonwealth Scientific & Industrial Research Organisation (Commonwealth Of Australia As Represented By Department Of Industry, Innovation, Climate Change, Science, Research And Tertiary Education)
• Original Assignee: Commonwealth Scientific & Industrial Research Organisation (Commonwealth Of Australia As Represented By Department Of Industry, Innovation, Climate Change, Science, Research And Tertiary Education)
• Inventors: LUCEY, Simon
• Granted Patent: US 9,836,846 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G01B 11/02 for measuring length, width or thickness G01B11/08 takes precedence
• CPC Class Codes: G01B 11/24 for measuring contours or curvatures
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/20024 Filtering details
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation

Abstract

The present invention relates to image analysis. In particular, but not limited to, the invention relates to estimating 3D facial geometry. First, images are acquired 205 of an object, typically a face. Then a first three-dimensional (3D) geometry of the object is estimated 215 based upon at least the first image. A calibration image of the object and a calibration rig 120 is acquired 405. A scaling factor is determined 420 of the first 3D geometry based upon the calibration image, a known size of the calibration rig 120 and a predetermined spatial configuration. Finally, scaling the first 3D geometry using the scaling factor. The invention also concerns a system and software.

Claims

1. A method of estimating a three-dimensional geometry of an object including:

acquiring a first image of an object;

estimating a first three-dimensional (3D) geometry of the object based upon at least the first image;

acquiring a calibration image of the object and a calibration rig, wherein the calibration rig has a known size and the calibration rig and the object are in a predetermined spatial configuration;

determining a scaling factor of the first 3D geometry based upon the calibration image, the known size of the calibration rig and the predetermined spatial configuration; and

scaling the first 3D geometry using the scaling factor.

2. The method of claim 1 wherein the object is a face of a person.

3. The method of claim 1 further comprising acquiring a first plurality of images of the object, wherein the first 3D geometry of the object is generated based upon at least a subset of the first plurality of images.

4. The method of claim 3, wherein the subset of the first plurality of images is selected based upon a pose of the object in each of the subset of images.

5. The method of claim 4, wherein selecting the subset of the first plurality of images comprises:

determining a pose of the object in each of the first plurality of images;

clustering the first plurality of images based upon the poses; and

selecting an image from each cluster.

6. The method of claim 5, wherein the object is a face of a person, and determining the pose of the face comprises fitting a cylindrical model to the face.

7. The method of claim 3, wherein estimating the 3D geometry of the object is based at least in part on occlusion boundary edges of the object and a geometric consistency across the first plurality of images.

8. The method of claim 3, further comprising:

estimating boundary edges of the object in the subset of images;

detecting features of the object in each of the subset of images using a correlation filter;

performing bundle adjustment across the subset of images for geometric consistency; and

applying a Lambertian appearance constraint on the subset of images.

9. The method of claim 8, wherein estimating boundary edges of the object comprises:

generating probability maps based upon a colour component of the image.

10. The method of claim 9 wherein the colour component comprises a red-chrominance component of the image.

11. The method of claim 8, wherein detecting features of the object is further based upon view specific models generated based upon training data.

12. The method of claim 8, wherein performing bundle adjustment comprises:

estimating a plurality of feature points of the object in the plurality of images;

transforming the feature points to compensate for pose variations; and

aligning the transformed feature points with corresponding feature points in neighbouring images.

13. The method of claim 12, wherein the neighbouring images comprise three neighbouring images.

14. The method of claim 1, wherein determining a scaling factor of the 3D geometry comprises:

estimating a pose of the calibration rig in the calibration image; and

estimating a pose of the object in the calibration image, wherein

determining the scaling factor is further based upon the estimated poses of the calibration rig and the object from the calibration image.

15. The method of claim 1, wherein the predetermined configuration between the calibration rig and the object comprises the calibration rig touching the object.

16. The method of claim 1, further comprising rendering a virtual item on the three-dimensional object.

17. The method of claim 16, wherein the virtual item comprises eye glasses.

18. A non-transitory computer-readable medium, including computer-executable instructions stored thereon, that when executed by a processor, causes the processor to perform the method of claim 1.

19. A system for estimating a three-dimensional geometry of an object, the system comprising:

a calibration rig;

a camera;

a processor coupled to the camera; and

a memory coupled to the processor, the memory including instruction code executable by the processor for;

acquiring a first image of an object using the camera;

estimating a 3D geometry of the object based upon at least the first image;

acquiring a calibration image of the object and the calibration rig, wherein the calibration rig and the object are in a predetermined spatial configuration; and

determining a scaling factor of the 3D geometry based upon a known size of the calibration rig and the predetermined spatial configuration.

20. The system of claim 19, wherein the calibration rig comprises an electronic device displaying a QR code.