METHOD AND DEVICE FOR REGISTERING AN IMAGE TO A MODEL

• US 20170278302A1
• Filed: 08/24/2015
• Published: 09/28/2017
• Est. Priority Date: 08/29/2014
• Status: Abandoned Application

• Current Assignee: Magnolia Licensing LLC (IP Edge LLC)
• Original Assignee: Thomson Licensing (Vantiva SA)
• Inventors: VARANASI, Kiran, SINGH, Praveer, JOUET, Pierrick
• Application Number:
  US15/505,644
• Publication Number:
  US 20170278302A1
• Time in Patent Office: Days
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2219/2004 Aligning objects, relative positioning of parts
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling
• CPC Class Codes: G06T 2219/2021 Shape modification
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/176 Dynamic expression

Abstract

A method of registering an image to a model, comprising: providing a 3D facial model, said 3D facial model being parameterized from a plurality of facial expressions in images of a reference person to obtain a plurality of sparse and spatially localized deformation components; tracking a set of facial landmarks in a sequence of facial images of a target person to provide sets of feature points defining sparse facial landmarks; computing, a set of localized affine transformations connecting a set of facial regions of the said 3D facial model to the sets of feature points defining the sparse facial landmarks; and applying the localized affine transformations to the 3D facial model and registering the sequence of facial images with the transformed 3D facial model.

Claims

1. A method of registering an image to a model, comprising:

providing a 3D facial model, said 3D facial model being parameterized from a plurality of facial expressions in images of a reference person;

tracking a set of facial landmarks in a sequence of facial images of a target person to provide sets of facial feature points defining the facial landmarks;

computing a set of localized affine transformations connecting facial regions of the said 3D facial model to corresponding sets of feature points defining the facial landmarks;

applying the set of localized affine transformations to the 3D facial model; and

registering the sequence of facial images of the target person with the transformed 3D facial model.

2. The method according to claim 1 wherein the 3D facial model is a blendshape model of a reference face parameterized to blend between different facial expressions.

3. The method according to claim 2 wherein the 3D blendshape model is parameterized into a plurality of sparse localized deformation components.

4. The method according to claim 2 wherein the blendshape model is a linear weighted sum of different blendshape targets representing sparse and spatially localized components of different facial expressions

5. The method according to claim 1 wherein a sparse spatial feature tracking algorithm is used to track the set of facial landmarks.

6. The method according to claim 5 wherein the sparse spatial feature tracking algorithm applies a point distribution model linearly modeling non-rigid shape variations around the facial landmarks.

7. The method according to claim 1 wherein each localized affine transformation is an affine warp comprising at least one of:

a global rigid transformation function;

a scaling function for scaling vertices of the 3D facial model; and

a residual local affine transformation that accounts for localized variation on the face.

8. The method according to claim 1 comprising registering the 3D facial model over the sets of facial feature points after applying at least one of a rigid transform and a non-rigid transform.

9. The method according to claim 1 comprising aligning and projecting dense 3D face points onto the appropriate face regions in an input face image of the target person.

11. A computer program product for a programmable apparatus, the computer program product comprising a sequence of instructions for implementing a method according to claim 1 when loaded into and executed by the programmable apparatus.

10. A device for registering an image to a model, the device comprising memory and at least one processor in communication with the memory, the memory including instructions that when executed by the processor cause the device to perform operations including:

tracking a set of facial landmarks in a sequence of facial images of a target person to provide sets of feature points defining facial landmarks;

computing, a set of localized affine transformations connecting a set of facial regions of a 3D facial model to the sets of feature points defining sparse facial landmarks; and

applying the localized affine transformations to the 3D facial modem and

registering the sequence of facial images with the transformed 3D facial model.