METHOD AND APPARATUS FOR MODELING THREE-DIMENSIONAL (3D) FACE, AND METHOD AND APPARATUS FOR TRACKING FACE

• US 20140009465A1
• Filed: 07/05/2013
• Published: 01/09/2014
• Est. Priority Date: 07/05/2012
• Status: Abandoned Application

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Feng, Xuetao, Zhang, Hui, Kim, Ji Yeun, Shen, Xiaolu, Kim, Jung Bae
• Application Number:
  US13/936,001
• Publication Number:
  US 20140009465A1
• Time in Patent Office: Days
• US Class Current: 345/420
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/176 Dynamic expression

Abstract

A method and apparatus for modeling a three-dimensional (3D) face, and a method and apparatus for tracking a face. The method for modeling the 3D face may set a predetermined reference 3D face to be a working model, and generate a result of tracking including at least one of a face characteristic point, an expression parameter, and a head pose parameter from a video frame, based on the working model, to output the result of the tracking.

Claims

1. A method for modeling a three-dimensional (3D) face, the method comprising:

setting a predetermined reference 3D face to be a working model, and tracking a face, based on the working model;

generating a result of the tracking including at least one of a face characteristic point, an expression parameter, and a head pose parameter from a video frame;

updating the working model, based on the result of the tracking.

2. The method of claim 1, wherein the tracking the face comprises tracking the face in a unit of the video frame, and wherein the face is included in the video frame.

3. The method of claim 1, wherein the 3D face comprises:

at least one of a 3D shape of a face, appearance parameters, expression parameters, and head pose parameters.

4. The method of claim 1, wherein the generating of the result of the tracking comprises:

generating results of the tracking corresponding to a predetermined number of video frames, based on a start frame designated among video frames inputted.

5. The method of claim 1, wherein the updating of the working model comprises:

determining whether to update the working model based on comparison of a difference between an appearance parameter of the updated working model and an appearance parameter of the working model prior to the updating with a predetermined threshold value.

6. The method of claim 1, wherein the working model of the 3D face is represented in an equation:

S(a, e, q)=T(Σ

a_iS_i^a+Σ

e_jS_j^e;

q),

wherein “

S”

denotes a 3D shape, “

a”

denotes an appearance component, “

e”

denotes an expression component, “

q”

denotes a head pose, “

T(S, q)”

denotes a function performing at least one of an operation of rotating the 3D shape “

S”

based on the head pose “

q” and

an operation of moving the 3D shape “

S”

based on the head pose “

q”

.

7. The method of claim 6, wherein the predetermined reference 3D face comprises:

an average shape “

s^o”

, an appearance component “

S_i^a”

, an expression component “

S_i^e”

, and a reference head pose “

q^o”

, and

“

i=1;

N, S_i^a”

denotes a change in a face appearance, and “

j=1;

M, S_j^e”

denotes a change in a facial expression.

8. The method of claim 1, further comprising:

training a reference 3D face, in advance, through off-line 3D face data, and setting the trained reference 3D face as a working model.

9. The method of claim 1, wherein the generating of the result of the tracking and the updating of the working model are performed simultaneously.

10. The method of claim 1, wherein the updating of the working model comprises:

selecting a video frame, from the generated result of the tracking, most similar to a neutral expression to be a neutral expression frame;

extracting a face sketch from the selected neutral face frame, based on a face characteristic point included in the neutral expression frame; and

updating the working model, based on the face characteristic point included in the neutral expression frame and the extracted face sketch.

11. The method of claim 10, wherein the selecting of the video frame comprises:

calculating expression parameters with respect to a plurality of video frames tracked;

setting an expression parameter appearing most frequently among the expression parameters to be a neutral expression value; and

selecting a video frame in which a deviation between a total of “

K”

number of expression parameters and the neutral expression value is less than a predetermined threshold value.

12. The method of claim 10, wherein the extracting of the face sketch comprises:

extracting a face sketch from the neutral expression frame, using an active contour model algorithm.

13. The method of claim 6, wherein the updating of the working model comprises:

updating the head pose “

q”

of the working model to be a head pose of the neutral expression frame;

setting an expression component “

e”

of the working model to be “

0”

; and

correcting the appearance component “

a”

of the working model by matching the working model “

S(a, e, q)”

to a location of the face characteristic point of the neutral expression frame, and matching a face sketch calculated through the “

S(a, e, q)”

to the face sketch extracted from the neutral expression frame.

14. The method of claim 1, wherein in the updating of the working model, the working model is continuously updated, and a result of the continuous updating in the working model is reflected.

15. The method of claim 1, wherein the generating of the result of the tracking comprises:

determining a number of video frames on which tracking is to be performed based on at least one of an input rate of a video frame inputted, a characteristic of noise, and an accuracy requirement for the tracking.

16. The method of claim 1, wherein the generating of the result of the tracking comprises:

obtaining at least one of a face characteristic point, an expression parameter, and a head pose parameter, using at least one of an active appearance model (AAM), an active shape model (ASM), and a composite constraint model (AAM).

17. An apparatus for modeling a three-dimensional (3D) face, the apparatus comprising:

a tracking unit to track a face based on a working model, and generate a result of tracking including at least one of a face characteristic point, an expression parameter, and a head pose parameter; and

a modeling unit to update the working model, based on the result of the tracking.

18. The apparatus of claim 17, wherein the tracking unit tracks the face based on the working model with respect to a video frame inputted.

19. The apparatus of claim 17, further comprising:

a training unit to train a 3D reference face, in advance, through off-line 3D face data, and setting the trained reference to be the working model.

20. The apparatus of claim 17, wherein the modeling unit comprises a plurality of modeling units to repeatedly perform updating of the working model through alternative use of the plurality of modeling units.