Head motion estimation from four feature points

• US 7,027,618 B2
• Filed: 09/28/2001
• Issued: 04/11/2006
• Est. Priority Date: 09/28/2001
• Status: Expired due to Fees

• Current Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Original Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Inventors: Trajkovic, Miroslav, Gutta, Srinivas, Philomin, Vasanth
• Primary Examiner(s): Ahmed, Samir
• Assistant Examiner(s): BHATNAGAR, ANAND P
• Application Number:
  US09/966,410
• Publication Number:
  US 20030063777A1
• Time in Patent Office: 1,656 Days
• Field of Search: 382/100, 382/103, 382/107, 734/88, 348/154, 348/155, 356/27
• US Class Current: 382/107
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions

Abstract

Linear method for performing head motion estimation from facial feature data, the method comprising the steps of: obtaining first facial image and detecting a head in the first image; detecting position of four points P of said first facial image where P={p₁, p₂, p₃, p₄}, and p_k=(x_k, y_k); obtaining second facial image and detecting a head in the second image; detecting position of four points P′ of the second facial image where P′={p′₁, p′₂, p′₃, p′₄} and p′_k=(x′_k, y′_k); and, determining the motion of the head represented by a rotation matrix R and translation vector T using the points P and P′.

The head motion estimation is governed according to an equation:
P′_i=RP_i+T,
where $R=\left[\begin{array}{c}{r}_1^T\\ r_2^T\\ r_3^T\end{array}\right]={\left[r_{\mathrm{ij}}\right]}_{3\times 3}$
and T=[T₁ T₂ T₃]^T represents camera rotation and translation respectively.

Claims

1. A linear method for performing head motion estimation from facial feature data, the method comprising the steps of:

obtaining a first facial image and detecting a head in said first image;

detecting position of only four points P of said first facial image where P ={p₁, p₂, p₃, p₄}, and p^k=(x_k, y_k);

obtaining a second facial image and detecting a head in said second image;

detecting position of only four points P′

of said second facial image where P′

={p′

₁, p′

₂, p′

₃, p′

₄}and p′

_k=(x′

_k, y′

_k); and

determining the motion of the head represented by a rotation matrix R and translation vector T using said points P and P′

.

2. The linear method of claim 1, wherein said only four points P of said first facial image and said only four points P′

of said second facial image include locations of outer corners of each eye and mouth of each respective first and second facial images.

3. The linear method of claim 1, wherein said head motion estimation is governed according to:

P′

_i=RP_i+T, where

4. A linear method for performing head motion estimation from facial feature data, the method comprising the steps of:

obtaining a first facial image and detecting a head in said first image;

detecting position of four points P of said first facial image where P={p₁, p₂, p₃, p₄}, and p_k=(x_k, y_k);

obtaining a second facial image and detecting a head in said second image;

detecting position of four points P′

of said second facial image where P′

={p′

₁, p′

₂, p′

₃, p′

₄}and p′

_k=[x′

_k, y′

_k]); and

,

determining the motion of the head represented by a rotation matrix R and translation vector T using said points P and P′

,

wherein said head motion estimation is governed according to;

P′

_i=RP_i+T, where

5. The linear method of claim 4, wherein components r1, r2 and r3 are computed as:

r^T₂(P₂−

P₁)=0

r^T₃(P₂−

P₁)=0

r^T₁(P₆−

P₅)=0

r^T₃(P₆−

P₅)=0.

6. The linear method of claim 5, wherein components r1, r2 and r3 are computed as:

r₃=(P₆−

P₅)×

(P₂−

P₁),

r₂=r₃×

(P₂−

P₁)

r₁=r₃×

r₃.

7. The linear method of claim 4, wherein $\left[\begin{array}{cccccc}{P}_i^T& 0^T& 0^T& 1& 0& 0\\ 0^T& P_i^T& 0^T& 0& 1& 0\\ 0^T& 0^T& P_i^T& 0& 0& 1\end{array}\right]$

[r1r2r3T]=Pi′

,

each point pair yielding 3 equations, whereby at least four point pairs are necessary to linearly solve for said rotation and translation.

8. The linear method of claim 7, further comprising the step of:

decomposing said rotation matrix R using Singular Value Decomposition (SVD) to obtain a form R=USV^r.

9. The linear method of claim 7, further comprising the step of computing a new rotation matrix according to R=UV^r.

10. A linear method for performing head motion estimation from facial feature data, the method comprising the steps of:

obtaining image position of four points P_k of a facial image;

determining a rotation matrix R that maps points P_k to F_k for characterizing a head pose, said points F₁, F₂, F₃, F₄ representing three-dimensional (3-D) coordinates of the respective four points of a reference, frontal view of said facial image, and P_k is the three-dimensional (3-D) coordinates of an arbitrary point where P_i=[X_i Y_i Z_i]^T, said mapping governed according to the relation;

R(P₂−

P₁)∝

[1 0 0]^T

R(P₆−

P₅)∝

[0 1 0]^T

wherein P₅ and P₆ are midpoints of respective line segments connecting points P₁P₂ and P₃P₄ and, line segment connecting points P₁P₂ is orthogonal to a line segment connecting points P₅P₆, and ∝

indicates a proportionality factor.

11. The linear method of claim 10, wherein components r1, r2 and r3 are computed as:

r^r₁(P₂−

P₁)=0

r^r₃(P₂−

P₁)=0

r^r₁(P₆−

P₅)=0

r^r₃(P₆−

P₅)=0.

12. The linear method of claim 11, wherein components r1, r2 and r3 are computed as:

r₃=(P₆−

P₅)=(P₂−

P₁)

r₂=r₃×

(P₂−

P₁)

r₁=r₂×

r₃.

13. The linear method of claim 12, wherein a motion of head points is represented according to P′

_i=RP_i=T

where $R=\left[\begin{array}{c}{r}_1^T\\ r_2^T\\ r_3^T\end{array}\right]={\left[r_{\mathrm{ij}}\right]}_{3\times <br><br>3}$

represents image rotation, T=[T₁ T₂ T₃]^r represents translation, and P′

_i denotes a 3-D image position of four points P_k of another facial image.

14. The linear method of claim 13, wherein $\left[\begin{array}{cccccc}{P}_i^T& 0^T& 0^T& 1& 0& 0\\ 0^T& P_i^T& 0^T& 0& 1& 0\\ 0^T& 0^T& P_i^T& 0& 0& 1\end{array}\right]$

[r1r2r3T]=Pi′

,

each point pair yielding 3 equations, whereby at least four point pairs are necessary to linearly solve for said rotation and translation.

15. The linear method of claim 14, further comprising the step of:

decomposing said rotation matrix R using Singular Value Decomposition (SVD) to obtain a form R=USV^r.

16. The linear method of claim 15, further comprising the step of computing a new rotation matrix according to R=UV^r.

17. A program storage device readable by machine, tangible embodying a program of instructions executable by the machine to perform method steps for performing head motion estimation from facial feature data, the method comprising the steps of:

obtaining a first facial image and detecting a head in said first image;

detecting position of only four points P of said first facial image where P={P₁, P₂, P₃, P₄}, and P_k=(x_k, y_k);

obtaining a second facial image and detecting a head in said second image;

detecting position of only four points P′

of said second facial image where ′

={p′

₁, p′

₂, p′

₃, p′

₄}and p′

_k=(x′

_k); and

,

determining the motion of the head represented by a rotation matrix R and translation vector T using said points P and P′

.

18. The program storage device readable by machine as claimed in claim 17, wherein said only four points P of said first facial image and only four points P′

of said second facial image include locations of outer corners of each eye and mouth of each respective first and second facial image.

19. The program storage device readable by machine as claimed in claim 17, wherein said head motion estimation is governed according to:

P′

_i=RP_i+T, where$R=\left[\begin{array}{c}{r}_1^T\\ r_2^T\\ r_3^T\end{array}\right]={\left[r_{\mathrm{ij}}\right]}_{3\times <br><br>3}$

and T=[T₁ T₂ T₃]^r represent camera rotation and translation respectively, said head pose estimation being a specific instance of head motion estimation.

20. A program storage device readable by machine, tangible embodying a program of instructions executable by the machine to perform method steps for performing head motion estimation from facial feature data, the method comprising the steps of:

obtaining a first facial image and detecting a head in said first image;

detecting position of four points P of said first facial image where P={P₁, P₂, P₃, P₄}, and P_k=(x_k, y_k);

obtaining a second facial image and detecting a head in said second image;

detecting position of four points P′

of said second facial image where P′

={p′

₁, p′

₂, p′

₃, p′

₄}and p′

_k=(x′

_k, y′

); and

determining the motion of the head represented by a rotation matrix R and translation vector T using said points P and P′

,

wherein said head motion estimation is governed according to;

Pv′

_i=RP_i+T, where$R=\left[\begin{array}{c}{r}_1^T\\ r_2^T\\ r_3^T\end{array}\right]={\left[r_{\mathrm{ij}}\right]}_{3\times <br><br>3}$

and T=[T₁ T₂ T₃]^r represent camera rotation and translation respectively, said head pose estimation being a specific instance of head motion estimation, and

wherein said head pose estimation is governed according to said rotation matrix R, said me;

hod further comprising the steps of;

determining rotation matrix R that maps points P_k to F_k for characterizing a head pose, said points F₁, F₂, F₃, F₄ representing three-dimensional (3-D) coordinates of the respective four points of a reference, frontal view of said facial image, and P_k is the three-dimensional (3-D) coordinates of an arbitrary point where P_i=[X_i Y_i Z_i], said mapping governed according to the relation;

R(P₂−

P₁)∝

[1 0 0]^r

R(P₆−

P₅)∝

[0 1 0]^r

wherein P₅ and P₆ are midpoints of respective line segments connecting points P₁P₂ and P₃P₄ and, line segment connecting points P₁P₂ is orthogonal to a line segment connecting points P₅P₆, and ∝

indicates a proportionality factor.

21. The program storage device readable by machine as claimed in claim 20, wherein components r1, r2 and r3 are computed as:

r^r₂(P₂−

P₁)=0

r^r₃(P₂−

P₁)=0

r^r₁(P₆−

P₅)=0

r^r₃(P₆−

P₅)=0.

22. The program storage device readable by machine as claimed in claim 20, wherein components r1, r2 and r3 are computed as:

r₃=(₆−

p₅)×

(P₂−

P₁)

r₂=r₃×

(P₂−

P₁)

r₁=r₂×

r₃.

23. The program storage device readable by machine as claimed in claim 20, wherein $\left[\begin{array}{cccccc}{P}_i^T& 0^T& 0^T& 1& 0& 0\\ 0^T& P_i^T& 0^T& 0& 1& 0\\ 0^T& 0^T& P_i^T& 0& 0& 1\end{array}\right]$

[r1r2r3T]=Pi′

,

each point pair yielding 3 equations, whereby at least four point pairs are necessary to linearly solve for said rotation and translation.

24. The program storage device readable by machine as claimed in claim 23, further comprising the steps of decomposing said rotation matrix R using Singular Value Decomposition (SVD) to obtain a form R=USV^r.

25. The program storage device readable by machine as claimed in claim 23, further comprising the steps of computing a new rotation matrix according to R=UV^r.