System and method for face recognition with two-dimensional sensing modality

• US 9,875,398 B1
• Filed: 06/30/2016
• Issued: 01/23/2018
• Est. Priority Date: 06/30/2016
• Status: Active Grant

• Current Assignee: United States Of America As Represented By The Secretary Of The Army (Government of the United States of America)
• Original Assignee: The United States Of America As Represented By The Secretary Of The Army
• Inventors: Young, Shiqiong Susan
• Primary Examiner(s): Bayat, Ali
• Application Number:
  US15/198,344
• Publication Number:
  US 20180005021A1
• Time in Patent Office: 572 Days
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 10/75 Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

A method and system in which facial image representations stored in a database are defined by facial coordinates in a plane common to other images in the database in order to facilitate comparison or likeness of the facial images by comparing the common plane facial coordinates, the common plane being determined by the locations of the eyes and mouth corners; at least one input operatively connected to the at least one processor and configured to input the corners of the eyes and mouth coordinates; the at least one processor configured to convert inputted coordinates for the corners of the eyes and mouth into estimated common plane coordinates by minimizing the error between the inputted corners of the eyes and mouth coordinates and the estimated coordinates corners of the eyes and mouth obtained from the least square estimation model of the common plane coordinates of the corners of eyes and mouth.

Claims

1. A system for facial recognition using an image of a human in which the eyes and mouth corners are not in a vertical plane when the image was taken to determine estimated virtual plane coordinates corresponding to estimated location of eyes and mouth corners of the human when the eyes and mouth corners are in a vertical plane;

the system comprising;

at least one processor configured to determine the virtual plane coordinates;

at least one input operatively connected to the at least one processor and configured to input the first corners of the eyes and mouth coordinates F₁, F₂, F₃, F₄, F₅, F₆, F₇, F₈, F₉, F₁₀, F₁₁, F₁₂ from the image into the at least one processor comprising data points in a vector

F=(F₁, . . . ,F₁₂)^t

where t represents the transpose, F₁ represents the horizontal coordinate of the left eye outer corner, F₂ represents the vertical coordinate of the left eye outer corner, F₃ represents the horizontal coordinate of the left eye inner corner, F₄ represents the vertical coordinate of the left eye inner corner, F₅ represents the horizontal coordinate of the right eye outer corner, F₆ represents the vertical coordinate of the right eye outer corner, F₇ represents the horizontal coordinate of the right eye inner corner, F₈ represents the vertical coordinate of the right eye inner corner, F₉ represents the horizontal coordinate of the left mouth corner, F₁₀ represents the vertical coordinate of the left mouth corner, F₁₁ represents the horizontal coordinate of the right mouth corner, and F₁₂ represents the vertical coordinate of the right mouth corner;

the at least one processor configured to convert the first coordinates F₁, F₂, F₃, F₄, F₅, F₆, F₇, F₈, F₉, F₁₀, F₁₁, F₁₂ into second coordinates for the corners of the eye and mouth in a virtual vertical plane comprising P₁=(−

x_e1,y_e1,0), P₂=(−

x_e2,y_e2,0), P₃=(x_e1,y_e1,0), P₄=(x_e2,y_e2,0), P₅=(−

x_m,y_m,0), and P₆=(x_m,y_m,0), where P₁ is the estimated left eye outer corner coordinates, P₂ is, the estimated left eye inner corner coordinates, P₃ is the estimated right eye outer corner coordinates, P₄ is the estimated right eye inner corner coordinates, P₅ is the estimated left mouth corner coordinates and P₆ is the estimated right mouth corner coordinates and x and y represent horizontal and vertical distances from a facial reference point, and to determine the head orientation of the human subject using roll, yaw and pitch relative to the virtual vertical plane where, θ

represents the yaw, φ

represents the pitch , and ψ

represents the roll;

the at least one processor configured to solve for the parameter vector V_p comprising 9 parameters x_e1, y_e1, x_e2, y_e2, x_m, y_m, θ

, φ

, ψ

using the following equation;

V_p=(x_e1,y_e1,x_e2,y_e2,x_m,y_m,θ

,φ

,ψ

)^t

where x_e1, y_e1, x_e2, y_e2, x_m, y_m, represent the virtual plane coordinates of the corners of the eyes and mouth in the vertical virtual plane, and wherein the error to be minimized is the error between the inputted corners of the eyes and mouth coordinates F₁, F₂, F₃, F₄, F₅, F₆, F₇, F₈, F₉, F₁₀, F₁₁, F₁₂ and the least square estimation model function ƒ

(V_p) coordinates of the estimated inputted corners of the eyes and mouth coordinate values comprising horizontal coordinates ƒ

₁(V_p), ƒ

₃(V_p), ƒ

₅(V_p), ƒ

₇(V_p) ƒ

₉(V_p), ƒ

₁₁(V_p), and vertical coordinates ƒ

₂(V_p), ƒ

₄(V_p), ƒ

₆(V_p), ƒ

₈(V_p), ƒ

₁₀(V_p), ƒ

₁₂(V_P) and where the least square model function ƒ

(V_p) is computed using

ƒ

₁(V_p)=[1,0,0]T P₁

ƒ

₂(V_p)=[0,1,0]T P₁

ƒ

₃(V_p)=[1,0,0]T P₂

ƒ

₄(V_p)=[0,1,0]T P₂

ƒ

₅(V_p)=[1,0,0]T P₃

ƒ

₆V_p)=[0,1,0]T P₃

ƒ

₇(V_p)=[1,0,0]T P₄

ƒ

₈(V_p)=[0,1,0]T P₄

ƒ

₉(V_p)=[1,0,0]T P₅

ƒ

₁₀(V_p)=[0,1,0]T P₅

ƒ

₁₁(V_p)=[1,0,0]T P₆

ƒ

₁₂(V_p)=[0,1,0]T P₆

where T correlates to the head orientation θ

, φ

, ψ and

the matrix T=T_θ

·

T_φ

·

T_ψ

where

2. The system of claim 1 wherein the virtual plane correlates to a plane having zero values for the yaw, pitch, and roll.

3. The system of claim 1 wherein if the subject of the image is photographed when the eyes and mouth coordinates are in a vertical plane, the vertical plane coordinates inputted into the database directly from the image.

4. The system of claim 1 wherein the plurality of images of people comprise suspected individuals, persons convicted of crimes, driver'"'"'s license images of general public, criminal detection, airline screening, access control, surveillance system, identification of dead bodies, terrorist watch list, and wherein each image is represented using virtual or vertical plane coordinates of the corners of the eyes and mouth for each image in the gallery.

5. The system of claim 1 wherein the image of a human correlates to a probe and wherein the database comprises targeted individuals and wherein the virtual plane coordinates are used to determine vertical plane coordinates of the corners of the eyes and mouth of the image in the probe.

6. The system of claim 1 wherein the virtual plane coordinates of the corners of the eyes and mouth of the image of a human are matched against a gallery of virtual or vertical plane coordinates of the corners of eyes and mouths to generate the matching scores and to determine a minimum match score to determine a match.

7. The method of claim 6 wherein the matching is based on the multiple hypotheses testing with the risk function of minimizing the probability of error of a matching result and comprises calculating the matrix distances between the probe vertical or virtual plane coordinates of the corners of the eyes and mouth and each gallery vertical or virtual plane coordinates of the corners of the eyes and mouth to generate matching scores.

8. The system of claim 1 wherein the at least one processor is configured to solve for the parameter vector V_p using the non-linear Gaussian Least Square Differential Correction algorithm, the at least one processor being configured to perform repeated iterations comprising first determining whether to update the vector V_p by calculating Δ

F=F−

ƒ

(V_p) to calculate the change of an estimated least square estimation model function relative to the actual values F, and then using the equation

9. A system for facial recognition in which facial images in a database are defined by facial coordinates in a plane common to other images in order to facilitate comparison or likeness of the facial images:

at least one input operatively connected to the at least one processor and configured to input the corners of the eyes and mouth coordinates F₁, F₂, F₃, F₄, F₅, F₆, F₇, F₈, F₉, F₁₀, F₁₁, F₁₂ from a facial image into the at least one processor using a vector

F=(F₁, . . . ,F₁₂)^t

where t represents the transpose, F₁ represents the horizontal coordinate of the left eye outer corner, F₂ represents the vertical coordinate of the left eye outer corner, F₃ represents the horizontal coordinate of the left eye inner corner, F₄ represents the vertical coordinate of the left eye inner corner, F₅ represents the horizontal coordinate of the right eye outer corner, F₆ represents the vertical coordinate of the right eye outer corner, F₇ represents the horizontal coordinate of the right eye inner corner, F₈ represents the vertical coordinate of the right eye inner corner, F₉ represents the horizontal coordinate of the left mouth corner, F₁₀ represents the vertical coordinate of the left mouth corner, F₁₁ represents the horizontal coordinate of the right mouth corner, and F₁₂ represents the vertical coordinate of the right mouth corner;

the at least one processor configured to convert the first coordinates F₁, F₂, F₃, F₄, F₅, F₆, F₇, F₈, F₉, F₁₀, F₁₁, F₁₂ into second coordinates for the corners of the eye and mouth in a virtual vertical plane comprising P₁=(−

x_e1,y_e1,0), P₂=(−

x_e2,y_e2,0), P₃=(x_e1,y_e1,0), P₄=(x_e2,y_e2,0), P₅=(−

x_m,y_m,0), and P₆=(x_m,y_m,0), where P₁ is the estimated left eye outer corner coordinates, P₂ is, the estimated left eye inner corner coordinates, P₃ is the estimated right eye outer corner coordinates, P₄ is the estimated right eye inner corner coordinates, P₅ is the estimated left mouth corner coordinates and P₆ is the estimated right mouth corner coordinates; and

wherein x and y represent horizontal and vertical distances from a facial reference point, x_e1 and −

x_e1 represent the horizontal coordinates of the outer corners of the eyes, x_e2 and −

x_e2 represent the horizontal coordinates of the inner corners of the eyes, y_e1 and y_e2 represent the vertical coordinates of the outer and inner corners of the eyes, x_m and −

x_m represent the horizontal coordinates of the corners of the mouth, y_m represent the vertical coordinates of the corners of the mouth in the virtual vertical plane;

the at least one processor being configured to solve for the parameter vector V_p using the equation V_p=(x_e1,y_e1,x_e2,y_e2,x_m,y_m,θ

,φ

,ψ

,α

)^t wherein the parameter vector V_p comprises 10 parameters x_e1, y_e1, x_e2, y_e2, x_m, y_m, θ

, φ

, ψ

, α and

wherein θ

represents the yaw, φ

represents the pitch, ψ

represents the roll and α

represents the scale in the following equation;

V_p=(x_e1,y_e1,x_e2,y_e2,x_m,y_m,θ

,φ

,ψ

,α

)^t

where x_e1, y_e1, x_e2, y_e2, x_m, y_m, represent the virtual plane coordinates of the corners of the eyes and mouth in the vertical virtual plane, and wherein the error to be minimized is the error between the inputted corners of the eyes and mouth coordinates F₁, F₂, F₃, F₄, F₅, F₆, F₇, F₈, F₉, F₁₀, F₁₁, F₁₂ and the least square estimation model function ƒ

(V_p) coordinates of the estimated inputted corners of the eyes and mouth coordinate values comprising horizontal coordinates ƒ

₁(V_p), ƒ

₃(V_p), ƒ

₅(V_p), ƒ

₇(V_p) ƒ

₉(V_p), ƒ

₁₁(V_p), and vertical coordinates ƒ

₂(V_p), ƒ

₄(V_p), ƒ

₆(V_p), ƒ

₈(V_p), ƒ

₁₀(V_p), ƒ

₁₂(V_P) and where the least square model function ƒ

(V_p) is computed using

ƒ

₁(V_p)=α

[1,0,0]T P₁

ƒ

₂(V_p)=α

[0,1,0]T P₁

ƒ

₃(V_p)=α

[1,0,0]T P₂

ƒ

₄(V_p)=α

[0,1,0]T P₂

ƒ

₅(V_p)=α

[1,0,0]T P₃

ƒ

₆V_p)=α

[0,1,0]T P₃

ƒ

₇(V_p)=α

[1,0,0]T P₄

ƒ

₈(V_p)=α

[0,1,0]T P₄

ƒ

₉(V_p)=α

[1,0,0]T P₅

ƒ

₁₀(V_p)=α

[0,1,0]T P₅

ƒ

₁₁(V_p)=α

[1,0,0]T P₆

ƒ

₁₂(V_p)=α

[0,1,0]T P₆

where T correlates to the head orientation θ

, φ

, ψ and

the matrix T=T_θ

·

T_φ

·

T_ψ

where

10. A system for facial recognition in which facial image representations stored in a database are defined by facial coordinates in a plane common to other images in the database:

at least one processor configured to compare the likeness of the facial images by comparing the common plane facial coordinates, the common plane being determined by the locations of the eyes and mouth corners;

at least one input operatively connected to the at least one processor and configured to input the corners of the eyes and mouth coordinates from a facial image into the at least one processor;

the at least one processor configured to convert the inputted coordinates for the corners of the eyes and mouth into estimated coordinates in a common plane by minimizing the error between the inputted corners of the eyes and mouth coordinates and the estimated coordinates corners of the eyes and mouth obtained from an estimation of the common plane coordinates of the corners of eyes and mouth;

the at least one processor configured to perform facial recognition to determine identification of a subject by comparing the common plane facial coordinates of an inputted facial image.

11. The system of claim 10 wherein inputted first corners of the eyes and mouth coordinates F₁, F₂, F₃, F₄, F₅, F₆, F₇, F₈, F₉, F₁₀, F₁₁, F₁₂ are determined from a facial image inputted into the at least one processor using a vector

F=(F₁, . . . ,F₁₂)^t

where t represents the transpose, F₁ represents the horizontal coordinate of the left eye outer corner, F₂ represents the vertical coordinate of the left eye outer corner, F₃ represents the horizontal coordinate of the left eye inner corner, F₄ represents the vertical coordinate of the left eye inner corner, F₅ represents the horizontal coordinate of the right eye outer corner, F₆ represents the vertical coordinate of the right eye outer corner, F₇ represents the horizontal coordinate of the right eye inner corner, F₈ represents the vertical coordinate of the right eye inner corner, F₉ represents the horizontal coordinate of the left mouth corner, F₁₀ represents the vertical coordinate of the left mouth corner, F₁₁ represents the horizontal coordinate of the right mouth corner, and F₁₂ represents the vertical coordinate of the right mouth corner;

and wherein the coordinates in the common plane for the corners of the eye and mouth comprise P₁=(−

x_e1,y_e1,0), P₂=(−

x_e2,y_e2,0), P₃=(x_e1,y_e1,0), P₄=(x_e2,y_e2,0), P₅=(−

x_m,y_m,0), and P₆=(x_m,y_m,0), where P₁ is the estimated left eye outer corner coordinates, P₂ is, the estimated left eye inner corner coordinates, P₃ is the estimated right eye outer corner coordinates, P₄ is the estimated right eye inner corner coordinates, P₅ is the estimated left mouth corner coordinates and P₆ is the estimated right mouth corner coordinates; and

wherein x and y represent horizontal and vertical distances from a facial reference point, x_e1 and −

x_e1 represent the horizontal coordinates of the outer corners of the eyes, x_e2 and −

x_e2 represent the horizontal coordinates of the inner corners of the eyes, y_e1 and y_e2 represent the vertical coordinates of the outer and inner corners of the eyes, x_m and −

x_m represent the horizontal coordinates of the corners of the mouth, y_m represent the vertical coordinates of the corners of the mouth in the virtual vertical plane;

and wherein the at least one processor converts the first coordinates F₁, F₂, F₃, F₄, F₅, F₆, F₇, F₈, F₉, F₁₀, F₁₁, F₁₂ into second coordinates for the corners of the eye and mouth in a common vertical plane by solving for the parameter vector V_p using the equation V_p=(x_e1,y_e1,x_e2,y_e2,x_m,y_m,θ

,φ

,ψ

)^t wherein the parameter vector V_p comprises 9 parameters x_e1, y_e1, x_e2, y_e2, x_m, y_m, θ

, φ

, ψ and

wherein θ

represents the yaw, φ

represents the pitch, and ψ

represents the roll of the facial image relative to the virtual vertical plane,

and wherein the error to be minimized is the error between the inputted corners of the eyes and mouth coordinates F₁, F₂, F₃, F₄, F₅, F₆, F₇, F₈, F₉, F₁₀, F₁₁, F₁₂ and the least square estimation model function ƒ

(V_p) coordinates of the estimated inputted corners of the eyes and mouth coordinate values comprising horizontal coordinates ƒ

₁(V_p), ƒ

₃(V_p), ƒ

₅(V_p), ƒ

₇(V_p) ƒ

₉(V_p), ƒ

₁₁(V_p), and vertical coordinates ƒ

₂(V_p), ƒ

₄(V_p), ƒ

₆(V_p), ƒ

₈(V_p), ƒ

₁₀(V_p), ƒ

₁₂(V_P) and where the least square model function ƒ

(V_p) is computed using

ƒ

₁(V_p)=[1,0,0]T P₁

ƒ

₂(V_p)=[0,1,0]T P₁

ƒ

₃(V_p)=[1,0,0]T P₂

ƒ

₄(V_p)=[0,1,0]T P₂

ƒ

₅(V_p)=[1,0,0]T P₃

ƒ

₆V_p)=[0,1,0]T P₃

ƒ

₇(V_p)=[1,0,0]T P₄

ƒ

₈(V_p)=[0,1,0]T P₄

ƒ

₉(V_p)=[1,0,0]T P₅

ƒ

₁₀(V_p)=[0,1,0]T P₅

ƒ

₁₁(V_p)=[1,0,0]T P₆

ƒ

₁₂(V_p)=[0,1,0]T P₆

where T correlates to the head orientation comprising yaw, pitch and roll (θ

,φ

,ψ

) and is a matrix T=T_θ

·

T_φ

·

T_ψ

where

12. The system of claim 10 wherein the at least one processor defines common plane facial coordinates correlating to the virtual locations of the inputted facial feature coordinates located in a virtual vertical plane wherein the yaw, pitch and roll are zero.

13. The system of claim 10 wherein the at least one processor is configured to perform facial recognition by comparing the location of the coordinates of the corners of the eyes and mouth coordinates of a subject to the locations of the corners of the eyes and mouth coordinates of the images of people in the database.

14. The system of claim 10 wherein processor is configured to convert the inputted coordinates for the corners of the eyes and mouth into estimated coordinates in a common plane by minimizing the error between the inputted corners of the eyes and mouth coordinates and the estimated coordinates corners of the eyes and mouth using a least square estimation of the common plane coordinates of the corners of eyes and mouth.

15. The system of claim 14 wherein the least square estimation of the common plane coordinates of the corners of eyes and mouth is computed using a non-linear Gaussian Least Square Differential Correction algorithm.

16. The system of claim 15 wherein the Gaussian Least Square Differential Correction algorithm comprises a least square model function ƒ

(V_p) that is computed using

ƒ

₁(V_p)=α

[1,0,0]T P₁

ƒ

₂(V_p)=α

[0,1,0]T P₁

ƒ

₃(V_p)=α

[1,0,0]T P₂

ƒ

₄(V_p)=α

[0,1,0]T P₂

ƒ

₅(V_p)=α

[1,0,0]T P₃

ƒ

₆V_p)=α

[0,1,0]T P₃

ƒ

₇(V_p)=α

[1,0,0]T P₄

ƒ

₈(V_p)=α

[0,1,0]T P₄

ƒ

₉(V_p)=α

[1,0,0]T P₅

ƒ

₁₀(V_p)=α

[0,1,0]T P₅

ƒ

₁₁(V_p)=α

[1,0,0]T P₆

ƒ

₁₂(V_p)=α

[0,1,0]T P₆

where T correlates to the head orientation θ

, φ

, ψ and

the matrix T=T_θ

·

T_φ

·

T_ψ

where

17. The system of claim 10 wherein the facial image representations stored in the database comprise one of facial images of suspected individuals, persons convicted of crimes, driver'"'"'s license images of general public, criminals for criminal detection, persons for airline screening, persons for access control, persons for a surveillance system, persons for identification of dead bodies, and terrorists on a watch list.