Using facial data for device authentication or subject identification

• US 9,652,663 B2
• Filed: 07/02/2015
• Issued: 05/16/2017
• Est. Priority Date: 07/12/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Aly, Alaa Abdel-Hakim, El Baz, Ahmed, Lau, James Kai Yu, Kaheel, Ayman, El-Saban, Motaz, Shawky, Mohamed, Gonzalez, Monica, Deif, Tamer
• Primary Examiner(s): Lim, Steven
• Assistant Examiner(s): Haile, Benyam
• Application Number:
  US14/791,055
• Publication Number:
  US 20150310259A1
• Time in Patent Office: 684 Days
• Field of Search: 340 582
• CPC Class Codes: G06F 21/31 User authentication
• CPC Class Codes: G06F 21/32 using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06V 10/422 for representing the structure of the pattern or shape of an object therefor
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/169 Holistic features and representations, i.e. based on the facial image taken as a whole
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/40 Spoof detection, e.g. liveness detection
• CPC Class Codes: G07C 9/00 Individual registration on entry or exit
• CPC Class Codes: G09C 5/00 Ciphering apparatus or methods not provided for in the preceding groups, e.g. involving the concealment or deformation of graphic data such as designs, written or printed messages
• CPC Class Codes: H04N 23/74 by influencing the scene brightness using illuminating means

Abstract

Exemplary methods, apparatus, and systems are disclosed for authenticating a user to computing device. In one exemplary embodiment, an indication of a request by a user to unlock a mobile device in a locked state is received. One or more images of the face of the user are captured. Facial components of the user from the one or more captured images are extracted. A determination is made as to whether the user is an authorized user or a non-authorized user based at least in part on a comparison of the facial components of the user extracted from the one or more captured images to facial components of the authorized user from one or more authentication images of the authorized stored on the mobile device. If the user is determined to be the authorized user, the mobile device unlocked; otherwise, the mobile device is maintained in its locked state.

Claims

1. A mobile device having a processor and memory, the mobile device being configured to perform operations comprising:

receiving an indication of a request by a user to unlock the mobile device from a locked state;

capturing one or more images of the face of the user as the face of the user is illuminated by a light source;

extracting facial components of the user from the one or more captured images;

determining whether the user is an authorized user based at least in part on a comparison of the facial components of the user extracted from the one or more captured images to facial components of the authorized user from at least one of two or more authentication images of the authorized user stored on the mobile device;

unlocking the mobile device if the user is determined to be the authorized user; and

maintaining the locked state of the mobile device if the user is determined not to be the authorized user,

wherein the capturing includes prompting the user to make a randomly selected facial expression or head motion, and

wherein the two or more authentication images include at least one authentication image of the user making the facial expression or head motion that is randomly selected.

2. The mobile device of claim 1, further comprising, in an enrollment phase in which the two or more authentication images are created, capturing a series of images of the user making different facial expressions or head motions, the user being prompted to make each different facial expression or head motion during the enrollment phase.

3. The mobile device of claim 1, wherein the capturing comprises capturing two or more images of the face of the user in succession, and

wherein the determining comprises determining whether the user is the authorized user based at least in part on a comparison of the facial components of the user extracted from the two or more captured images to the facial components of the authorized user from the two or more authentication images stored on the mobile device.

4. The mobile device of claim 1, wherein the capturing comprises capturing a plurality of consecutive frames of the user as the user is making a head movement, and

wherein the operations further comprise performing a detection technique using the plurality of consecutive frames to determine that the face of the user is a real face and not a two-dimensional representation of the face of the user.

5. The mobile device of claim 4, wherein the detection technique determines whether the face of the user is a real face and not a two-dimensional representation of the face of the user by, in part, estimating an affine transform between the consecutive frames.

6. The mobile device of claim 4, wherein the detection technique determines whether the face of the user is a real face and not a two-dimensional representation of the face of the user by a process comprising:

detecting the face of the user in the consecutive frames;

tracking one or more feature points on the detected face over the consecutive frames;

estimating affine transforms between respective ones of the consecutive frames;

measuring residuals between the affine transform estimates; and

determining whether the face is real or fake based on the residuals.

7. A mobile device having a processor and memory, the mobile device being configured to perform operations comprising:

receiving an indication of a request by a user to unlock a computing device in a locked state;

capturing one or more images of the face of the user as the face of the user is illuminated by a light source;

extracting facial components of the user from the one or more captured images;

determining whether the user is an authorized user based at least in part on a comparison of the facial components of the user extracted from the one or more captured images to facial components of the authorized user from two or more authentication images of the authorized user stored on the mobile device, wherein the two or more authentication images are taken under substantially different lighting intensities, at least one image of the user being taken in a low-lighting situation;

unlocking the mobile device if the user is determined to be the authorized user; and

maintaining the locked state of the mobile device if the user is determined not to be the authorized user,

wherein the capturing comprises capturing a plurality of consecutive frames of the user as the user is making a head movement,

wherein the method further comprises performing a detection technique using the plurality of consecutive frames to determine that the face of the user is a real face and not a two-dimensional representation of the face of the user,

the method further comprising pre-processing the one or more captured images to compensate for lighting conditions in which the one or more images are captured, wherein the pre-processing comprises downsampling the one or more images of the face of the user to a lower resolution, and wherein the extracting and determining are performed using the downsampled one or more images.

8. The mobile device of claim 7, wherein the detection technique determines whether the face of the user is a real face and not a two-dimensional representation of the face of the user by, in part, estimating an affine transform between the consecutive frames.

9. The mobile device of claim 7, wherein the detection technique determines whether the face of the user is a real face and not a two-dimensional representation of the face of the user by a process comprising:

detecting the face of the user in the consecutive frames;

tracking one or more feature points on the detected face over the consecutive frames;

estimating affine transforms between respective ones of the consecutive frames;

measuring residuals between the affine transform estimates; and

determining whether the face is real or fake based on the residuals.

10. The mobile device of claim 7, wherein the comparison is performed by generating feature descriptors representative of the facial components of the user in the one or more captured images, and determining a difference between the feature descriptors of the user in the one or more captured images and respective facial descriptors of the authorized user generated from the two or more authentication images.

11. The mobile device of claim 7, wherein the comparison is performed by generating a graph-based representation of the facial components of the user in the one or more captured images, and determining a difference between the graph-based representation of the facial components of the user and respective graph-based representations of the authorized user generated from the two or more authentication images,

the graph-based representation of the facial components of the user being indicative of appearances and relative geometric relationships of the facial components of the user.

12. The mobile device of claim 7, wherein the determining comprises:

if the facial components of the user extracted from the one or more captured images match the facial components of the authorized user from the two or more authentication images, generating an indication that the user is authorized; and

if the facial components of the user extracted from the one or more captured images do not match the facial components of the authorized user from the two or more authentication images, causing a secondary authentication method to be performed on the mobile device.

13. A method, comprising:

receiving an indication of a request by a user to unlock a mobile device in a locked state;

capturing one or more images of the face of the user as the face of the user is illuminated by a light source, wherein the capturing is facilitated through an image capture screen of the mobile device in which a first portion of the image capture screen is dedicated to providing the light source that illuminates the face of the user and a second portion of the image capture screen displays an image currently being received by a camera of the mobile device;

extracting facial components of the user from the one or more captured images;

determining whether the user is an authorized user based at least in part on a comparison of the facial components of the user extracted from the one or more captured images to one or more facial components of the authorized user obtained respectively from one or more authentication images of the authorized user stored on the mobile device;

unlocking the mobile device if the user is determined to be the authorized user; and

maintaining the locked state of the mobile device if the user is determined not to be the authorized user,

the method further comprising pre-processing the one or more captured images to compensate for lighting conditions in which the one or more images are captured, wherein the pre-processing comprises downsampling the one or more images of the face of the user to a lower resolution, and wherein the extracting and determining are performed using the downsampled one or more images.

14. The method of claim 13, wherein at least one of the authentication images is an image of the user in which the user'"'"'s face is illuminated by the image capture screen of the mobile device in which the first portion of the image capture screen is dedicated to providing the light source that illuminates the face of the user and the second portion of the image capture screen displays an image currently being received by the camera of the mobile device.

15. The method of claim 13, wherein the pre-processing further comprises performing one or more of white balancing, global exposure compensation, or local exposure compensation of the one or more captured images.

16. The method of claim 13, wherein the comparison is performed by generating feature descriptors representative of the facial components of the user in the one or more captured images, and determining a difference between the feature descriptors of the user in the one or more captured images and respective facial descriptors of the authorized user generated from the two or more authentication images.

17. The method of claim 13, wherein the comparison is performed by generating a graph-based representation of the facial components of the user in the one or more captured images, and determining a difference between the graph-based representation of the facial components of the user and respective graph-based representations of the authorized user generated from the two or more authentication images,

the graph-based representation of the facial components of the user being indicative of appearances and relative geometric relationships of the facial components of the user.

18. The method of claim 13, wherein the capturing comprises capturing a plurality of consecutive frames of the user as the user is making a head movement, and

wherein the method further comprises performing a detection technique using the plurality of consecutive frames to determine that the face of the user is a real face and not a two-dimensional representation of the face of the user.

19. The method of claim 13, wherein the image capture screen further includes a touch screen button for triggering the capturing.

20. A mobile device having a processor and memory, the mobile device being configured to perform operations comprising:

receiving an indication of a request by a user to unlock a computing device in a locked state;

capturing one or more images of the face of the user as the face of the user is illuminated by a light source;

extracting facial components of the user from the one or more captured images;

determining whether the user is an authorized user based at least in part on a comparison of the facial components of the user extracted from the one or more captured images to facial components of the authorized user from two or more authentication images of the authorized user stored on the mobile device, wherein the two or more authentication images are taken under substantially different lighting intensities, at least one image of the user being taken in a low-lighting situation;

unlocking the mobile device if the user is determined to be the authorized user;

maintaining the locked state of the mobile device if the user is determined not to be the authorized user,

wherein the capturing comprises capturing a plurality of consecutive frames of the user as the user is making a head movement,

wherein the method further comprises performing a detection technique using the plurality of consecutive frames to determine that the face of the user is a real face and not a two-dimensional representation of the face of the user,

wherein the capturing further comprises prompting the user to make a randomly selected facial expression or head motion, and

wherein the two or more authentication images include at least one authentication image of the user making the facial expression or head motion that is randomly selected.