Facial recognition

• US 8,457,367 B1
• Filed: 06/26/2012
• Issued: 06/04/2013
• Est. Priority Date: 06/26/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Sipe, Michael Andrew, Schneiderman, Henry Will, Nechyba, Michael Christian
• Primary Examiner(s): Werner, Brian P
• Application Number:
  US13/533,811
• Time in Patent Office: 343 Days
• Field of Search: None
• US Class Current: 382/118
• CPC Class Codes: G06F 21/32 using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/40 Spoof detection, e.g. liveness detection
• CPC Class Codes: G06V 40/45 Detection of the body part being alive

Abstract

An example method includes receiving first and second images of a face of a user, where one or both images have been granted a match by facial recognition. The method includes identifying at least one facial landmark in the first image and at least one corresponding facial landmark in the second image, and extracting a first sub-image from the first image, where the first sub-image includes a representation of the at least one facial landmark. The method includes extracting a second sub-image from the second image, where the second sub-image includes a representation of the at least one corresponding facial landmark, detecting a facial gesture by determining whether a sufficient difference exists between the second sub-image and first sub-image to indicate the facial gesture, and determining, based on detecting the facial gesture, whether to deny authentication to the user with respect to accessing functionalities controlled by the computing device.

Claims

1. A method comprising:

receiving, from an image capture device coupled to a computing device, a first image of a face of a user, and a second image of the face of the user;

identifying at least one facial landmark in the first image and at least one corresponding facial landmark in the second image;

extracting, by the computing device, a first sub-image from the first image, wherein the first sub-image represents a portion of the first image, and wherein the first sub-image includes a representation of the at least one facial landmark;

extracting, by the computing device, a second sub-image from the second image, wherein the second sub-image represents a portion of the second image, and wherein the second sub-image includes a representation of the at least one corresponding facial landmark;

detecting a facial gesture by determining, by the computing device, whether a sufficient difference exists between the second sub-image and the first sub-image to indicate the facial gesture;

determining, by the computing device, whether the first image of the face of the user and the second image of the face of the user are associated with a consistent user identity, based at least in part on determining whether a sufficient similarity exists between the first image of the face of the user and the second image of the face of the user; and

determining, based at least in part on detecting the facial gesture and on determining whether the first image of the face of the user and the second image of the face of the user are associated with the consistent user identity, whether to deny authentication to the user with respect to accessing one or more functionalities controlled by the computing device.

2. The method of claim 1, further comprising:

prompting the user to perform a predetermined facial gesture;

determining whether the detected facial gesture matches the predetermined facial gesture;

if the detected facial gesture does not match the predetermined facial gesture, denying authentication to the user with respect to accessing one or more functionalities controlled by the computing device; and

if the detected facial gesture matches the predetermined facial gesture, granting authentication to the user with respect to accessing one or more functionalities controlled by the computing device.

3. The method of claim 1, wherein the at least one facial landmark comprises at least one of an eye, an eyebrow, a mouth area, a forehead area, and a nose, and wherein the indicated facial gesture includes at least one of a blink gesture, a wink gesture, an ocular movement, a smile gesture, a frown gesture, a tongue protrusion gesture, an open mouth gesture, an eyebrow movement, a forehead wrinkle gesture, and a nose wrinkle gesture.

4. The method of claim 1,

wherein the image capture device has video capture capabilities, and

wherein at least one of the first image and the second image form a portion of video data received from the image capture device.

5. The method of claim 1, further comprising:

performing facial recognition analysis on each of the first image and the second image;

if at least one of the first image and second image fail the facial recognition analysis, denying authentication to the user with respect to accessing one or more functionalities controlled by the computing device; and

if both of the first image and the second image pass the facial recognition analysis and the facial gesture is detected, granting authentication to the user with respect to accessing one or more functionalities controlled by the computing device.

6. The method of claim 1, further comprising:

adjusting one or both of the first sub-image and the second sub-image to obtain a standard normalized correlation of the first sub-image and the second sub-image.

7. The method of claim 6, wherein adjusting one or both of the first sub-image and the second sub-image comprises at least one of 1) rotating one or both of the first sub-image and the second sub-image, and 2) scaling one or both of the first sub-image and the second sub-image.

8. The method of claim 7, wherein scaling one or both of the first sub-image and the second sub-image is associated with correcting an image size discrepancy between the first sub-image and the second sub-image.

9. The method of claim 1, wherein receiving the first image and the second image further comprises selecting the first image and the second image from a plurality of images that includes at least the first image, the second image, and a third image.

10. The method of claim 9, wherein selecting the first image and the second image from the plurality of images comprises selecting based at least in part on the detected facial gesture.

11. The method of claim 1, further comprising:

receiving the first image and second image as frames of a video stream;

determining whether the video stream has a sufficiently high frame rate to indicate continuous frame capture;

determining whether each frame of the video stream includes a representation of a human face;

if at least one frame of the video stream does not include the representation of the human face, denying authentication to the user with respect to accessing one or more functionalities controlled by the computing device; and

if each of the first image and the second image includes the representation of the human face, granting authentication to the user with respect to accessing one or more functionalities controlled by the computing device.

12. The method of claim 1, wherein determining whether a sufficient similarity exists between the first image and the second image to indicate a consistent user identity further comprises at least one of:

1) performing facial recognition analysis on both of the first image and the second image, and

     2) comparing one or more characteristics of the first image to one or more corresponding characteristics of the second image, and

     3) determining a similarity score based on the comparison, wherein the sufficient similarity is determined to exist when the similarity score exceeds a similarity score threshold.

13. The method of claim 1, wherein the indicated facial gesture is a first facial gesture, the method further comprising:

receiving, from the image capture device, a third image of the face of the user, and a fourth image of the face of the user;

extracting a third sub-image from the third image and a fourth sub-image from the fourth image, wherein the third sub-image includes a representation of a supplementary facial landmark and the fourth sub-image includes a representation of a corresponding supplementary facial landmark; and

if the sufficient difference exists between the second sub-image and the first sub-image to indicate the first facial gesture;

detecting a second facial gesture based at least in part on determining whether a sufficient difference exists between the third sub-image and the fourth sub-image to indicate the second facial gesture,

wherein determining whether to deny authentication to the user comprises determining, based at least in part on the detected first facial gesture and the detected second facial gesture, whether to deny authentication to the user with respect to accessing one or more functionalities controlled by the computing device.

14. The method of claim 12, further comprising:

prompting the user to perform a series of facial gestures comprising the first facial gesture and the second facial gesture.

15. The method of claim 13, wherein the at least one landmark shares a common area with the supplementary facial landmark.

16. The method of claim 13, wherein the at least one landmark shares no common area with the supplementary facial landmark.

17. The method of claim 1, wherein the second sub-image is larger than the first sub-image extracted from the first image.

18. The method of claim 1, further comprising:

detecting a liveness gesture based on at least one of a yaw angle of the second image relative to the first image and a pitch angle of the second image relative to the first image, wherein the yaw angle corresponds to a transition centered around a vertical axis, and wherein the pitch angle corresponds to a transition centered around a horizontal axis;

generating a liveness score based on at least one of a yaw angle magnitude associated with the yaw angle and a pitch angle magnitude associated with the pitch angle; and

comparing the liveness score to a threshold value,

wherein determining whether to deny authentication to the user with respect to accessing one or more functionalities controlled by the computing device is further based on the comparison.

19. The method of claim 1, further comprising:

receiving, from the image capture device, a third image of the face of the user and a fourth image of the face of the user;

detecting a liveness gesture based on at least one of a yaw angle of the fourth image relative to the third image and a pitch angle of the fourth image relative to the third image, wherein the yaw angle corresponds to a transition centered around a vertical axis, and wherein the pitch angle corresponds to a transition centered around a horizontal axis;

generating a liveness score based on at least one of a yaw angle magnitude associated with the yaw angle and a pitch angle magnitude associated with the pitch angle; and

comparing the liveness score to a threshold value,

wherein determining whether to deny authentication to the user with respect to accessing one or more functionalities controlled by the computing device is further based on the comparison.

20. The method of claim 18, wherein determining whether to deny authentication to the user with respect to accessing one or more functionalities controlled by the computing device is further based on an order associated with the liveness gesture and the facial gesture.

21. The method of claim 1, further comprising:

determining whether motion blur exists in either the first image of the face of the user or the second image of the face of the user;

and if the motion blur exists in either the first image of the face of the user or the second image of the face of the user, disregarding both of the first image of the face of the user or the second image of the face of the user for purposes of detecting the facial gesture.

24. The method of claim 1, wherein determining whether the first image of the face and the second image of the face are each associated with a common user identity comprises determining based at least in part on comparing one or more characteristics of the first image to one or more corresponding characteristics of the second image.

25. The method of claim 24, wherein comparing one or more characteristics of the first image to one or more corresponding characteristics of the second image comprises comparing one or more of iris color, dimension of an eye, dimension of a nose, dimension of a mouth, hair color, and dimension of a head.

22. A computing device comprising:

a memory,

at least one processor;

at least one image capture device operable by the at least one processor to capture a first image of a face of a user, and a second image of the face of the user;

a facial recognition module operable by the at least one processor to grant a match, by facial recognition, to at least one of the first image and the second image;

a landmark detection module operable by the at least one processor to identify at least one facial landmark in the first image and at least one corresponding facial landmark in the second image;

a sub-image extraction module operable by the at least one processor to;

extract a first sub-image from the first image, wherein the first sub-image represents a portion of the first image, and wherein the first sub-image includes a representation of the at least one facial landmark, and

extract a second sub-image from the second image, wherein the second sub-image represents a portion of the second image, and wherein the second sub-image includes a representation of the at least one corresponding facial landmark; and

an anti-spoofing module operable by the at least one processor to;

detect a facial gesture by determining whether a sufficient difference exists between the second sub-image and the first sub-image to indicate the facial gesture,

determine whether the first image of the face and the second image of the face are associated with a consistent user identity, based at least in part on determining whether a sufficient similarity exists between the first image of the face of the user and the second image of the face of the user, and

determine, based at least in part on detecting the facial gesture and on determining whether the first image of the face and the second image of the face are associated with the consistent user identity, whether to deny authentication to the user with respect to accessing one or more functionalities controlled by the computing device.

23. A computer-readable storage device encoded with instructions that, when executed, cause one or more processors of a computing device to perform operations comprising:

receiving, from an image capture device coupled to the computing device, a first image of a face of a user, and a second image of the face of the user;

identifying at least one facial landmark in the first image and at least one corresponding facial landmark in the second image;

extracting, by the computing device, a first sub-image from the first image, wherein the first sub-image represents a portion of the first image, and wherein the first sub-image includes a representation of the at least one facial landmark;

extracting, by the computing device, a second sub-image from the second image, wherein the second sub-image represents a portion of the second image, and wherein the second sub-image includes a representation of the at least one corresponding facial landmark;

detecting a facial gesture by determining, by the computing device, whether a sufficient difference exists between the second sub-image and the first sub-image to indicate the facial gesture;

determining whether the first image of the face and the second image of the face are associated with a consistent user identity, based at least in part on determining whether a sufficient similarity exists between the first image of the face of the user and the second image of the face of the user; and

determining, based at least in part on detecting the facial gesture and on determining whether the first image of the face and the second image of the face are associated with the consistent user identity, whether to deny authentication to the user with respect to accessing one or more functionalities controlled by the computing device.