Facial signature methods, systems and software

• US 10,853,625 B2
• Filed: 05/12/2016
• Issued: 12/01/2020
• Est. Priority Date: 03/21/2015
• Status: Active Grant

• Current Assignee: Mine One GmbH
• Original Assignee: Mine One GmbH
• Inventors: McCombe, James A, Herken, Rolf, Smith, Brian W.
• Primary Examiner(s): Yang, Jianxun
• Application Number:
  US15/573,475
• Publication Number:
  US 20180189550A1
• Time in Patent Office: 1,664 Days
• CPC Class Codes: G06V 10/758 Involving statistics of pixels or of feature values, e.g. histogram matching
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/50 Maintenance of biometric data or enrolment thereof

Abstract

Methods, systems and computer program products (“software”) enable a virtual three-dimensional visual experience (referred to herein as “V3D”) in videoconferencing and other applications; the capturing, processing and displaying of images and image streams; and generation of a facial signature based on images of a given human user'"'"'s or subject'"'"'s face, or lace and head, for accurate, reliable identification or authentication of a human user or subject, in a secure, difficult to forge manner.

Claims

1. A method of generating a facial signature for use in identifying a human user or subject, the method comprising:

capturing images of the face of the user or subject for which the facial signature is to be generated, the capturing comprising utilizing at least one camera having a view of the user'"'"'s or subject'"'"'s face;

executing an image rectification function to compensate for optical distortion and alignment of the at least one camera;

executing a feature correspondence function by detecting common features between corresponding images of the user'"'"'s or subject'"'"'s face captured by the at least one camera and measuring a relative distance in image space between the common features, to generate disparity values and a feature correspondence data representation representative of the captured images and the corresponding disparity values; and

utilizing the feature correspondence data representation to generate a facial signature data representation of the user'"'"'s or subject'"'"'s face, the facial signature data representation comprising data representative of the captured images and the corresponding disparity values, and depth information for multiple image elements of a given captured image of the user'"'"'s or subject'"'"'s face, the facial signature data representation being usable to accurately identify the user or subject in a secure, difficult to forge manner.

2. The method of claim 1 wherein:

the capturing comprises utilizing at least first and second cameras, each having a view of the user'"'"'s or subject'"'"'s face;

the cameras being arranged along an axis to configure a stereo camera pair; and

executing a feature correspondence function comprises;

detecting common features between corresponding images captured by the respective cameras and measuring a relative distance in image space between the common features, to generate disparity values.

3. The method of claim 2 wherein the facial suture is utilized as a security factor in an authentication system.

4. The method of claim 3 wherein the facial signature is utilized as a security factor in an authentication system in combination with a second security factor, and wherein the second security factor comprises any of a pass-code, a fingerprint or other biometric information.

5. The method of claim 1 wherein:

the capturing comprises utilizing at least one camera having a view of the user'"'"'s or subject'"'"'s face and which is an infra-red time-of-flight camera or structured light camera that directly provides depth information for multiple image elements of a given captured image of the user'"'"'s or subject'"'"'s face; and

the feature correspondence data representation is representative of the captured images and corresponding depth information.

6. The method of claim 5 wherein facial signature is utilized as a security factor in an authentication system.

7. The method of claim 6 wherein the facial signature is utilized as a security factor in an authentication system in combination with a second security factor, and wherein the second security factor comprises any of a pass-code, a fingerprint or other biometric information.

8. The method of claim 1 wherein the facial signature data representation comprises 3D facial contour data, and the 3D facial contour data is utilized in combination with a 2D image from one or more cameras to create a hybrid 3D/2D face identification method.

9. The method of claim 8 wherein 3D facial contour data is utilized to confirm that a face having credible 3D human facial proportions was presented to the cameras at an overlapping spatial location of the captured 2D image.

10. The method of claim 1 wherein 3D facial contour data, stored in the facial signature data representation, is utilized to confirm that a face having credible 3D human facial proportions was presented to the cameras at an overlapping spatial location of the captured 2D image.

11. The method of claim 1 wherein generating a facial signature data representation further comprises executing an enrollment phase, the enrollment phase comprising:

prompting the user or subject to present to the cameras a plurality of selected head movements or positions, or a series of selected facial poses, and collecting image frames from a plurality of head positions or facial poses for use in generating the unique facial signature representative of the user or subject.

12. The method of claim 11 further comprising a matching phase, the matching phase comprising:

utilizing the cameras to capture, over an interval of time, a plurality of frames of 3D and 2D data representative of the user'"'"'s or subject'"'"'s face;

correlating the captured data with the facial signature generated during the enrollment phase, thereby to generate a probability of match score; and

comparing the probability of match score with a selected threshold value, thereby to confirm or deny an identity match.

13. The method of claim 12 further comprising:

during the enrollment phase, generating an enrolled facial signature containing data corresponding to multiple image scans of a user'"'"'s or subject'"'"'s face, the multiple image scans corresponding to a plurality of the user'"'"'s or subject'"'"'s head positions or facial poses; and

during the matching phase, requiring at least a minimum number of captured image frames corresponding to different facial or head positions matching the multiple scans within the enrolled signature.

14. The method of claim 1 wherein:

the capturing comprises utilizing at least three cameras having a view of the scene, wherein a first pair of cameras is disposed along a first axis and second pair of cameras is disposed along a second axis intersecting with, but angularly displaced from, the first axis, and wherein the first and second pairs of cameras represent respective first and second independent stereo axes that share a common camera;

executing a feature correspondence function by detecting common features between corresponding images captured by the at least three cameras and measuring a relative distance in image space between the common features, to generate disparity values; and

generating a data representation, representative of the captured images and the corresponding disparity values.

15. The method of claim 1 wherein the facial signature data representation comprises 3D facial contour data.

16. The method of claim 1 wherein the feature correspondence function utilizes a disparity histogram-based method of integrating data and determining correspondence, the disparity histogram-based method comprising:

constructing a disparity histogram indicating the relative probability of a given disparity value being correct for a given pixel; and

optimizing generation of disparity values on a GPU computing structure, the optimizing comprising;

generating, in the GPU computing structure, a plurality of output pixel threads; and

for each output pixel thread, maintaining a private disparity histogram, in a storage element associated with the GPU computing structure.

17. A method of generating a facial signature for use in identifying a human user or subject, the method comprising:

capturing images of the user'"'"'s face, the capturing comprising utilizing at least one camera having a view of the user'"'"'s or subject'"'"'s face;

executing an image rectification function to compensate for optical distortion and alignment of the at least one camera;

executing a feature correspondence function by detecting common features between corresponding images captured by the at least one camera and measuring a relative distance in image space between the common features, to generate disparity values and a feature correspondence data representation representative of the captured images and the corresponding disparity values; and

utilizing the feature correspondence data representation to generate a facial signature data representation, the facial signature data representation being usable to accurately identify the user or subject in a secure, difficult to forge manner;

wherein the capturing comprises utilizing at least first and second cameras, each having a view of the user'"'"'s or subject'"'"'s face;

the cameras being arranged along an axis to configure a stereo camera pair; and

wherein executing a feature correspondence function comprises;

detecting common features between corresponding images captured by the respective cameras and measuring a relative distance in image space between the common features, to generate disparity values;

generating a disparity solution based on the disparity values; and

applying an injective constraint to the disparity solution based on domain and co-domain, wherein the domain comprises pixels for a given image captured by the first camera and the co-domain comprises pixels for a corresponding image captured by the second camera, to enable correction of error in the disparity solution in response to violation of the injective constraint, wherein the injective constraint is that no element in the co-domain is referenced more than once by elements in the domain.

18. A method of generating a facial signature for use in identifying a human user or subject, the method comprising:

capturing images of the user'"'"'s face, the capturing comprising utilizing at least one camera having a view of the user'"'"'s or subject'"'"'s face;

executing an image rectification function to compensate for optical distortion and alignment of the at least one camera;

executing a feature correspondence function by detecting common features between corresponding images captured by the at least one camera and measuring a relative distance in image space between the common features, to generate disparity values and a feature correspondence data representation representative of the captured images and the corresponding disparity values; and

utilizing the feature correspondence data representation to generate a facial signature data representation, the facial signature data representation being usable to accurately identify the user or subject in a secure, difficult to forge manner;

wherein the capturing comprises utilizing at least three cameras having a view of the scene, wherein a first pair of cameras is disposed along a first axis and second pair of cameras is disposed along a second axis intersecting with, but angularly displaced from, the first axis, and wherein the first and second pairs of cameras represent respective first and second independent stereo axes that share a common camera;

executing a feature correspondence function by detecting, common features between corresponding images captured by the at least three cameras and measuring a relative distance in image space between the common features, to generate disparity values;

generating a data representation, representative of the captured images and the corresponding disparity values; and

utilizing an unrectified, undistorted (URUD) image space to integrate disparity data for pixels between the first and second stereo axes, thereby to combine disparity data from the first and second axes, wherein the URUD space is an image space in which polynomial lens distortion has been removed from the image data but the captured image remains unrectified.

19. A digital processing system for generating a facial signature for use in identifying a human user or subject, the digital processing system comprising at least one camera having a view of the user'"'"'s or subject'"'"'s face, and a digital processing resource comprising at least one digital processor, the digital processing resource being operable to;

capture images of the face of the user or subject for which the facial signature is to be generated, utilizing the at least one camera;

execute an image rectification function to compensate for optical distortion and alignment of the at least one camera;

execute a feature correspondence function by detecting common features between corresponding images of the user'"'"'s or subject'"'"'s face captured by the at least one camera and measuring a relative distance in image space between the common features, to generate disparity values and a feature correspondence data representation representative of the captured images and the corresponding disparity values; and

utilize the feature correspondence data representation to generate a facial signature data representation of the user'"'"'s or subject'"'"'s face, the facial signature data representation comprising data representative of the captured images and the corresponding disparity values and, depth information for multiple image elements of a given captured image of the user'"'"'s or subject'"'"'s face, the facial signature data representation being usable to accurately identify the user or subject in a secure, difficult to forge manner.

20. A non transitory digital processor-readable medium for use with a digital processing system, for generating a facial signature for use in identifying a human user or subject, the digital processing system comprising at least one camera having a view of the user'"'"'s or subject'"'"'s face, and a digital processing resource comprising at least one digital processor, the non-transitory digital processor-readable medium comprising digital processor-executable program instructions stared on the non-transitory processor-readable medium, which when executed in the digital processing resource cause the digital processing resource to:

capture images of the face of the user or subject for which the facial signature is to be generated, utilizing the at least one camera;

execute an image rectification function to compensate for optical distortion and alignment of the at least one camera;

execute a feature correspondence function by detecting common features between corresponding images of the user'"'"'s or subject'"'"'s face captured by the at least one camera and measuring a relative distance in image space between the common features, to generate disparity values and a feature correspondence data representation representative of the captured images and the corresponding disparity values; and

utilize the feature correspondence data representation to generate a facial signature data representation of the user'"'"'s or subject'"'"'s face, the facial signature data representation comprising data representative of the captured images and the corresponding disparity values, and depth information for multiple image elements of a given captured image of the user'"'"'s or subject'"'"'s face, the facial signature data representation being usable to accurately identify the user or subject in a secure, difficult to forge manner.