AUGMENTED REALITY IDENTITY VERIFICATION

• US 20170351909A1
• Filed: 06/01/2017
• Published: 12/07/2017
• Est. Priority Date: 06/03/2016
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Kaehler, Adrian
• Granted Patent: US 10,534,954 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/2148 characterised by the process organisation or structure, e.g. boosting cascade
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 30/413 Classification of content, e.g. text, photographs or tables
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

An augmented reality device (ARD) can present virtual content which can provide enhanced experiences with the user'"'"'s physical environment. For example, the ARD can detect a linkage between a person in the FOV of the ARD and a physical object (e.g., a document presented by the person) or detect linkages between the documents. The linkages may be used in identity verification or document verification.

Claims

1. An augmented reality (AR) system for detecting a linkage in an AR environment, the augmented reality system comprising:

an outward-facing imaging system configured to image an environment of the AR system;

an AR display configured to present virtual content in a three-dimensional (3D) view to a user of the AR system; and

a hardware processor programmed to;

obtain, with the outward-facing imaging system, an image of the environment;

detect a first face and a second face in the image, wherein the first face is the face of a person in the environment and wherein the second face is a face on an identification document;

recognize the first face based on first facial features associated with the first face;

recognize the second face based on the second facial features;

analyze the first facial features and the second facial features to detect a linkage between the person and the identification document; and

instruct the AR display to present a virtual annotation indicating a result of the analysis of the first facial features and the second facial features.

2. The AR system of claim 1, wherein to detect the first face and the second face, the hardware processor is programmed to apply at least one of the following algorithms on the image:

a wavelet-based boosted cascade algorithm or a deep neural network algorithm.

3. The AR system of claim 1, wherein the hardware processor is further programmed to:

detect that the second face is the face on the identification document by analyzing a movement of the second face; and

determine whether the movement is described by a single planar homography.

4. The AR system of claim 1, wherein to recognize the first face or the second face, the hardware processor is programmed to:

calculate a first feature vector associated with the first face based at least partly on the first facial features or calculate a second feature vector associated with the second face based at least partly on the second facial features, respectively, by applying at least one of;

a facial landmark detection algorithm, a deep neural network algorithm, or a template matching algorithm.

5. The AR system of claim 4, wherein to detect the linkage between the person and the identification document, the hardware processor is programmed to:

calculate a distance between the first feature vector and the second feature vector,

compare the distance to a threshold value; and

detect the linkage in response to a determination that the distance passes the threshold value.

6. The AR system of claim 5, wherein the distance is a Euclidean distance.

7. The AR system of claim 1, wherein the identification document has a label comprising one or more of the following:

a quick response code, a bar code, or an iris code.

8. The AR system of claim 7, wherein the hardware processor is further programmed to:

identify the label from the image of the environment; and

access an external data source using the label to retrieve biometric information of the person.

9. The AR system of claim 1, wherein AR system further comprises an optical sensor configured to illuminate light outside of a human visible spectrum (HVS), and the hardware processor is further programmed to:

instruct the optical sensor to illuminate the light toward the identification document to reveal hidden information in the identification document;

analyze an image of the identification document wherein the image is acquired when the identification document is illuminated with the light; and

extract biometric information from the image, wherein the extracted biometric information is used to detect the linkage between the person and the identification document.

10. The AR system of claim 1, wherein the hardware processor is programmed to calculate a likelihood of a match between the first facial features and the second facial features.

11. The AR system of claim 1, wherein the annotation comprises a visual focus indicator linking the person and the identification document.

12. A method for detecting a linkage in an augmented reality environment, the method comprising:

under control of an augmented reality device comprising an outward-imaging imaging system and a hardware processor, the augmented reality device configured to display virtual content to a wearer of the augmented reality device;

obtaining an image of the environment;

detecting a person, a first document, and a second document in the image;

extracting first personal information based at least partly on an analysis of the image of the first document;

accessing second personal information associated with second document;

extracting third personal information of the person based at least partly on an analysis of the image of the person, wherein the first personal information, the second personal information, and the third personal information are in a same category;

determining a likelihood of match among the first personal information, the second personal information, and the third personal information; and

displaying a linkage of among the first document, the second document, and the person in response to a determination that the likelihood of match exceeds a threshold condition.

13. The method of claim 12, wherein obtaining the image of the environment comprises accessing the image acquired by the outward-facing imaging system of the augmented reality device.

14. The method of claim 12, wherein extracting the first personal information and the third personal information comprises:

detecting a first face in the image, wherein the first face is included in the first document;

detecting a second face in the image, wherein the second face is associated with the person in the environment;

identifying first facial features associated with the first face and second facial features associated with the second face; and

recognizing the first face and the second face based on the first facial features and the second facial features respectively.

15. The method of claim 14, wherein detecting the first face or detecting the second face comprises applying:

a wavelet-based boosted cascade algorithm or a deep neural network algorithm.

16. The method of claim 14, wherein recognizing the first face and recognizing the second face comprises:

calculating a first feature vector associated with the first face based at least partly on the first facial features; and

calculating a second feature vector associated with the second face based at least partly on the second facial features, respectively, by applying at least one of;

a facial landmark detection algorithm, a deep neural network algorithm, or a template matching algorithm.

17. The method of claim 12, wherein accessing the second personal information comprises:

acquiring an image of the second document when a light is shed onto the second document and wherein at least a portion of the light is outside of human visible spectrum; and

identifying the second personal information based on the acquired image of the second document, wherein the second personal information is not directly visible to the human under a normal optical condition.

18. The method of claim 12, wherein accessing the second personal information comprises:

identifying the label from the image of the environment; and

accessing a data source storing personal information of a plurality of persons using the label to retrieve biometric information of the person.

19. The method of claim 12, wherein determining a likelihood of match comprises:

comparing the first personal information and the second personal information;

calculating a confidence score based at least in part on the similarities or dissimilarities between the first personal information and the second personal information.

20. The method of claim 19, further comprising:

displaying a virtual annotation indicating at least one of the first document or the second document as valid in response to a determination that the confidence score exceeds a threshold value.