User authentication using audiovisual synchrony detection

• US 10,559,312 B2
• Filed: 08/25/2016
• Issued: 02/11/2020
• Est. Priority Date: 08/25/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Aronowitz, Hagai, Aides, Amit
• Primary Examiner(s): Pham, Luu T
• Assistant Examiner(s): Wilcox, James J
• Application Number:
  US15/246,574
• Publication Number:
  US 20180063106A1
• Time in Patent Office: 1,265 Days
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/803 of input or preprocessed data
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/40 Spoof detection, e.g. liveness detection
• CPC Class Codes: G10L 17/10 Multimodal systems, i.e. based on the integration of multiple recognition engines or fusion of expert systems
• CPC Class Codes: G10L 17/24 the user being prompted to utter a password or a predefined phrase
• CPC Class Codes: G10L 25/24 the extracted parameters being the cepstrum
• CPC Class Codes: G10L 25/57 for processing of video signals
• CPC Class Codes: H04L 63/08 for authentication of entities cryptographic mechanisms or cryptographic arrangements for entity authentication H04L9/32
• CPC Class Codes: H04L 63/0861 using biometrical features, e.g. fingerprint, retina-scan cryptographic mechanisms or cryptographic arrangements for entity authentication using biological data H04L9/3231

Abstract

Methods, computing systems and computer program products implement embodiments of the present invention that include receiving, at a first time, first video and first audio signals generated in response to a user uttering a passphrase, and receiving, at a second time subsequent to the first time, second video and second audio signals generated in response the user uttering the passphrase. Upon computing an audio temporal alignment between the first and the second audio signals and computing a video temporal alignment between the first and the second video signal, the user can be authenticated by comparing the audio temporal alignment to the video temporal alignment.

Claims

1. A method for preventing a replay attack, comprising:

receiving, at a first time, first video and first audio signals generated in response to a user uttering a passphrase;

receiving, at a second time subsequent to the first time, second video and second audio signals generated respectively by a camera and a microphone in response the user uttering the passphrase;

extracting, from the received audio signals, speech-based features;

extracting, from the received video signals, visual-based features;

computing, by a processor, an audio temporal alignment between the first and the second audio signals, by computing a dynamic time warping on the audio-based features extracted from the first and second audio signals, the audio temporal alignment comprising a first registration that synchronizes the first and the second audio signals;

computing, by the processor, a video temporal alignment between the first and the second video signals, by computing a dynamic time warping on the video-based features extracted from the first and second video signals, the video temporal alignment comprising a second registration that synchronizes the first and the second video signals;

comparing the audio temporal alignment between the first and the second audio signals to the video temporal alignment between the first and the second video signals; and

successfully authenticating the user upon detecting, as a result of the comparing, that the audio and the video temporal alignments are synchronized; and

failing the authentication of the user upon detecting, as a result of the comparison, that the audio and the video temporal alignments are not synchronized.

2. The method according to claim 1, wherein the speech-based features comprise mel-frequency cepstral coefficients.

3. The method according to claim 1, wherein computing the visual-based features comprises computing a local binary pattern analysis.

4. The method according to claim 1, wherein the visual-based features comprise a histogram of oriented gradients.

5. The method according to claim 1, wherein computing the visual-based features comprises performing a region of interest analysis.

6. The method according to claim 1, wherein comparing the audio temporal alignment between the first and the second audio signals to the video temporal alignment between the first and the second video signals comprises projecting the audio temporal alignment on a video domain comprising the first and the second video signals and comparing the projected audio temporal alignment to the video temporal alignment.

7. The method according to claim 1, wherein comparing the audio temporal alignment between the first and the second audio signals to the video temporal alignment between the first and the second video signals comprises projecting the video temporal alignment on an audio domain comprising the first and second audio signals and comparing the projected video temporal alignment to the audio temporal alignment.

8. An apparatus for preventing a replay attack, comprising:

a microphone;

a display configured to present a passphrase; and

a processor configured;

to receive, at a first time, first video and first audio signals generated in response to a user uttering the passphrase,

to receive, at a second time subsequent to the first time, second video and second audio signals generated respectively by a camera and the microphone in response to the user uttering the passphrase, to extract, from the received audio signals, speech-based features,

to extract, from the received video signals, visual-based features,

to compute an audio temporal alignment between the first and the second audio signals, by computing a dynamic time warping on the audio-based features extracted from the first and second audio signals, the audio temporal alignment comprising a first registration that synchronizes the first and the second audio signals,

to compute a video temporal alignment between the first and the second video signals, by computing a dynamic time warping on the video-based features extracted from the first and second video signals, the video temporal alignment comprising a second registration that synchronizes the first and the second video signals,

to compare the audio temporal alignment between the first and the second audio signals to the video temporal alignment between the first and the second video signals, and to successfully authenticate the user upon detecting, as a result of the comparing, that the audio and video temporal alignments are synchronized, and

to fail the authentication of the user upon detecting, as a result of the comparison that the audio and the video temporal alignments are not synchronized.

9. The apparatus according to claim 8, wherein the speech-based features comprise mel-frequency cepstral coefficients.

10. The apparatus according to claim 8, wherein the processor is configured to compute the visual-based features by computing a local binary pattern analysis.

11. The apparatus according to claim 8, wherein the visual-based features comprise a histogram of oriented gradients.

12. A computer program product for preventing a replay attack, the computer program product comprising:

a non-transitory computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising;

computer readable program code configured to receive, at a first time, first video and first audio signals generated in response to a user uttering a passphrase;

computer readable program code configured to receive, at a second time subsequent to the first time, second video and second audio signals generated respectively by a camera and a microphone in response the user uttering the passphrase;

computer readable program code configured to extract, from the received audio signals, speech-based features;

computer readable program code configured to extract, from the received video signals;

computer readable program code configured to compute an audio temporal alignment between the first and the second audio signals, by computing a dynamic time warping on the audio-based features extracted from the first and second audio signals, the audio temporal alignment comprising a first registration that synchronizes the first and the second audio signals;

computer readable program code configured to compute, by the processor, a video temporal alignment between the first and the second video signals, by computing a dynamic time warping on the video-based features extracted from the first and second video signals, the video temporal alignment comprising a second registration that synchronizes the first and the second video signals; and

computer readable program code configured to compare the audio temporal alignment between the first and the second audio signals to the video temporal alignment between the first and the second video signals; and

computer readable program code configured to successfully authenticate the user upon detecting, as a result of the comparing, that the audio and the video temporal alignments are synchronized; and

computer readable program code configured to fail the authentication of the user upon detecting, as a result of the comparison that the audio and the video temporal alignments are not synchronized.

13. The computer program product according to claim 12, wherein the computer readable program code is configured to compute the visual-based features by computing a local binary pattern analysis.