Method and apparatus for recognizing identity of individuals employing synchronized biometrics

• US 6,219,639 B1
• Filed: 04/28/1998
• Issued: 04/17/2001
• Est. Priority Date: 04/28/1998
• Status: Expired due to Term

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: Bakis, Raimo, Kanevsky, Dimitri, Maes, Stephane Herman
• Primary Examiner(s): Hudspeth, David
• Assistant Examiner(s): Lerner, Martin
• Application Number:
  US09/067,829
• Time in Patent Office: 1,085 Days
• Field of Search: 704/246, 704/247, 704/249, 704/250, 704/236, 704/241, 382/115, 382/116, 382/118, 382/119, 382/120, 382/121, 382/122, 382/124
• US Class Current: 704/246
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G10L 17/10 Multimodal systems, i.e. based on the integration of multiple recognition engines or fusion of expert systems

Abstract

A method for recognizing an individual based on attributes associated with the individual comprises the steps of: pre-storing at least two distinctive attributes of the individual during at least one enrollment session; contemporaneously extracting the at least two distinctive attributes from the individual during a common recognition session; segmenting the pre-stored attributes and the extracted attributes according to a sequence of segmentation units; indexing the segmented pre-stored and extracted attributes so that the segmented pre-stored and extracted attributes corresponding to an identical segmentation unit in the sequence of segmentation units are associated to an identical index; and respectively comparing the segmented pre-stored and extracted attributes associated to the identical index to each other to recognize the individual.

Claims

1. A method for recognizing an individual based on attributes associated with the individual, comprising the steps of:

pre-storing at least two distinctive attributes of the individual during at least one enrollment session;

contemporaneously extracting the at least two distinctive attributes from the individual during a common recognition session;

segmenting the pre-stored attributes and the extracted attributes according to a sequence of segmentation units;

indexing the segmented pre-stored and extracted attributes so that the segmented pre-stored and extracted attributes corresponding to an identical segmentation unit in the sequence of segmentation units are associated to an identical index; and

respectively comparing the segmented pre-stored and extracted attributes associated to the identical index to each other to recognize the individual wherein at least one of the at least two distinctive attributes is lip shape and the comparing step is performed by a lip reading system and a lip recognition system.

2. The method of claim 1, wherein the at least two distinctive attributes are selected from one of biometric and non-biometric attributes.

3. The method of claim 1, wherein at least one of the at least two distinctive attributes is selected from one of face recognition, signature recognition, face temperature infrared pattern, hand geometry, writing instrument velocity, writing instrument pressure, fingerprint, retinal print, a password, a personal identification number (PIN), and personal information.

4. The method of claim 1, wherein the pre-storing step is performed by a server.

5. The method of claim 1, wherein one of the at least two distinctive attributes is voice print and the comparing step is performed by a speaker recognition system.

6. The method of claim 5, wherein the speaker recognition system is a text-dependent speaker recognition system.

7. The method of claim 5, wherein the speaker recognition system is a text-independent speaker recognition system.

8. The method of claim 7, further comprising a labeller for clustering lefemes having comparable lip positions.

9. The method of claim 8, wherein the lip reading system is a text-independent lip reading system.

10. The method of claim 9, wherein the lip reading system verifies if a lip shape extracted for a lefeme matches an expected lip shape of an associated cluster.

11. The method of claim 8, wherein the lip reading system is a text-dependent lip reading system.

12. The method of claim 11, wherein a lip shape corresponding to a lefeme is characterized by a deviation of the geometry relative to a corresponding speaker independent lip shape.

13. The method of claim 12, wherein a substantial match of a speaker dependent lip shape relative to the speaker independent lip shape indicates a positive verification of the individual.

14. The method of claim 1, wherein the comparing step includes comparing features of the lip shape selected from one of static features, dynamic features, and a combination of the aforementioned.

15. The method of claim 8, wherein two of the at least two attributes correspond to audio and video characteristics of the individual, the recognition of the individual depending on the individual correctly speaking a password phrase, the segmenting and indexing steps comprising the steps of:

(a) decoding synchronized utterances of audio and video data corresponding to the individual via application of a joint audio-video joint likelihood function to produce a decoded password phrase W, the joint likelihood function being a combination of likelihood functions for audio and video components;

(b) reducing an effect of the video component of the joint audio-video likelihood function;

(c) decoding the video data utilizing only the audio component of the joint likelihood function to produce a decoded password phrase V;

(d) adjusting the audio component of the joint likelihood function to produce a better match of the decoded password phrase V to the decoded password phrase W;

(e) decoding the synchronized utterances of audio and video data utilizing an iteratively updated joint audio-video likelihood function that includes the video component and the adjusted audio component to produce a new decoded password phrase W′

; and

(f) iteratively repeating steps (b) through (e).

16. The method of claim 15, wherein the adjustment step further comprises the step of comparing the decoded password phrase V obtained via the joint audio-video likelihood function to the audio and video components from a previous iteration to produce a better match to the decoded password phrase W.

17. The method of claim 16, wherein the adjustment step further comprises the step of modifying some parameters of the video component of the joint likelihood function.

18. The method of claim 15, wherein the repeating step repeats steps (b) through (e) until a match between the decoded password phrase W′

and the correct password phrase is established.

19. The method of claim 18, wherein if the match is a full match, then the individual is considered recognized.

20. The method of claim 18, wherein if the match is one of varying and degrading with each iteration, then the individual is considered to be an imposter.

21. The method of claim 18, wherein if the match is one of varying and degrading with each iteration, then the data are not properly indexed.

22. The method of claim 15, wherein the repeating step repeats steps (b) through (e) until the decoded password phrase W′

does not change.

23. The method of claim 15, wherein the audio and video data are produced over small time segments.

24. The method of claim 15, wherein the video component corresponds to likelihood functions for lip contour geometry and mouth region.

25. The method of claim 8, wherein two of the at least two attributes correspond to audio and video characteristics of the individual, the recognition of the individual depending on the individual correctly speaking a password phrase, the segmenting and indexing steps comprising the steps of:

(a) decoding synchronized utterances of audio and video data corresponding to the individual via application of a joint audio-video joint likelihood function to produce a decoded password phrase W, the joint likelihood function being a combination of likelihood functions for audio and video components;

(b) reducing an effect of the audio component of the joint audio-video likelihood function;

(c) decoding the audio data utilizing only the video component of the joint likelihood function to produce a decoded password phrase A;

(d) adjusting the video component of the joint likelihood function to produce a better match of the decoded password phrase A to the decoded password phrase W;

(e) decoding the synchronized utterances of audio and video data utilizing an iteratively updated joint audio-video likelihood function that includes the audio component and the adjusted video component to produce a new decoded password phrase W′

; and

(f) iteratively repeating steps (b) through (e).

26. The method of claim 8, wherein two of the at least two attributes correspond to audio and video characteristics of the individual, the recognition of the individual depending on the individual correctly speaking a password phrase, the segmenting and indexing steps comprising the steps of:

(a) decoding synchronized utterances of audio and video data corresponding to the individual via application of a joint audio-video joint likelihood function to produce a decoded password phrase W, the joint likelihood function being a combination of likelihood functions for audio and video components;

(b) reducing an effect of the video component of the joint audio-video likelihood function;

(c) decoding the video data utilizing only the audio component of the joint likelihood function to produce a decoded password phrase V;

(d) adjusting the audio component of the joint likelihood function to produce a better match of the decoded password phrase V to the decoded password phrase W;

(e) reducing an effect of the audio component of the joint audio-video likelihood function;

(f) decoding the audio data utilizing only the video component of the joint likelihood function to produce a decoded password phrase A;

(g) adjusting the video component of the joint likelihood function to produce a better match of the decoded password phrase A to the decoded password phrase W;

(h) decoding the synchronized utterances of audio and video data utilizing an iteratively updated joint audio-video likelihood function that includes the adjusted audio and video components to produce a new decoded password phrase W′

; and

(i) iteratively repeating steps (b) through (h).

27. The method of claim 26, wherein the repeating step repeats steps (b) through (h) until a match between the decoded password phrase W′

and the correct password phrase is established.

28. The method of claim 26, wherein the repeating step repeats steps (b) through (h) until the decoded password phrase W′

does not change.

29. A method for recognizing an individual based on attributes associated with the individual, comprising the steps of:

pre-storing at least two distinctive attributes of the individual during at least one enrollment session;

contemporaneously extracting the at least two distinctive attributes from the individual during a common recognition session;

segmenting the pre-stored attributes and the extracted attributes according to a sequence of segmentation units;

indexing the segmented pre-stored and extracted attributes so that the segmented pre-stored and extracted attributes corresponding to an identical segmentation unit in the sequence of segmentation units are associated to an identical index; and

respectively comparing the segmented pre-stored and extracted attributes associated to the identical index to each other to recognize the individual wherein three of the at least two distinctive attributes are pen pressure, pen velocity, and hand geometry.

30. An apparatus for recognizing an individual based on attributes associated with the individual, comprising:

a store for pre-storing at least two distinctive attributes of the individual during at least one enrollment session;

contemporaneous extraction means for contemporaneously extracting the at least two distinctive attributes from the individual during a common recognition session;

segmentation means for segmenting the pre-stored attributes and the extracted attributes according to a sequence of segmentation units;

indexing means for indexing the segmented pre-stored and extracted attributes so that the segmented pre-stored and extracted attributes corresponding to an identical segmentation unit in the sequence of segmentation units are associated to an identical index; and

comparing means for respectively comparing the segmented pre-stored and extracted attributes associated to the identical index to each other to recognize the individual, wherein at least one of the at least two distinctive attributes is lip shape and said comparing means comprises a lip reading system and a lip recognition system.

31. The apparatus of claim 30, wherein the at least two distinctive attributes are selected from one of biometric and non-biometric attributes.

32. The apparatus of claim 30, wherein at least one of the at least two distinctive attributes is selected from one of voice print, face recognition, signature recognition, face temperature infrared pattern, hand geometry, writing instrument velocity, writing instrument pressure, fingerprint, retinal print, a password, a personal identification number (PIN), and personal information.

33. The apparatus of claim 30, wherein two of the at least two attributes correspond to audio and video characteristics of the individual, the audio characteristics being text-independent, the recognition of the individual depending on the individual correctly speaking a password phrase, the segmenting and the indexing means further comprising:

(a) an audio and video joint likelihood function decoder for decoding synchronized utterances of audio and video data corresponding to the individual via application of a joint audio-video joint likelihood function to produce a decoded password phrase W, the joint likelihood function being a combination of likelihood functions for audio and video components;

(b) an audio likelihood function decoder operatively coupled to the audio and video joint likelihood function decoder for decoding the video data utilizing only the audio component of the joint likelihood function to produce a decoded password phrase V;

(c) a video likelihood function decoder operatively coupled to the audio and video joint likelihood function decoder for decoding the audio data utilizing only the video component of the joint likelihood function to produce a decoded password phrase A; and

(d) an adjustment module operatively coupled to the audio and video joint likelihood function decoder, the audio likelihood function decoder and the video likelihood function decoder for adjusting the audio component of the joint likelihood function to produce a better match of the decoded password phrase V to the decoded password phrase W, and for adjusting the video component of the joint likelihood function to produce a better match of the decoded password phrase A to the decoded password phrase W.

34. An apparatus employing synchronized speaker recognition, lip reading and lip recognition for recognizing an individual, comprising:

a speaker recognition system for performing speaker recognition;

a lip reading system for performing lip reading;

a lip recognition system for performing lip recognition;

a camera and pointing system for generating images of the individual for use by the lip reading and lip recognition systems, the speaker recognition system, the lip reading system, the lip recognition system, and the camera and pointing system contemporaneously extracting biometric attributes corresponding to speaker recognition, lip reading and lip recognition;

a store operatively coupled to the speaker recognition system, the lip reading system, and the lip recognition system for pre-storing biometric attributes during at least one enrollment session, the pre-stored attributes respectively corresponding to speaker recognition, lip reading and lip recognition;

segmentation means for segmenting the pre-stored attributes and the extracted attributes according to a sequence of segmentation units;

indexing means for indexing the segmented pre-stored and extracted attributes so that the segmented pre-stored and extracted attributes corresponding to an identical segmentation unit in the sequence of segmentation units are associated to an identical index, the speaker recognition system, the lip reading system, and the lip recognition system respectively comparing the segmented pre-stored and extracted attributes associated to the identical index to each other to recognize the individual; and

a controller for processing the results of the comparisons such that the individual is considered recognized if the segmented pre-stored attributes associated to the identical index substantially match the segmented extracted attributes associated to the identical index.

35. The apparatus of claim 34, wherein the lip reading system comprises a lip contour extractor and classifier for extracting and classifying the lip contour geometry of the individual.

36. The apparatus of claim 34, wherein the lip recognition system comprises a lip region classifier for classifying the lip shape of the individual.

37. The apparatus of claim 34, wherein the speaker recognition system is a textindependent speaker recognition system.

38. The apparatus of claim 37, further comprising a labeller for clustering lefemes having comparable lip positions.

39. The apparatus of claim 38, wherein the lip reading system is a textindependent lip reading system.

40. The apparatus of claim 39, wherein the lip reading comprises verifying if a lip shape extracted for a lefeme matches an expected lip shape of an associated cluster.

41. The apparatus of claim 34, wherein the lip reading system is a text-dependent lip reading system.

42. The apparatus of claim 41, wherein a lip shape corresponding to a lefeme is characterized by a deviation of the geometry relative to a corresponding speaker independent lip shape.

43. The apparatus of claim 42, wherein a close match of the speaker dependent shape relative to the speaker independent shape indicates a positive verification.