Method and apparatus for audio-visual speech detection and recognition

• US 20030018475A1
• Filed: 08/30/2002
• Published: 01/23/2003
• Est. Priority Date: 08/06/1999
• Status: Active Grant

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: de Cuetos, Philippe Christian, Senior, Andrew William, Maes, Stephane Herman, Neti, Chalapathy Venkata, Basu, Sankar
• Granted Patent: US 6,816,836 B2
• Time in Patent Office: Days
• US Class Current: 704/270
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G10L 15/25 using position of the lips, movement of the lips or face analysis
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10

Abstract

Techniques for providing speech recognition comprise the steps of processing a video signal associated with an arbitrary content video source, processing an audio signal associated with the video signal, and recognizing at least a portion of the processed audio signal, using at least a portion of the processed video signal, to generate an output signal representative of the audio signal.

Claims

1. A method of providing speech recognition, the method comprising the steps of:

processing a video signal associated with an arbitrary content video source;

processing an audio signal associated with the video signal; and

recognizing at least a portion of the processed audio signal, using at least a portion of the processed video signal, to generate an output signal representative of the audio signal.

2. The method of claim 1, wherein the audio signal is representative of conversational speech.

3. The method of claim 1, wherein the video signal processing operation comprises the step of detecting whether the video signal associated with the arbitrary content video source contains one or more face candidates.

4. The method of claim 3, wherein the video signal processing operation further comprises the step of detecting one or more facial features on one or more detected face candidates.

5. The method of claim 4, wherein the video signal processing operation further comprises the step of detecting whether the one or more detected facial features are characteristic of respective face candidates in frontal poses.

6. The method of claim 5, wherein at least one of face, facial feature and pose detection employ Fisher linear discriminant (FLD) analysis.

7. The method of claim 5, wherein at least one of face, facial feature and pose detection employ a distance from face space (DFFS) measure.

8. The method of claim 3, wherein the video signal processing operation comprises the step of extracting visual feature vectors from the video signal when a face is detected, the visual feature vectors representing one or more facial features of the detected face.

9. The method of claim 8, wherein the visual feature vectors represent grey scale parameters.

10. The method of claim 8, wherein the visual feature vectors represent geometric features.

11. The method of claim 8, wherein the visual feature vectors are normalized with respect to pose estimates.

12. The method of claim 8, wherein the audio signal processing operation comprises the step of extracting audio feature vectors from the associated audio signal, the audio feature vectors representing one or more acoustic features of the audio signal.

13. The method of claim 12, wherein probabilities are assigned to phonemes based on the audio feature vectors.

14. The method of claim 13, wherein probabilities are assigned to one of phonemes and visemes based on the visual feature vectors.

15. The method of claim 14, wherein respective joint probabilities are formed from corresponding probabilities associated with the visual feature vectors and the audio feature vectors.

16. The method of claim 15, wherein, in forming the joint probabilities, the corresponding probabilities associated with the visual feature vectors and the audio feature vectors are assigned weights based on a confidence measure.

17. The method of claim 16, wherein the joint probabilities are used to search for an appropriate output signal representative of the audio signal.

18. The method of claim 17, wherein the acoustic probabilities are used to search for a word hypothesis representative of the audio signal.

19. The method of claim 18, wherein the word hypothesis is re-calculated using the visual probabilities to generate an appropriate output signal representative of the audio signal.

20. The method of claim 12, wherein visual feature vectors are combined with corresponding audio feature vectors to form audio-visual feature vectors.

21. The method of claim 20, wherein probabilities are assigned to one of phonemes and visemes based on the audio-visual feature vectors.

22. The method of claim 21, wherein the probabilities are used to search for an appropriate output signal representative of the audio signal.

23. The method of claim 1, wherein at least one of the video signal and the audio signal are compressed signals.

24. The method of claim 1, wherein compressed signals are decompressed prior to processing operations.

25. The method of claim 1, wherein the arbitrary content video source provides MPEG-2 standard signals.

26. The method of claim 1, wherein the video signal includes at least one of visible electromagnetic spectrum images, non-visible electromagnetic spectrum images, and images from other sensing techniques.

28. The apparatus of claim 27, wherein the audio signal is representative of conversational speech.

29. The apparatus of claim 27, wherein the video signal processing operation comprises the step of detecting whether the video signal associated with the arbitrary content video source contains one or more face candidates.

30. The apparatus of claim 29, wherein the video signal processing operation further comprises the step of detecting one or more facial features on one or more detected face candidates.

31. The apparatus of claim 30, wherein the video signal processing operation further comprises the step of detecting whether the one or more detected facial features are characteristic of respective face candidates in frontal poses.

32. The apparatus of claim 31, wherein at least one of face, facial feature and pose detection employ Fisher linear discriminant (FLD) analysis.

33. The apparatus of claim 31, wherein at least one of face, facial feature and pose detection employ a distance from face space (DFFS) measure.

34. The apparatus of claim 29, wherein the video signal processing operation comprises the step of extracting visual feature vectors from the video signal when a face is detected, the visual feature vectors representing one or more facial features of the detected face.

35. The apparatus of claim 34, wherein the visual feature vectors represent grey scale parameters.

36. The apparatus of claim 34, wherein the visual feature vectors represent geometric features.

37. The apparatus of claim 34, wherein the visual feature vectors are normalized with respect to pose estimates.

38. The apparatus of claim 34, wherein the audio signal processing operation comprises the step of extracting audio feature vectors from the associated audio signal, the audio feature vectors representing one or more acoustic features of the audio signal.

39. The apparatus of claim 38, wherein probabilities are assigned to phonemes based on the audio feature vectors.

40. The apparatus of claim 39, wherein probabilities are assigned to one of phonemes and visemes based on the visual feature vectors.

41. The apparatus of claim 40, wherein respective joint probabilities are formed from corresponding probabilities associated with the visual feature vectors and the audio feature vectors.

42. The apparatus of claim 41, wherein, in forming the joint probabilities, the corresponding probabilities associated with the visual feature vectors and the audio feature vectors are assigned weights based on a confidence measure.

43. The apparatus of claim 42, wherein the joint probabilities are used to search for an appropriate output signal representative of the audio signal.

44. The apparatus of claim 39, wherein the acoustic probabilities are used to search for a word hypothesis representative of the audio signal.

45. The apparatus of claim 44, wherein the word hypothesis is re-calculated using the visual probabilities to generate an appropriate output signal representative of the audio signal.

46. The apparatus of claim 38, wherein visual feature vectors are combined with corresponding audio feature vectors to form audio-visual feature vectors.

47. The apparatus of claim 46, wherein probabilities are assigned to one of phonemes and visemes based on the audio-visual feature vectors.

48. The apparatus of claim 47, wherein the probabilities are used to search for an appropriate output signal representative of the audio signal.

49. The apparatus of claim 27, wherein at least one of the video signal and the audio signal are compressed signals.

50. The apparatus of claim 27, wherein compressed signals are decompressed prior to processing operations.

51. The apparatus of claim 27, wherein the arbitrary content video source provides MPEG-2 standard signals.

52. The apparatus of claim 27, wherein the video signal includes at least one of visible electromagnetic spectrum images, non-visible electromagnetic spectrum images, and images from other sensing techniques.

53. The apparatus of claim 27, wherein a speech detection decision is made using information from at least one of the video signal, the audio signal, and a correlation between the video signal and the audio signal.

27. Apparatus for providing speech recognition, the apparatus comprising:

at least one processor operable to;

(i) process a video signal associated with an arbitrary content video source;

(ii) process an audio signal associated with the video signal; and

(iii) recognize at least a portion of the processed audio signal, using at least a portion of the processed video signal, to generate an output signal representative of the audio signal; and

memory, coupled to the at least one processor, for storing at least a portion of results associated with at least one of the processing and recognizing operations.

54. A method of providing speech recognition, the method comprising the steps of:

processing an image signal associated with an arbitrary content image source;

processing an audio signal associated with the image signal; and

recognizing at least a portion of the processed audio signal, using at least a portion of the processed image signal, to generate an output signal representative of the audio signal.

55. Apparatus for providing speech recognition, the apparatus comprising:

at least one processor operable to;

(i) process an image signal associated with an arbitrary content image source, (ii) process an audio signal associated with the image signal, and (iii) recognize at least a portion of the processed audio signal, using at least a portion of the processed image signal, to generate an output signal representative of the audio signal; and

memory, coupled to the at least one processor, for storing at least a portion of results associated with at least one of the processing and recognizing operations.

56. Apparatus for providing speech recognition, the apparatus comprising:

means for processing a video signal associated with an arbitrary content video source;

means for processing an audio signal associated with the video signal; and

means for recognizing at least a portion of the processed audio signal, using at least a portion of the processed video signal, to generate an output signal representative of the audio signal.