Audio-visual feature fusion and support vector machine useful for continuous speech recognition

• US 7,472,063 B2
• Filed: 12/19/2002
• Issued: 12/30/2008
• Est. Priority Date: 12/19/2002
• Status: Expired due to Fees

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Liang, Luhong, Nefian, Ara V., Zhao, Yibao, Liu, Xiaoxing, Pi, Xiaobo
• Primary Examiner(s): Smits; Talivaldis Ivars
• Application Number:
  US10/326,368
• Publication Number:
  US 20040122675A1
• Time in Patent Office: 2,203 Days
• Field of Search: None
• US Class Current: 704/256.1
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G10L 15/25 using position of the lips, movement of the lips or face analysis

Abstract

A speech recognition method includes several embodiments describing application of support vector machine analysis to a mouth region. Lip position can be accurately determined and used in conjunction with synchronous or asynchronous audio data to enhance speech recognition probabilities.

Claims

1. A method for recognizing speech by fusing audio and visual features, comprising

generating an audio vector representing detected audio data of a speech utterance,

detecting a face in a video data stream linked to the audio data of the speech utterance,

applying a cascade of linear support vector machine classifiers to the detected face to locate a mouth region,

generating vector data for the mouth region,

training a hidden Markov model (HMM) by fusing audio and visual vector data with the HMM, and

recognizing an input speech by extracting audio and visual features and by comparing the extracted audio and visual features with HMMs obtained at least in part through audio and visual fusion.

2. The method of claim 1, wherein the linear support vector machine classifiers are trained using positive and negative training sets.

3. The method of claim 1, further comprising use of a mouth and beard classifier.

4. The method of claim 1, wherein a cascade algorithm operates on a multidimensional mouth feature space obtained through principal component analysis decomposition.

5. The method of claim 1, further comprising mirroring, rotating, rescaling, and normalizing the mouth region.

6. The method of claim 1, further comprising applying a smoothing filter to the mouth region.

7. The method of claim 1, further comprising fusing audio and visual vector data using a two stream coupled hidden Markov model.

8. The method of claim 1, further comprising using asynchronous audio and video data.

9. The method of claim 1, further comprising applying a graph decoder and a Viterbi beam search.

10. The method of claim 1, further comprising modifying audio and video probabilities as a function of noise.

11. An article comprising a computer readable medium to store computer executable instructions, the instructions defined to cause a computer to recognize speech by fusing audio and visual features via operations including:

generating an audio vector representing detected audio data of a speech utterance,

detecting a face in a video data stream linked to the audio data of the speech utterance,

applying a cascade of linear support vector machine classifiers to the detected face to locate a mouth region,

generating vector data for the mouth region,

training a hidden Markov model (HMM) by fusing audio and visual vector data with the HMM, and

recognizing an input speech by extracting audio and visual features and by comparing the extracted audio and visual features with HMMs obtained at least in part through audio and visual fusion.

12. The article of claim 11, wherein the linear support vector machine classifiers are trained using positive and negative training sets.

13. The article of claim 11, wherein the operations further comprise use of a mouth and beard classifier.

14. The article of claim 11, wherein a cascade algorithm operates on a multidimensional mouth feature space obtained through principal component analysis decomposition.

15. The article of claim 11, wherein the operations further comprise mirroring, rotating, rescaling, and normalizing the mouth region.

16. The article of claim 11, wherein the operations further comprise applying a smoothing filter to the mouth region.

17. The article of claim 11, wherein the operations further comprise fusing audio and visual vector data using a two stream coupled hidden Markov model.

18. The article of claim 11, wherein the operations further comprise using asynchronous audio and video data.

19. The article of claim 11, wherein the operations further comprise applying a graph decoder and a Viterbi beam search.

20. The article of claim 11, wherein the operations further comprise modifying audio and video probabilities as a function of noise.