Coupled hidden markov model for audiovisual speech recognition

• US 20030212557A1
• Filed: 05/09/2002
• Published: 11/13/2003
• Est. Priority Date: 05/09/2002
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Nefian, Ara V.
• Granted Patent: US 7,165,029 B2
• Time in Patent Office: Days
• US Class Current: 704/256
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/142 Hidden Markov Models [HMMs]
• CPC Class Codes: G10L 15/25 using position of the lips, movement of the lips or face analysis

Abstract

A speech recognition method includes use of synchronous or asynchronous audio and a video data to enhance speech recognition probabilities. A two stream coupled hidden Markov model is trained and used to identify speech. At least one stream is derived from audio data and a second stream is derived from mouth pattern data. Gestural or other suitable data streams can optionally be combined to reduce speech recognition error rates in noisy environments.

Claims

1. A speech recognition method comprising

using an audio and a video data set that respectively provide a first data stream of speech data and a second data stream of face image data, and

applying a two stream coupled hidden Markov model to the first and second data streams for speech recognition.

2. The method of claim 1, wherein the audio and video data sets providing the first and second data streams are asynchronous.

3. The method of claim 1, further comprising parallel processing of the first and second data streams.

4. The method of claim 1, further comprising visual feature extraction of a mouth region from the video data set.

5. The method of claim 1, further comprising visual feature extraction from the video data set using a variable shape window and application of a two dimensional discrete transform.

6. The method of claim 1, further comprising visual feature extraction from the video data set using linear discriminant analysis.

7. The method of claim 1, further comprising training of the two stream coupled hidden Markov model using a Viterbi algorithm.

8. An article comprising a computer readable medium to store computer executable instructions, the instructions defined to cause a computer to

use an audio and a video data set that respectively provide a first data stream of speech data and a second data stream of face image data, and

apply a two stream coupled hidden Markov model to the first and second data streams for speech recognition.

9. The article comprising a computer readable medium to store computer executable instructions of claim 8, wherein the instructions further cause a computer to process asynchronous first and second data streams.

10. The article comprising a computer readable medium to store computer executable instructions of claim 8, wherein the instructions further cause a computer to parallel process the first and second data streams.

11. The article comprising a computer readable medium to store computer executable instructions of claim 8, wherein the instructions further cause a computer provide visual feature extraction of a mouth region from the video data set.

12. The article comprising a computer readable medium to store computer executable instructions of claim 8, wherein the instructions further cause a computer to provide visual feature extraction from the video data set using a variable shape window and application of a two dimensional discrete transform.

13. The article comprising a computer readable medium to store computer executable instructions of claim 8, wherein the instructions further cause a computer to provide visual feature extraction from the video data set using linear discriminant analysis.

14. The article comprising a computer readable medium to store computer executable instructions of claim 8, wherein the instructions further cause a computer to train the two stream coupled hidden Markov model using a Viterbi algorithm.

15. A speech recognition system comprising

an audiovisual capture module to capture an audio and a video data set that respectively provide a first data stream of speech data and a second data stream of face image data, and

a speech recognition module that applies a two stream coupled hidden Markov model to the first and second data streams for speech recognition.

16. The speech recognition system of claim 15, further comprising asynchronous audio and video data sets.

17. The speech recognition system of claim 15, further comprising parallel processing of the first and second data streams by the speech recognition module.

18. The speech recognition system of claim 15, further comprising visual feature extraction of a mouth region from the video data set by the audiovisual capture module.

19. The speech recognition system of claim 15, further comprising visual feature extraction from the video data set using a variable shape window and application of a two dimensional discrete transform by the audiovisual capture module.

20. The speech recognition system of claim 15, further comprising visual feature extraction from the video data set by the audiovisual capture module using linear discriminant analysis.

21. The speech recognition system of claim 15, further comprising training of the two stream coupled hidden Markov model by the speech recognition module using a Viterbi algorithm.