Factorial hidden markov model for audiovisual speech recognition

• US 7,209,883 B2
• Filed: 05/09/2002
• Issued: 04/24/2007
• Est. Priority Date: 05/09/2002
• Status: Expired due to Fees

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Nefian, Ara V.
• Primary Examiner(s): Hudspeth; David
• Assistant Examiner(s): Wozniak; James S.
• Application Number:
  US10/142,447
• Publication Number:
  US 20030212556A1
• Time in Patent Office: 1,811 Days
• Field of Search: 704/271, 704/256, 704/235, 704/251, 704/256.2, 704/244, 382/115, 382/118
• US Class Current: 704/256
• CPC Class Codes: G06F 18/295 Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G10L 15/142 Hidden Markov Models [HMMs]
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features

Abstract

A speech recognition method includes use of synchronous or asynchronous audio and a video data to enhance speech recognition probabilities. A two stream factorial hidden Markov model is trained and used to identify speech. At least one stream is derived from audio data and a second stream is derived from mouth pattern data. Gestural or other suitable data streams can optionally be combined to reduce speech recognition error rates in noisy environments.

Claims

1. A speech recognition method for audiovisual data comprising

obtaining a first data stream of speech data and a second data stream of face image data while a speaker is speaking;

extracting visual features from the second data stream by masking, resizing, rotating, and normalizing a mouth region in a face image, and by using a two-dimensional discrete cosine transform;

constructing a factorial hidden Markov model for the first data stream and the second data stream, the factorial hidden Markov model including a plurality of hidden Markov models with each hidden Markov model having a plurality of discrete nodes and continuous observable nodes, wherein discrete nodes at a first time for each hidden Markov model are conditioned by discrete nodes at a second time of the plurality of hidden Markov models; and

providing maximum likelihood training for the factorial hidden Markov model to identify words.

2. The method of claim 1, further comprising maximum likelihood initialization using a Viterbi algorithm.

3. The method of claim 1, wherein the first data stream and the second data stream each corresponds to one of the plurality of hidden Markov models.

4. The method of claim 3, wherein the first data stream and the second data stream are asynchronous.

5. The method of claim 1, further comprising using a Viterbi algorithm to determine optimal sequence of states for the coupled nodes of the the first and the second streams that maximizes the observation likelihood during maximum likelihood training.

6. A speech recognition method comprising

using an audio and a video data set that respectively provide a first data stream of speech data and a second data stream of face image data,

extracting visual features from the second data stream by masking, resizing, rotating, and normalizing a mouth region in a face image, and by using a two-dimensional discrete cosine transform, and

applying a two stream factorial hidden Markov model (“

HMM”

) to the first and second data streams for speech recognition, the factorial HMM including two HMMs with one corresponding to the first data stream and the other corresponding to the second data stream, each HMM having a plurality of discrete nodes and continuous observable nodes, wherein discrete nodes at a first time for each HMM are conditioned by discrete nodes at a second time of the two HMMs.

7. The method of claim 6, wherein the audio and video data sets providing the first and second data streams are asynchronous.

8. The method of claim 6, further comprising parallel processing of the first and second data streams.

9. The method of claim 6, further comprising training of the two stream factorial hidden Markov model using a Viterbi algorithm.

10. An article comprising a computer readable medium to store computer executable instructions, the instructions defined to cause a computer to

use an audio and a video data set that respectively provide a first data stream of speech data and a second data stream of face image data

extract visual features from the second data stream by masking, resizing, rotating, and normalizing a mouth region in a face image, and by using a two-dimensional discrete cosine transform, and

apply a two stream factorial hidden Markov model (“

HMM”

) to the first and second data streams for speech recognition, the factorial HMM including two HMMs with one corresponding to the first data stream and the other corresponding to the second data stream, each HMM having a plurality of discrete nodes and continuous observable nodes, wherein discrete nodes at a first time for each HMM are conditioned by discrete nodes at a second time of the two HMMs.

11. The article comprising a computer readable medium to store computer executable instructions of claim 10, wherein the instructions further cause a computer to process asynchronous first and second data streams.

12. The article comprising a computer readable medium to store computer executable instructions of claim 10, wherein the instructions further cause a computer to parallel process the first and second data streams.

13. The article comprising a computer readable medium to store computer executable instructions of claim 10, wherein the instructions further cause a computer to train the two stream factorial hidden Markov model using a Viterbi algorithm.

14. A speech recognition system comprising

an audiovisual capture module to capture an audio and a video data set that respectively provide a first data stream of speech data and a second data stream of face image data,

a feature extraction module to extract visual features from the second data stream by masking, resizing, rotating, and normalizing a mouth region in a face image, and by using a two-dimensional discrete cosine transform, and

a speech recognition module that applies a two stream factorial hidden Markov model (“

HMM”

) to the first and second data streams for speech recognition, the factorial HMM including two HMMs with one corresponding to the first data stream and the other corresponding to the second data stream, each HMM having a plurality of discrete nodes and continuous observable nodes, wherein discrete nodes at a first time for each HMM are conditioned by discrete nodes at a second time of the two HMMs.

15. The speech recognition system of claim 14, further comprising asynchronous audio and video data sets.

16. The speech recognition system of claim 14, wherein the speech recognition module processes the first and second data streams in parallel.

17. The speech recognition system of claim 14, wherein the speech recognition module trains the two stream factorial hidden Markov model using a Viterbi algorithm.