Face recognition procedure useful for audiovisual speech recognition

• US 20030212552A1
• Filed: 05/09/2002
• Published: 11/13/2003
• Est. Priority Date: 05/09/2002
• Status: Abandoned Application

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Pi, Xiaobo, Mao, Crusoe, Nefian, Ara V., Liu, Xiaoxing, Liang, Lu Hong
• Application Number:
  US10/143,459
• Publication Number:
  US 20030212552A1
• Time in Patent Office: Days
• US Class Current: 704/231
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G10L 15/25 using position of the lips, movement of the lips or face analysis

Abstract

A visual feature extraction method includes application of multiclass linear discriminant analysis to the mouth region. Lip position can be accurately determined and used in conjunction with synchronous or asynchronous audio data to enhance speech recognition probabilities.

Claims

1. A visual feature extraction method comprising

segmenting a mouth region from the detected face,

finding the contour of the lips, and widowing the mouth region to emphasize the region inside the lip contour,

applying the two dimensional discrete cosine transform on blocks within the mouth region,

applying multiclass linear discriminant analysis to the windowed mouth region.

2. The visual feature extraction method of claim 1, wherein the linear discriminant space is computed using a set of segmented images of the lip and face regions.

3. The visual feature extraction method of claim 1, wherein contour of the lips is obtained through binary chain encoding.

4. The visual feature extraction method of claim 1, wherein a refined position of the mouth. corners is obtained by applying a corner finding filter.

5. The visual feature extraction method of claim 1, further comprising masking, resizing, rotating, normalizing the mouth region.

6. The method of claim 1, further comprising visual feature extraction from the video data set using a variable shape window and application of a two dimensional discrete transform.

7. The visual feature extraction method of claim 1, further comprising use of block two dimension discrete cosine transform coefficients to determine visual observation vectors.

8. The visual feature extraction method of claim 1, further comprising using an audio and a video data set that respectively provide a first data stream of speech data and a second data stream of face image data and applying a two stream coupled hidden Markov model to the first and second data streams for speech recognition.

9. The method of claim 8, wherein the audio and video data sets providing the first and second data streams are asynchronous.

10. The method of claim 8, further comprising training of the two stream coupled hidden Markov model using a Viterbi algorithm.

19. The method of claim 8, wherein the audio and video data sets providing the first and second data streams are asynchronous.

20. The method of claim 8, further comprising training of the two stream coupled hidden Markov model using a Viterbi algorithm.

11. An article comprising a computer readable medium to store computer executable instructions, the instructions defined to cause a computer to

detect a face in video data,

segment a mouth region in the detected face,

apply multiclass linear discriminant analysis to the mouth region.

12. The article comprising a computer readable medium to store computer executable instructions of claim 11, wherein the instructions further cause a computer to compute the linear discriminant space using a set of segmented images of the lip and face regions.

13. The article comprising a computer readable medium to store computer executable instructions of claim 11, wherein the instructions further cause a computer to obtain a contour of the lips through binary chain encoding.

14. The article comprising a computer readable medium to store computer executable instructions of claim 11, wherein the instructions further cause a computer to obtain a refined position of the mouth corners by applying a corner finding filter.

15. The article comprising a computer readable medium to store computer executable instructions of claim 11, wherein the instructions further cause a computer to mask, resize, rotate, and normalize the mouth region.

16. The article comprising a computer readable medium to store computer executable instructions of claim 11, wherein the instructions further cause a computer to perform visual feature extraction from the video data set using a variable shape window and application of a two dimensional discrete transform.

17. The article comprising a computer readable medium to store computer executable instructions of claim 11, wherein the instructions further cause a computer to use block two dimension discrete cosine transform coefficients to determine visual observation vectors.

18. The article comprising a computer readable medium to store computer executable instructions of claim 11, wherein the instructions further cause a computer use an audio and a video data set that respectively provide a first data stream of speech data and a second data stream of face image data and apply a two stream coupled hidden Markov model to the first and second data streams for speech recognition.

21. A speech recognition system comprising

an audiovisual capture module to respectively provide a first data stream of speech data and a second data stream of video data,

a visual feature extraction module that detects a face in the second data stream of video data, discriminates a mouth region in the detected face, and applies multiclass linear discriminant analysis to the mouth region, and

a speech recognition module that applies a two stream coupled hidden Markov model to the first data stream of speech data and the second video data stream processed by the visual feature extraction module.

22. The speech recognition system of claim 21, further comprising asynchronous audio and video data.

23. The speech recognition system of claim 21, further comprising parallel processing of the first and second data streams by the speech recognition module.

24. The speech recognition system of claim 21, further comprising visual feature extraction from the video data set using a variable shape window and application of a two dimensional discrete transform by the visual feature extraction module.