Microphone array method and system, and speech recognition method and system using the same

• US 20040220800A1
• Filed: 05/03/2004
• Published: 11/04/2004
• Est. Priority Date: 05/02/2003
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Bang, Seok-won, Choi, Chang-kyu, Kong, Dong-geon, Lee, Bon-young
• Granted Patent: US 7,567,678 B2
• Time in Patent Office: Days
• US Class Current: 704/205
• CPC Class Codes: G10L 2021/02166 Microphone arrays; Beamforming
• CPC Class Codes: G10L 21/0208 Noise filtering
• CPC Class Codes: H04R 1/406 microphones
• CPC Class Codes: H04R 2201/401 2D or 3D arrays of transducers
• CPC Class Codes: H04R 2201/403 Linear arrays of transducers
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407

Abstract

A microphone array system including an input unit to receive sound signals using a plurality of microphones; a frequency splitter splitting each sound signal received into a plurality of narrowband signals; an average spatial covariance matrix estimator using spatial smoothing to obtain a spatial covariance matrix for each frequency component of the sound signal, by which spatial covariance matrices for a plurality of virtual sub-arrays, which are configured in the plurality of microphones, are obtained with respect to each frequency component of the sound signal and an average spatial covariance matrix is calculated; a signal source location detector to detect an incidence angle of the sound signal according to the average spatial covariance matrix calculated; a signal distortion compensator to calculates a weight for each frequency component of the sound signal based on the incidence angle of the sound signal and multiply the calculated weight by each frequency component.

Claims

1. A microphone array system comprising:

an input unit to receive sound signals using a plurality of microphones;

a frequency splitter to split each sound signal received through the input unit into a plurality of narrowband signals;

an average spatial covariance matrix estimator which uses spatial smoothing to obtain a spatial covariance matrix for each frequency component of the sound signal, by which spatial covariance matrices for a plurality of virtual sub-arrays, which are configured in the plurality of microphones, are obtained with respect to each frequency component of the sound signal processed by the frequency splitter and an average spatial covariance matrix is calculated,

a signal source location detector to detect an incidence angle of the sound signal according to the average spatial covariance matrix calculated using the spatial smoothing;

a signal distortion compensator to calculate a weight for each frequency component of the sound signal based on the incidence angle of the sound signal and multiply the calculated weight by each frequency component, thereby compensating for distortion of each frequency component; and

a signal restoring unit to restore a sound signal using the distortion compensated frequency components.

2. The microphone array system of claim 1, wherein the frequency splitter uses discrete Fourier transform to split each sound signal into the plurality of narrowband signals, and the signal restoring unit uses inverse discrete Fourier transform to restore the sound signal.

3. The microphone array system of claim 1, wherein the spatial smoothing is performed according to an equation:

4. The microphone array system of claim 1, wherein the signal source location detector splits each sound signal received from the input unit into the frequency components, into which the frequency splitter splits the sound signal, and performs a multiple signal classification algorithm only to frequency components selected according to a predetermined reference from among the split frequency components, thereby determining the incidence angle of the sound signal.

5. The microphone array system of claim 4, wherein the signal source location detector comprises:

a speech signal detector to split each sound signal received from the input unit into the frequency components, into which the frequency splitter further splits the sound signal, to group the sound signals having the same frequency component, thereby generating a plurality of groups for the respective frequency components, and to measure a speech presence probability in each group;

a group selector to select a predetermined number of groups in descending order of speech presence probability from among the plurality of groups; and

an arithmetic unit to perform the multiple signal classification algorithm with respect to frequency components corresponding to the respective selected groups.

6. A speech recognition system comprising:

a microphone array system;

a feature extractor to extract a feature of a sound signal received from the microphone array system;

a reference pattern storage unit to store reference patterns to be compared with the extracted feature;

a comparator to compare the extracted feature with the reference patterns stored in the reference pattern storage unit; and

a determiner to determine whether a speech is recognized based on the compared result, wherein the microphone array system comprises;

an input unit to receive sound signals using a plurality of microphones;

a frequency splitter to split each sound signal received through the input unit into a plurality of narrowband signals;

an average spatial covariance matrix estimator which uses spatial smoothing to obtain a spatial covariance matrix for each frequency component of the sound signal, by which spatial covariance matrices for a plurality of virtual sub-arrays, which are configured in the plurality of microphones, are obtained with respect to each frequency component of the sound signal processed by the frequency splitter and then an average spatial covariance matrix is calculated,

a signal source location detector to detect an incidence angle of the sound signal according to the average spatial covariance matrix calculated using the spatial smoothing;

a signal distortion compensator to calculate a weight for each frequency components of the sound signal based on the incidence angle of the sound signal and multiply the calculated weight by each frequency component, thereby compensating for distortion of each frequency component; and

a signal restoring unit to restore a sound signal using the distortion compensated frequency components.

7. The speech recognition system of claim 6, wherein the spatial smoothing is performed according to an equation:

8. The speech recognition system of claim 6, wherein the signal source location detector splits each sound signal received from the input unit into the frequency components, into which the frequency splitter splits the sound signal, and performs a multiple signal classification multiple signal classification algorithm only to frequency components selected according to a predetermined reference from among the split frequency components, thereby determining the incidence angle of the sound signal.

9. The speech recognition system of claim 8, wherein the signal source location detector comprises:

a speech signal detector to split each sound signal received from the input unit into the frequency components, the frequency splitter further splits the sound signal, to group the sound signals having the same frequency component, thereby generating a plurality of groups for the respective frequency components, and to measure a speech presence probability in each group;

a group selector to select a predetermined number of groups in descending order of speech presence probability from among the plurality of groups; and

an arithmetic unit to perform the multiple signal classification algorithm with respect to frequency components corresponding to the respective selected groups.

10. A microphone array method comprising:

receiving a plurality of wideband sound signals from an array having a plurality of microphones;

splitting each wideband sound signal into a plurality of narrowbands;

obtaining spatial covariance matrices for a plurality of virtual sub-arrays, which include a plurality of microphones constituting the array of the plurality of microphones, with respect to each narrowband using a predetermined scheme and averaging the obtained spatial covariance matrices, thereby obtaining an average spatial covariance matrix for each narrowband;

calculating an incidence angle of each wideband sound signal using the average spatial covariance matrix for each narrowband and a predetermined algorithm;

calculating weights to be respectively multiplied with the narrowbands according to the incidence angle of the wideband sound signal and multiplying the weights by the respective narrowbands; and

restoring a wideband sound signal using the narrowbands after being multiplied by the weights respectively.

11. The microphone array method of claim 10, wherein the splitting is based on discrete Fourier transform, and the restoring is based on inverse discrete Fourier transform.

12. The microphone array method of claim 10, wherein the obtaining of the spatial covariance matrices comprises performing the spatial smoothing according to an equation:

13. The microphone array method of claim 10, wherein the calculating of the incidence angle comprises:

splitting each sound signal received from the array having the plurality of microphones into the frequency components of the split sound signal; and

performing a multiple signal classification algorithm with respect to only frequency components selected according to a predetermined reference from among the split frequency components, thereby determining the incidence angle of the sound signal.

14. The microphone array method of claim 13, wherein the calculating of the incidence angle further comprises:

splitting each sound signal received from the array having the plurality of microphones into the frequency components of the spilt sound signal;

grouping the sound signals having the same frequency component, thereby generating a plurality of groups for the respective frequency components to measure a speech presence probability in each group;

selecting a predetermined number of groups in descending order of speech presence probability from among the plurality of groups; and

performing the multiple signal classification algorithm with respect to frequency components corresponding to the respective selected groups.

15. A microphone array method comprising:

receiving wideband sound signals from an array having a plurality of microphones;

splitting each wideband sound signal into a plurality of narrowbands;

obtaining spatial covariance matrices for a plurality of virtual sub-arrays, which include a plurality of microphones constituting the array of the plurality of microphones, with respect to each narrowband using a predetermined scheme, and averaging the obtained spatial covariance matrices, thereby obtaining an average spatial covariance matrix for each narrowband;

calculating an incidence angle of each wideband sound signal using the average spatial covariance matrix for each narrowband and a predetermined algorithm;

calculating weights to be respectively multiplied with the narrowbands based on the incidence angle of the wideband sound signal and multiplying the weights by the respective narrowbands;

restoring a wideband sound signal using the narrowbands after being multiplied by the weights respectively;

extracting a feature of a sound signal received from the microphone array system;

storing reference patterns to be compared with the extracted feature;

comparing the extracted feature with the reference patterns stored; and

determining based on a comparison result whether a speech is recognized.

16. The microphone array method of claim 15, wherein the splitting is based on discrete Fourier transform, and the restoring is based on inverse discrete Fourier transform.

17. The microphone array method of claim 15, wherein the obtaining of the spatial covariance matrices comprises performing the spatial smoothing according to an equation:

18. The microphone array method of claim 15, wherein the calculating step of the incidence angle, comprises:

splitting each sound signal received from the array having the plurality of microphones into the frequency components of the split sound signal; and

performing a multiple signal classification algorithm with respect to only frequency components selected according to a predetermined reference from among the split frequency components, thereby determining the incidence angle of the sound signal.

19. The microphone array method of claim 18, wherein the calculating step of the incidence angle further comprises:

splitting each sound signal received from the array having the plurality of microphones into the frequency components of the split sound signal;

grouping the sound signals having the same frequency component, thereby generating a plurality of groups for the respective frequency components and measuring a speech presence probability in each group;

selecting a predetermined number of groups in descending order of speech presence probability from among the plurality of groups; and

performing the MUSIC algorithm with respect to frequency components corresponding to the respective selected groups.

20. A speech recognition system comprising:

an input unit to receive sound signals using a plurality of microphones;

a frequency splitter to split each sound signal received through the input unit into a plurality of narrowband signals;

a signal processor to perform a multiple signal classification algorithm with respect to frequency components selected according to a predetermined reference from among the split frequency components of the sound signal split by the frequency splitter; and

a direction detector to detect a direction of a speech signal using the processing result output from the signal processor.

21. The speech recognition system of claim 20, wherein the frequency splitter uses discrete Fourier transform to split each sound signal into the plurality of narrowband signals.

22. The speech recognition system of claim 20, wherein the signal processor comprises:

a speech signal detector to split each sound signal received from the input unit into the frequency components, the frequency splitter further splits the sound signal, to group the sound signals having the same frequency component, thereby generating a plurality of groups for the respective frequency components, and to measure a speech presence probability in each group;

a group selector to select a predetermined number of groups in descending order of speech presence probability from among the plurality of groups; and

an arithmetic unit to perform the multiple signal classification algorithm with respect to frequency components corresponding to the respective selected groups.

23. A speech recognition method comprising:

receiving sound signals using a plurality of microphones;

splitting each sound signal received through the input unit into a plurality of narrowband signals;

performing a multiple signal classification algorithm with respect to frequency components selected according to a predetermined reference from among the split frequency components of the split sound signal; and

detecting a direction of a speech signal using the processing result of the performing of the multiple signal classification algorithm.

24. The speech recognition method of claim 23, wherein the splitting comprises splitting each sound signal into the plurality of narrowband signals using discrete Fourier transform.

25. The speech recognition method of claim 23, wherein the performing of the multiple signal classification algorithm comprises:

splitting each sound signal received from the plurality of microphones into the frequency components of the split sound signal, grouping the sound signals having the same frequency component, thereby generating a plurality of groups for the respective frequency components, and measuring a speech presence probability in each group;

selecting a predetermined number of groups in descending order of speech presence probability from among the plurality of groups; and

performing the MUSIC algorithm with respect to frequency components corresponding to the respective selected groups.

26. A microphone array input type speech recognition system using spatial filtering and having a microphone array to receive sound signals, the system comprising:

an average spatial covariance matrix estimator which uses spatial smoothing to produce a spatial covariance matrix for each frequency component of the received sound signals, by which spatial covariance matrices for a plurality of virtual sub-arrays, which are configured in the microphones array, are obtained with respect to each frequency component of the sound signals and an average spatial covariance matrix is calculated;

a signal source location detector to detect a source location of each of the sound signals using the average spatial covariance matrices; and

a signal distortion compensator to calculate a weight matrix to be multiplied by each frequency component using the detected source location of each of the sound signals in order to compensate for distortion due to noise and an echo of a sound signal.

27. The microphone array input type speech recognition system of claim 26, further comprising a signal restoring unit to restore each of the sound signals using the distortion compensated frequency components.

28. The microphone array input type speech recognition system of claim 26, further comprising an input unit to receive each of the sound signals, the input unit having an array of M microphones and a plurality of virtual sub-arrays of L microphones.

29. The microphone array input type speech recognition system of claim 28, further comprising a frequency splitter to split each of the sound signals received through the input unit into a plurality of narrowband frequency signals.

30. The microphone array input type speech recognition system of claim 28, wherein the frequency splitter uses a discrete Fourier transform to split each of the sound signals received into narrowband frequency signals.

31. The microphone array input type speech recognition system of claim 26, further comprising a signal restoring unit to restore each of the sound signals using the distortion compensated frequency components from the signal distortion compensator.

32. The microphone array input type speech recognition system of claim 31, wherein the signal restoring unit uses inverse a discrete Fourier transform to restore each of the sound signals.

33. The microphone array input type speech recognition system of claim 28, wherein the spatial smoothing is performed according to an equation:

34. The microphone array input type speech recognition system of claim 30, wherein the signal source location detector splits each of the sound signals received from the input unit into the frequency components, into which the frequency splitter splits each of the sound signals, and performs a multiple signal classification algorithm only to frequency components selected according to a predetermined reference from among the split frequency components, thereby determining the location of each of the sound signals.

35. The microphone array input type speech recognition system of claim 34, wherein the signal source location detector detects the location of each of the sound signals using a respective incidence angle.

36. The microphone array input type speech recognition system of claim 26, wherein the signal source location detector is a wideband multiple signal classification unit and the signal distortion compensator is a wideband minimum variance unit.

37. The microphone array input type speech recognition system of claim 27, further comprising a speech recognition module to obtain a speech recognition result by comparing a feature of each of the restored sound signals with a plurality of reference patterns to determine a sound most similar to the restored sound signal.

38. The microphone array input type speech recognition system of claim 37, wherein the speech recognition module further comprises:

a feature extractor unit to extract a feature vector of each of the restored sound signals;

a reference pattern storage unit to store the reference patterns for a plurality of sounds;

a determination unit to compare the extracted feature vector with the reference patterns stored to search for a sound similar to the restored sound signal, wherein the reference pattern with a highest correlation value exceeding a predetermined value is recognized as the sound signal.

39. The microphone array input type speech recognition system of claim 26, further comprising a frequency bin selector to select frequency bins likely to include a speech signal according to a predetermined reference such that the signal source location detector performs the multiple signal classification algorithm with respect to only frequency components corresponding to the respective selected frequency bins.

40. The microphone array input type speech recognition system of claim 39, further comprising a discrete Fourier transformer to perform a fast Fourier transform on each of the input sound signals.

41. The microphone array input type speech recognition system of claim 39, wherein the signal source detector further comprises a peak detector to determine a direction of each of the sound signals.

42. A microphone array input type speech recognition method of receiving sound signals and using spatial filtering to acquire a high-quality speech signal for recognizing speech, the method comprising:

obtaining a spatial covariance matrix for each frequency component of the received sound signals, using spatial smoothing, by which spatial covariance matrices for a plurality of virtual sub-arrays, which are configured in the microphones array, are obtained with respect to each frequency component of the sound signals and an average spatial covariance matrix is calculated;

detecting a source location of each of the sound signals using the average spatial covariance matrices; and

calculating a weight matrix to be multiplied by each frequency component using the detected source location of each of the sound signals in order to compensate for distortion due to noise and an echo of a sound signal.

43. The microphone array input type speech recognition method of claim 42, further restoring each of the sound signals using the distortion compensated frequency components.

44. The microphone array input type speech recognition method of claim 42, further comprising splitting each of the sound signals received into a plurality of narrowband frequency signals.

45. The microphone array input type speech recognition method of claim 44, further comprising receiving each of the sound signals through an array of M microphones a plurality of virtual sub-arrays of L microphones.

46. The microphone array input type speech recognition method of claim 45, further comprising using a discrete Fourier transform to split each of the sound signals into narrowband frequency signals.

47. The microphone array input type speech recognition method of claim 42, further comprising restoring each of the sound signals using the distortion compensated frequency components.

48. The microphone array input type speech recognition method of claim 42, wherein the restoring is calculated using a discrete Fourier transform.

49. The microphone array input type speech recognition method of claim 42, wherein the spatial smoothing is performed according to an equation:

50. The microphone array input type speech recognition method of claim 44, wherein the detecting the source location of each of the sound signals, comprises:

splitting each of the sound signals received into the frequency components of each of the split sound signals; and

performing a multiple signal classification algorithm with respect to only frequency components selected according to a predetermined reference from among the split frequency components, thereby determining the source location of each of the sound signals.

51. The microphone array input type speech recognition method of claim 50, wherein the detecting the source location of each of the sound signals, further comprises:

splitting each of the sound signals received into the frequency components of each of the split sound signals;

grouping each of the sound signals having the same frequency component, thereby generating a plurality of groups for the respective frequency components to measure a speech presence probability in each group;

selecting a predetermined number of groups in descending order of speech presence probability from among the plurality of groups; and

performing the multiple signal classification algorithm with respect to frequency components corresponding to the respective selected groups.

52. The microphone array input type speech recognition method of claim 43, further comprising obtaining a speech recognition result by comparing a feature of each of the restored sound signals with a plurality of reference patterns to determine a sound most similar to the restored sound signal.

53. The microphone array input type speech recognition method of claim 52, wherein the speech recognition module further comprises:

extracting a feature vector of each of the restored sound signals;

storing the reference patterns for a plurality of sounds;

comparing the extracted feature vector with the reference patterns stored to search for a sound similar to the restored sound signal, wherein the reference pattern with a highest correlation value exceeding a predetermined value is recognized as the sound signal.

54. The microphone array input type speech recognition method of claim 42, further comprising selecting frequency bins likely to include a speech signal according to a predetermined reference such that the multiple signal classification algorithm is performed with respect to only frequency components corresponding to the respective selected frequency bins.

55. The microphone array input type speech recognition method of claim 54, further comprising performing a fast Fourier transform on each of the input sound signals.

56. The microphone array input type speech recognition method of claim 54, further comprising detecting a peak of the each of the sound signals to determine a direction of each of the sound signals.

57. A microphone array input type speech recognition system using spatial filtering and having a microphone array to receive sound signals, the system comprising:

an average spatial covariance matrix estimator which uses spatial smoothing to produce a spatial covariance matrix for each frequency component of the received sound signals, by which spatial covariance matrices for a plurality of virtual sub-arrays, which are configured in the microphones array, are obtained with respect to each frequency component of the sound signals and an average spatial covariance matrix is calculated;

a signal source location detector to detect a source location of each of the sound signals using the average spatial covariance matrices;

a signal distortion compensator to calculate a weight matrix to be multiplied by each frequency component using the detected source location of each of the sound signals in order to compensate for distortion due to noise and an echo of a sound signal;

a signal restoring unit to restore each of the sound signals using the distortion compensated frequency components; and

a speech recognition module to obtain a speech recognition result by comparing a feature of each of the restored sound signals with a plurality of reference patterns to determine a sound most similar to the restored sound signal.

58. The microphone array input type speech recognition system of claim 57, further comprising an input unit to receive each of the sound signals, the input unit having an array of M microphones and a plurality of virtual sub-arrays of L microphones.

59. The microphone array input type speech recognition system of claim 58, wherein the signal source location detector splits each of the sound signals received from the input unit into the frequency components, into which the frequency splitter splits each of the sound signals, and performs a multiple signal classification algorithm only to frequency components selected according to a predetermined reference from among the split frequency components, thereby determining the location of each of the sound signals.

60. The microphone array input type speech recognition system of claim 59, wherein the speech recognition module further comprises:

a feature extractor unit to extract a feature vector of each of the restored sound signals;

a reference pattern storage unit to store the reference patterns for a plurality of sounds;

a determination unit to compare the extracted feature vector with the reference patterns stored to search for a sound similar to the restored sound signal, wherein the reference pattern with a highest correlation value exceeding a predetermined value is recognized as the sound signal.

61. The microphone array input type speech recognition system of claim 60, further comprising a frequency bin selector to select frequency bins likely to include a speech signal according to a predetermined reference such that the signal source location detector performs the multiple signal classification algorithm with respect to only frequency components corresponding to the respective selected frequency bins.

62. The microphone array input type speech recognition method using spatial filtering, the method comprising:

selecting a predetermined number of channels based on a distribution of averaged speech presence probabilities such that an amount of computation for a multiple signal classification algorithm is reduced, the channels having a speech presence probability greater than a predetermined value are selected for the multiple signal classification, wherein each of the channels is a unit into which a predetermined number of frequency bins are grouped

63. The microphone array input type speech recognition method of claim 62, wherein the number of frequency bins subject to the multiple signal classification algorithm is constant.

64. The microphone array input type speech recognition method of claim 62, further comprising selecting the channels in descending order of power, wherein power of each of the sound signals input is measured with respect to each frequency bin.

65. A microphone array input type speech recognition system using spatial filtering and having a microphone array to receive sound signals, the system comprising:

a microphone array module to decompose received sound signals using a spatial smoothing technique; and

a speech recognition module to receive the decomposed sound signals from the microphone array system and recognize speech, wherein there is a reduction in the removal of a target signal from the sound signals having noise and interference.