System and method of mixing accelerometer and microphone signals to improve voice quality in a mobile device

• US 9,363,596 B2
• Filed: 03/15/2013
• Issued: 06/07/2016
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Dusan, Sorin V., Lindahl, Aram, Andersen, Esge B.
• Primary Examiner(s): Hong, Harry S
• Application Number:
  US13/840,667
• Publication Number:
  US 20140270231A1
• Time in Patent Office: 1,180 Days
• Field of Search: 381/74, 381/92, 381/110
• US Class Current: 1/1
• CPC Class Codes: G10L 2021/02166 Microphone arrays; Beamforming
• CPC Class Codes: G10L 2021/02168 the estimation exclusively taking place during speech pauses
• CPC Class Codes: G10L 21/0216 characterised by the method used for estimating noise
• CPC Class Codes: G10L 25/90 Pitch determination of speech signals
• CPC Class Codes: H04R 1/1033 Cables or cables storage, e.g. cable reels cord reels per se H02G11/02; arrangements for storing and repeatedly paying-out and re-storing lengths of conductors or cables B65H75/34; extensible conductors or cables, e.g. self-coiling cords H01B7/06
• CPC Class Codes: H04R 1/1041 Mechanical or electronic switches, or control elements switches in general H01H
• CPC Class Codes: H04R 1/1083 Reduction of ambient noise active noise reduction per se G10K11/175; protective devices for the ear, e.g. providing acoustic protection A61F11/06
• CPC Class Codes: H04R 1/46 Special adaptations for use as contact microphones, e.g. on musical instrument, on stethoscope throat mountings H04R1/14
• CPC Class Codes: H04R 2201/107 Monophonic and stereophonic headphones with microphone for two-way hands free communication
• CPC Class Codes: H04R 2460/13 Hearing devices using bone conduction transducers
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407

Abstract

A method of improving voice quality in a mobile device starts by receiving acoustic signals from microphones included in earbuds and the microphone array included on a headset wire. The headset may include the pair of earbuds and the headset wire. An output from an accelerometer that is included in the pair of earbuds is then received. The accelerometer may detect vibration of the user'"'"'s vocal chords filtered by the vocal tract based on vibrations in bones and tissue of the user'"'"'s head. A spectral mixer included in the mobile device may then perform spectral mixing of the scaled output from the accelerometer with the acoustic signals from the microphone array to generate a mixed signal. Performing spectral mixing includes scaling the output from the inertial sensor by a scaling factor based on a power ratio between the acoustic signals from the microphone array and the output from the inertial sensor. Other embodiments are also described.

Claims

1. A method of improving voice quality in a mobile device comprising:

receiving acoustic signals from one or more microphones included with a pair of earbuds, wherein a headset includes the pair of earbuds and a headset wire;

receiving an output from an inertial sensor that is included in the pair of earbuds;

performing spectral mixing of the output from the inertial sensor with the acoustic signals from the one or more microphones to generate a mixed signal, wherein performing spectral mixing includes scaling the output from the inertial sensor by a scaling factor based on a power ratio between the acoustic signals from the one or more microphones and the output from the inertial sensor.

2. The method of claim 1, wherein the one or more microphones included with the pair of earbuds comprises:

a front microphone and a rear microphone in each of the earbuds.

3. The method of claim 1, wherein the inertial sensor is an accelerometer that is included in each of the earbuds.

4. The method of claim 3, performing spectral mixing to generate the mixed signal further comprises:

pre-emphasizing the output from the accelerometer to account for lip radiation characteristic to generate a pre-emphasized accelerometer signal.

5. The method of claim 4, performing spectral mixing to generate the mixed signal further comprises:

receiving from a voice activity detector (VAD) a VAD output that is based on (i) the acoustic signals from the one or more microphones and (ii) the data output by the accelerometer;

when the VAD output indicates that no voice activity is detected, computing an acoustic noise power signal and an accelerometer noise power signal, wherein the acoustic noise power signal is a noise power signal in the acoustic signal from the one or more microphones and the accelerometer noise power signal is a noise power signal in the pre-emphasized accelerometer signal;

when an alternative non-stationary noise detector is employed it estimates the noise power in the acoustic signal and the accelerometer signal during intervals with either voice activity or no voice activity;

when the VAD output indicates that voice activity is detected, computing an acoustic power signal and an accelerometer power signal, wherein the acoustic power signal is a power signal during speech in the acoustic signal from the one or more microphones and the accelerometer power signal is a power signal during speech in the pre-emphasized accelerometer signal; and

generating (i) a final acoustic power signal by removing the acoustic noise power signal from the acoustic power signal and (ii) a final accelerometer power signal by removing the accelerometer noise power signal from the accelerometer power signal.

6. The method of claim 5, wherein performing spectral mixing to generate the mixed signal further comprises:

applying limits to the noise powers subtracted by the noise subtraction module in order to generate a positive low-frequency final accelerometer power signal and a positive low-frequency final acoustic power signal;

computing the power ratio between the low-frequency final accelerometer power signal and the low-frequency final acoustic power signal, wherein the low-frequency final accelerometer power signal and the low-frequency final acoustic power signal are within a same low frequency band; and

computing the scaling factor by smoothing the power ratio, limiting it to an allowable range, and by extracting the square root from the smoothed and limited power ratio.

7. The method of claim 6, wherein performing spectral mixing to generate the mixed signal further comprises:

applying a low-pass filter with a cutoff frequency (Fc) to the pre-emphasized accelerometer signal to generate a low-pass filtered pre-emphasized accelerometer signal; and

scaling the low-pass filtered pre-emphasized accelerometer signal using the scaling factor to generate a final accelerometer signal during the time when voice activity is detected (VAD=1); and

applying a certain fixed attenuation to the low-pass filtered pre-emphasized accelerometer signal when voice activity is not detected (VAD=0).

8. The method of claim 7, wherein performing spectral mixing to generate the mixed signal further comprises:

applying a high-pass filter with the cutoff frequency (Fc) to the acoustic signals from the one or more microphones to generate a final acoustic signal from the one or more microphones; and

mixing the scaled accelerometer signal with the final acoustic signal from the one or more microphones to generate the mixed signal.

9. The method of claim 8, further comprising:

calculating a delay between the final acoustic signal and the scaled accelerometer signal based on cross-correlation; and

applying the delay to the scaled accelerometer signal before mixing the scaled accelerometer signal with the final acoustic signal to generate the mixed signal.

10. The method of claim 9, further comprising:

receiving by a switch (i) the mixed signal and (ii) a speech signal from a beamformer, wherein the acoustic signals from the one or more microphones are received by the beamformer;

outputting by the switch the mixed signal when the acoustic noise power signal is greater than a noise threshold or when wind noise is detected by the one or more microphones; and

outputting by the switch the speech signal from the beamformer when the acoustic noise power signal is lesser than or equal to the noise threshold and when wind noise is not detected by the one or more microphones.

11. The method of claim 10, further comprising:

receiving by a noise suppressor (i) the output from the switch, (ii) the VAD output and (iii) a noise beam output from the beamformer; and

suppressing by the noise suppressor noise included in the output from the switch based on the VAD output and using a noise estimate from the noise beam output.

12. The method of claim 11, further comprising:

generating pitch estimate by a pitch detector based on autocorrelation method and using the output from the accelerometer, wherein the pitch estimate is obtained by (i) using an X, Y, or Z signal generated by the accelerometer that has a highest power level or (ii) using a combination of the X, Y, and Z signals generated by the accelerometer.

13. The method of claim 3, wherein receiving the output from the accelerometer further comprises:

receiving an output signal for each of the three axes of the accelerometer, wherein the output signal for each of the three axes are X, Y, and Z signals generated by the accelerometer, respectively;

determining a total power in each of the X, Y, and Z signals generated by the accelerometer, respectively; and

selecting the X, Y, or Z signal having the highest power as the output from the accelerometer.

14. The method of claim 3, wherein receiving the output from the accelerometer further comprises:

receiving an output signal for each of the three axes of the accelerometer, wherein the output signal for each of the three axes are X, Y, and Z signals generated by the accelerometer, respectively; and

computing an average of the X, Y, and Z signals to generate the output from the accelerometer.

15. The method of claim 3, wherein receiving the output from the accelerometer further comprises:

receiving an output signal for each of the three axes of the accelerometer, wherein the output signal for each of the three axes are X, Y, and Z signals generated by the accelerometer, respectively;

computing using cross-correlation a delay between the X and Y signals, a delay between the X and Z signals, and a delay between the Y and Z signals;

determining a most advanced signal from the X, Y, and Z signals based on the computed delays;

delaying a remaining two signals from the X, Y, and Z signals, the remaining two signals not including the most advanced signal; and

computing an average of the most advanced signal and the delayed remaining two signals to obtain the output of the accelerometer.

16. A system for improving voice quality in a mobile device comprising:

a headset including a pair of earbuds and a headset wire, wherein at least one of the earbuds includes an accelerometer, wherein the headset includes one or more microphones; and

a spectral mixer coupled to the headset to perform spectral mixing of the output from the accelerometer with acoustic signals from the one or more microphones to generate a mixed signal, wherein performing spectral mixing includes scaling the output from the accelerometer by a scaling factor based on a power ratio between the acoustic signals from the one or more microphones and the output from the accelerometer.

17. The system of claim 16, wherein the one or more microphones comprises a front microphone and a rear microphone in each of the earbuds.

18. The system of claim 16, wherein the spectral mixer pre-emphasizes the output from the accelerometer to account for lip radiation characteristic to generate a pre-emphasized accelerometer signal.

19. The system of claim 18, further comprising:

a voice activity detector (VAD) coupled to the headset, the VAD to generate a VAD output based on (i) acoustic signals received from the one or more microphones and (ii) data output by the accelerometer,

wherein

when the VAD output indicates that no voice activity is detected, the spectral mixer computes an acoustic noise power signal and an accelerometer noise power signal, wherein the acoustic noise power signal is a noise power signal in the acoustic signal from the one or more microphones and the accelerometer noise power signal is a noise power signal in the pre-emphasized accelerometer signal;

when an alternative non-stationary noise detector is employed it estimates the noise power in the acoustic signal and the accelerometer signal during intervals with either voice activity or no voice activity;

when the VAD output indicates that voice activity is detected, the spectral mixer computes an acoustic power signal and an accelerometer power signal, wherein the acoustic power signal is a power signal during speech in the acoustic signal from the or more microphones and the accelerometer power signal is a power signal during speech in the pre-emphasized accelerometer signal; and

the spectral mixer generates (i) a final acoustic power signal by removing the acoustic noise power signal from the acoustic power signal and (ii) a final accelerometer power signal by removing the accelerometer noise power signal from the accelerometer power signal.

20. The system of claim 19, wherein the spectral mixer further:

applies limits to the noise removed in order to generate a positive low-frequency final accelerometer power signal and a positive low-frequency final acoustic power signal;

computes the power ratio between the low-frequency final acoustic power signal and the low-frequency final accelerometer power signal, wherein the low-frequency final accelerometer power signal and the low-frequency final acoustic power signal are within a same low frequency band; and

computes the scaling factor by smoothing the power ratio, limiting the power ratio to an allowable range, and by computing the square root of the smoothed and limited power ratio.

21. The system of claim 20, wherein the spectral mixer further:

applies a low-pass filter with a cutoff frequency (Fc) to the pre-emphasized accelerometer signal to generate a low-pass filtered pre-emphasized accelerometer signal; and

scales the low-pass filtered pre-emphasized accelerometer signal using the scaling factor to generate a final accelerometer signal when voice activity is detected (VAD=1); and

applies a certain fixed attenuation to the low-pass filtered pre-emphasized accelerometer signal with when voice activity is not detected (VAD=0).

22. The system of claim 21, wherein the spectral mixer further:

applies a high-pass filter with the cutoff frequency (Fc) to the acoustic signals from the one or more microphones to generate a final acoustic signal from the one or more microphones; and

mixes the final accelerometer signal with the final acoustic signal from the one or more microphones to generate the mixed signal.

23. The system of claim 22, wherein the spectral mixer further:

calculates a delay between the final accelerometer signal and the final acoustic signal based on cross-correlation; and

applies the delay to the final accelerometer signal before mixing with the final acoustic signal to generate the mixed signal.

24. The system of claim 23, further comprising:

a beamformer to receive the acoustic signals from the one or more microphones and generate an enhanced acoustic signal; and

a switch to receive (i) the mixed signal from the spectral mixer and (ii) a speech signal from the beamformer, and to output the mixed signal when the acoustic noise power signal is greater than a threshold or when wind noise is detected by the one or more microphones, and to output the speech signal from the beamformer when the acoustic noise power signal is lesser than or equal to a threshold and when wind noise is not detected.

25. The system of claim 24, further comprising:

a noise suppressor coupled to the switch and the VAD, the noise suppressor to suppress noise from the output from the switch based on the VAD output and a noise estimate and to output a noise suppressed speech output.

26. The system of claim 25, further comprising:

a pitch detector to generate a pitch estimate based on the output from the accelerometer, wherein the pitch detector generates the pitch estimate based on autocorrelation method by (i) using an X, Y, or Z signal generated by the accelerometer that has a highest power level or (ii) using a combination of the X, Y, and Z signals generated by the accelerometer.

27. The system of claim 26, further comprising:

a speech codec coupled to the noise suppressor, the VAD, and the pitch detector, the speech codec to employ an enhanced pitch and an enhanced VAD, both computed based on the accelerometer signal.

28. The system of claim 21, wherein the spectral mixer further:

receives an enhanced acoustic signal from a beamformer that receives acoustic signals from the one or more microphones and an output from the VAD;

applies a high-pass filter with the cutoff frequency (Fc) to the enhanced acoustic signal from the beamformer to generate a final acoustic signal from the beamformer; and

mixes the final scaled accelerometer signal with the final acoustic signal from the beamformer to generate the mixed signal.