Sensor Fusion to Improve Speech/Audio Processing in a Mobile Device

• US 20130332156A1
• Filed: 02/22/2013
• Published: 12/12/2013
• Est. Priority Date: 06/11/2012
• Status: Abandoned Application

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Zhang, Haining, Mathias, Arun G., Tackin, Onur Ergin, Karahan, Sinan, Theverapperuma, Lalin S., Shao, Tiange
• Application Number:
  US13/775,100
• Publication Number:
  US 20130332156A1
• Time in Patent Office: Days
• US Class Current: 704/226
• CPC Class Codes: G10L 21/0216 characterised by the method used for estimating noise
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: H03G 3/3005 in amplifiers suitable for low-frequencies, e.g. audio amplifiers H03G3/32, H03G3/34 take precedence
• CPC Class Codes: H03G 3/3089 Control of digital or coded signals
• CPC Class Codes: H04M 1/6008 in the transmitter circuit
• CPC Class Codes: H04M 1/6041 Portable telephones adapted for handsfree use
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion telephones including GPS signal receivers H04M2250/10
• CPC Class Codes: H04R 2499/11 Transducers incorporated or for use in hand-held devices, e.g. mobile phones, PDA's, camera's
• CPC Class Codes: H04R 29/004 for microphones H04R29/007 takes precedence
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04R 3/02 for preventing acoustic reaction , i.e. acoustic oscillatory feedback specially adapted for hearing aids H04R25/453

Abstract

The disclosed system and method for a mobile device combines information derived from onboard sensors with conventional signal processing information derived from a speech or audio signal to assist in noise and echo cancellation. In some implementations, an Angle and Distance Processing (ADP) module is employed on a mobile device and configured to provide runtime angle and distance information to an adaptive beamformer for canceling noise signals, provides a means for building a table of filter coefficients for adaptive filters used in echo cancellation, provides faster and more accurate Automatic Gain Control (AGC), provides delay information for a classifier in a Voice Activity Detector (VAD), provides a means for automatic switching between a speakerphone and handset mode of the mobile device, or primary microphone and reference microphones and assists in separating echo path changes from double talk.

Claims

1. A computer-implemented method performed by one or more processors of a mobile device, comprising:

receiving data from one or more sensors of a mobile device;

calculating an orientation and distance of a signal source relative to a first microphone of the mobile device based on the data;

receiving a signal from the source through the first microphone; and

processing the signal based on the calculated orientation and distance.

2. The method of claim 1, where processing comprises:

calculating a gain based on the distance; and

automatically applying the gain to the signal received through the first microphone.

3. The method of claim 2, where automatically applying the gain, comprises:

comparing the calculated gain with an estimated gain; and

determining whether to apply the calculated gain to the signal based on results of the comparison.

4. The method of claim 3, where processing comprises:

determining a gain error based on the calculated gain and the estimated gain; and

applying either the calculated gain or the estimated gain to the signal received through the first microphone based on the gain error.

5. The method of claim 1, where processing comprises:

automatically tracking the source of the signal received through the first microphone using the calculated orientation and distance.

6. The method of claim 5, where the tracking is performed by a Minimum Variance Distortionless Response (MVDR) beamformer.

7. The method of claim 1, where processing comprises:

selecting coefficients of an adaptive filter of an echo canceller based on the orientation or distance.

8. The method of claim 1, further comprising:

estimating a delay between receipt of the signal at the first microphone and receipt of the signal at a second microphone of the mobile device, the second microphone having a fixed orientation and distance relative to the first microphone.

9. The method of claim 8, further comprising:

detecting whether the signal includes speech based on the estimated delay.

10. The method of claim 8, further comprising:

aligning signals received through the first and second microphones in time using the estimated delay;

estimating noise on the aligned signals; and

canceling noise from a combined signal using the estimated noise, where the combined signal includes the signals received through the first and second microphones.

11. A computer-implemented method performed by one or more processors of a mobile device, comprising:

receiving sensor data;

computing an angle and distance from the sensor data, the angle defining a relative orientation of a speech signal source and a microphone of the mobile device, the distance defining a distance between the speech signal source and the microphone;

receiving a speech signal from the speech signal source through the microphone; and

performing at least one of noise cancellation, echo cancellation, voice activity detection, switching from handset to speakerphone mode, or automatic gain control based on the angle.

12. A system comprising:

a first microphone;

a sensor configured for providing sensor output data in response to a change of position of the system;

a processor coupled to the sensor and the first microphone and programmed for;

receiving data from one or more sensors of a mobile device;

calculating an orientation and distance of a signal source relative to a first microphone of the mobile device based on the data;

receiving a signal from the source through the first microphone; and

processing the signal based on the calculated orientation and distance.

13. The system of claim 12, where the processor is programmed for:

calculating a gain based on the distance; and

automatically applying the gain to the signal received through the first microphone.

14. The system of claim 13, where automatically applying the gain, comprises:

comparing the calculated gain with an estimated gain; and

determining whether to apply the calculated gain to the signal based on results of the comparison.

15. The system of claim 12, where the processor is programmed for:

determining a gain error based on the calculated gain and the estimated gain; and

applying either the calculated gain or the estimated gain to the signal received through the first microphone based on the gain error.

16. The system of claim 12, where the processor is programmed for:

automatically tracking the source of the signal received through the first microphone using the calculated orientation and distance.

17. The system of claim 16, where the tracking is performed by a Minimum Variance Distortionless Response (MVDR) beamformer.

18. The system of claim 12, where the processor is programmed for:

selecting coefficients of an adaptive filter of an echo canceller based on the orientation or distance.

19. The system of claim 12, where the processor is programmed for:

estimating a delay between receipt of the signal at the first microphone and receipt of the signal at a second microphone of the mobile device, the second microphone having a fixed orientation and distance relative to the first microphone.

20. The system of claim 19, where the processor is further programmed for:

detecting whether the signal includes speech based on the estimated delay.

21. The system of claim 19, where the processor is further programmed for:

aligning signals received through the first and second microphones in time using the estimated delay;

estimating noise on the aligned signals; and

canceling noise from a combined signal using the estimated noise, where the combined signal includes the signals received through the first and second microphones.

22. A mobile device comprising:

one or more sensors configured to generate data in response to motion of the mobile device;

one or more microphones;

one or more processors coupled to the one or more sensors and the one or more microphones and programmed for;

receiving data from the sensor;

calculating an orientation and distance of a signal source relative to the one or more microphones based on the sensor data;

receiving a signal from the source through at least one microphone; and

processing the signal based on the calculated orientation and distance.

23. The mobile device of claim 22, where processing includes performing at least one of noise cancellation, echo cancellation, voice activity detection, switching from handset to speakerphone mode, or automatic gain control based on the orientation or distance.