Method and apparatus for multi-sensory speech enhancement on a mobile device

• US 20050185813A1
• Filed: 02/24/2004
• Published: 08/25/2005
• Est. Priority Date: 02/24/2004
• Status: Active Grant

• Current Assignee: Zhigu Holdings Limited
• Original Assignee: Microsoft Corporation
• Inventors: Sinclair, Michael J., Zhang, Zhengyou, Huang, Xuedong David
• Granted Patent: US 7,499,686 B2
• Time in Patent Office: Days
• US Class Current: 381/380
• CPC Class Codes: G10L 21/0208 Noise filtering
• CPC Class Codes: H04M 1/6008 in the transmitter circuit
• CPC Class Codes: H04M 1/6016 in the receiver circuit
• CPC Class Codes: H04M 1/605 involving control of the receiver volume to provide a dual operational mode at close or far distance from the user
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion telephones including GPS signal receivers H04M2250/10
• CPC Class Codes: H04R 2460/13 Hearing devices using bone conduction transducers
• CPC Class Codes: H04R 2499/11 Transducers incorporated or for use in hand-held devices, e.g. mobile phones, PDA's, camera's
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407

Abstract

A mobile device is provided that includes a digit input that can be manipulated by a user'"'"'s fingers or thumb, an air conduction microphone and an alternative sensor that provides an alternative sensor signal indicative of speech. Under some embodiments, the mobile device also includes a proximity sensor that provides a proximity signal indicative of the distance from the mobile device to an object. Under some embodiments, the signal from the air conduction microphone, the alternative sensor signal, and the proximity signal are used to form an estimate of a clean speech value. In further embodiments, a sound is produced through a speaker in the mobile device based on the amount of noise in the clean speech value. In other embodiments, the sound produced through the speaker is based on the proximity sensor signal.

Claims

1. A mobile hand-held device comprising:

an air conduction microphone that converts acoustic waves into an electric microphone signal;

an alternative sensor that provides an electric alternative sensor signal indicative of speech; and

a processor that uses the microphone signal and the alternative sensor signal to estimate a clean speech value.

2. The mobile hand-held device of claim 1 wherein the alternative sensor comprises a bone conduction sensor.

3. The mobile hand-held device of claim 1 further comprising a second alternative sensor that provides a second alternative sensor signal.

4. The mobile hand-held device of claim 3 wherein the alternative sensor and the second alternative sensor comprise bone conduction sensors.

5. The mobile hand-held device of claim 4 wherein the mobile hand-held device has a left side and a right side opposite the left side and wherein the alternative sensor is located near the left side and the second alternative sensor is located near the right side.

6. The mobile hand-held device of claim 3 further comprising a selection unit that selects one of the alternative sensor signal and the second alternative sensor signal.

7. The mobile hand-held device of claim 6 wherein the selection unit selects one of the alternative sensor signal and the second alternative sensor signal based on the magnitudes of the alternative sensor signal and the second alternative sensor signal.

8. The mobile hand-held device of claim 1 further comprising a speaker that generates a sound based on the amount of noise in the clean speech value.

9. The mobile hand-held device of claim 1 further comprising a proximity sensor that produces a proximity signal indicative of the distance between the mobile hand-held device and an object.

10. The mobile hand-held device of claim 9 wherein the processor determines the clean speech value based on the microphone signal, the alternative sensor signal and the proximity signal.

11. The mobile hand-held device of claim 10 wherein the processor determines the clean speech value through a process comprising:

determining a microphone contribution to the clean speech value based on the microphone signal;

determining an alternative sensor contribution to the clean speech value based on the alternative sensor signal; and

weighting the microphone contribution and the alternative sensor contribution based on the proximity signal.

12. The mobile hand-held device of claim 9 further comprising a speaker that generates a sound based on the proximity signal.

13. The mobile hand-held device of claim 1 wherein the alternative sensor comprises a pressure transducer that is hydraulically coupled to a pad filled with a medium.

14. The mobile hand-held device of claim 13 wherein the mobile hand-held device has a left side and a right side opposite the left side and wherein the pad has a first portion on the left side and a second portion on the right side.

15. The mobile hand-held device of claim 13 wherein the alternative sensor further provides a proximity signal.

16. The mobile hand-held device of claim 15 wherein the proximity signal comprises a DC component of an electrical signal produced by the pressure transducer.

17. The mobile hand-held device of claim 16 wherein the alternative sensor signal comprises an AC component of the electrical signal produced by the pressure transducer.

18. The mobile hand-held device of claim 1 wherein the alternative sensor comprises an infrared sensor.

19. A mobile device comprising:

an air conduction microphone that converts acoustic waves into an electric microphone signal;

an alternative sensor that provides an electric alternative sensor signal indicative of speech; and

a proximity sensor that provides an electric proximity signal that is indicative of the distance from the mobile device to an object.

20. The mobile device of claim 19 wherein the mobile device uses the microphone signal, the alternative sensor signal and the proximity signal to remove noise from the microphone signal and thereby produce an enhanced clean speech signal.

21. The mobile device of claim 20 further comprising a speaker that produces a sound based on an estimate of the level of noise in the enhanced clean speech signal.

22. The mobile device of claim 20 wherein the mobile device produces the enhanced clean speech signal by using the proximity signal to weight a contribution to the enhanced clean speech signal that is formed from the alternative sensor signal.

23. The mobile device of claim 22 wherein weighting the contribution comprises giving the contribution less weight when the proximity signal indicates that the mobile device is far from the object.

24. The mobile device of claim 19 further comprising a speaker that produces a sound based on the proximity signal.

25. The mobile device of claim 24 wherein the volume of the sound increases as the proximity signal indicates that the distance between the mobile device and the object increases.

26. The mobile device of claim 19 wherein the alternative sensor signal and the proximity sensor signal are produced by a single sensor.

27. The mobile device of claim 26 wherein the single sensor comprises a pressure transducer that provides an electrical signal, the electrical signal having a DC component that represents the proximity signal and an AC component that represents the alternative sensor signal.

28. A method in a mobile device, the method comprising:

receiving an air conduction microphone signal;

receiving an alternative sensor signal that is indicative of speech;

estimating an enhanced clean speech value based on the air conduction microphone signal and the alternative sensor signal;

estimating the noise in the enhanced clean speech value; and

using the estimate of the noise to generate a sound through a speaker in the mobile device.

29. The method of claim 28 wherein the volume of the sound increases as the estimate of the noise increases.

30. The method of claim 28 further comprising receiving a proximity sensor signal that indicates the distance between the mobile device and an object and using the proximity sensor signal when estimating the enhanced clean speech value.

31. The method of claim 30 wherein using the proximity sensor signal comprises weighting a contribution to the enhanced clean speech value that is derived from the alternative sensor signal based on the proximity sensor signal.

32. The method of claim 31 wherein weighting a contribution comprises applying a greater weight to the contribution derived from the alternative sensor signal when the proximity sensor signal indicates that the mobile device is close to an object.

33. The method of claim 32 wherein the object is a user'"'"'s head.