Systems, methods, apparatus, and computer-readable media for spatially selective audio augmentation

• US 9,037,458 B2
• Filed: 02/21/2012
• Issued: 05/19/2015
• Est. Priority Date: 02/23/2011
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Park, Hyun Jin, Chan, Kwokleung, Li, Ren
• Primary Examiner(s): ALBERTALLI, BRIAN LOUIS
• Application Number:
  US13/401,688
• Publication Number:
  US 20120215519A1
• Time in Patent Office: 1,183 Days
• US Class Current: 704/226
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G10K 11/17823 Reference signals, e.g. ambient acoustic environment
• CPC Class Codes: G10K 11/17837 by retaining part of the ambient acoustic environment, e.g. speech or alarm signals that the user needs to hear
• CPC Class Codes: G10K 11/17853 of the filter
• CPC Class Codes: G10K 11/17854 the filter being an adaptive filter
• CPC Class Codes: G10K 11/17857 Geometric disposition, e.g. placement of microphones
• CPC Class Codes: G10K 11/17881 the reference signal being an acoustic signal, e.g. recorded with a microphone
• CPC Class Codes: G10L 2021/02166 Microphone arrays; Beamforming
• CPC Class Codes: G10L 21/0208 Noise filtering
• CPC Class Codes: H04R 1/1083 Reduction of ambient noise active noise reduction per se G10K11/175; protective devices for the ear, e.g. providing acoustic protection A61F11/06
• CPC Class Codes: H04R 2460/01 Hearing devices using active noise cancellation
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04R 5/033 Headphones for stereophonic communication details thereof, e.g. relating to batteries, cables or control elements H04R1/10

Abstract

Spatially selective augmentation of a multichannel audio signal is described.

Claims

1. A method of processing a multichannel signal that includes a first audio input channel and a second audio input channel, said method comprising:

performing an active noise cancellation operation on the first audio input channel to produce an antinoise signal;

processing the multichannel signal to obtain a source component, said processing including performing a spatially selective processing operation on the multichannel signal to separate the source component from a background component, wherein said spatially selective processing operation is based on at least gain differences between the first and second audio input channels;

processing the obtained source component to produce a target component; and

combining the antinoise signal and an augmented signal that is based on the target component to produce an audio output signal.

2. The method of signal processing according to claim 1, wherein said method comprises combining the target component and the separated background component to produce the augmented signal.

3. The method of signal processing according to claim 1, wherein said spatially selective processing operation includes applying a beam in a source direction to produce the source component.

4. The method of signal processing according to claim 1, wherein said spatially selective processing operation includes applying a null beam in the source direction to produce the background component.

5. The method of signal processing according to claim 4, wherein said spatially selective processing operation includes applying a second null beam in a direction of a noise source to produce the background component.

6. The method of signal processing according to claim 1, wherein said spatially selective processing operation is based on at least phase differences between the first and second audio input channels.

7. The method of signal processing according to claim 1, wherein said processing the obtained source component comprises increasing a perceptibility of the obtained source component by at least one among amplifying the separated source component, compressing a dynamic range of the separated source component, dereverberating the separated source component, and altering a frequency spectrum of the separated source component.

8. The method of signal processing according to claim 7, wherein said increasing a perceptibility includes increasing a perceptibility of speech content of the separated source component.

9. The method of signal processing according to claim 8, wherein said increasing a perceptibility of said speech content includes at least one among (A) varying a speed of said speech content and (B) translating said speech content from a first spoken language to a second spoken language that is different from the first spoken language.

10. The method of signal processing according to claim 8, wherein said increasing a perceptibility of said speech content includes producing said target component to have a greater number of pitch periods than said separated source component.

11. The method of signal processing according to claim 1, wherein said method comprises driving a loudspeaker to produce an acoustic signal that is based on the audio output signal, wherein the loudspeaker is disposed to be worn at an ear of a user and directed at an ear canal of the user, and

wherein said active noise cancellation operation is based on a signal that is produced by an error microphone, wherein the error microphone is disposed to be directed at the ear canal.

12. The method of signal processing according to claim 1, wherein said performing a spatially selective processing operation on the multichannel signal to separate the source component from the background component includes separating the voice of a person other than the user from the background component.

13. An apparatus for processing a multichannel signal that includes a first audio input channel and a second audio input channel, said apparatus comprising:

means for performing an active noise cancellation operation on the first audio input channel to produce an antinoise signal;

means for processing the multichannel signal to obtain a source component, said means for processing including means for performing a spatially selective processing operation on the multichannel signal to separate the source component from a background component, wherein said spatially selective processing operation is based on at least gain differences between the first and second audio input channels;

means for processing the obtained source component to produce a target component; and

means for combining the antinoise signal and an augmented signal that is based on the target component to produce an audio output signal.

14. The apparatus according to claim 13, wherein said apparatus includes means for combining the target component and the separated background component to produce the augmented signal.

15. The apparatus according to claim 13, wherein said spatially selective processing operation includes applying a beam in a source direction to produce the source component.

16. The apparatus according to claim 13, wherein said spatially selective processing operation includes applying a null beam in the source direction to produce the background component.

17. The apparatus according to claim 16, wherein said spatially selective processing operation includes applying a second null beam in a direction of a noise source to produce the background component.

18. The apparatus according to claim 13, wherein said spatially selective processing operation is based on at least phase differences between the first and second audio input channels.

19. The apparatus according to claim 13, wherein said means for processing the obtained source component comprises means for increasing a perceptibility of the obtained source component by at least one among amplifying the separated source component, compressing a dynamic range of the separated source component, dereverberating the separated source component, and altering a frequency spectrum of the separated source component.

20. The apparatus according to claim 19, wherein said increasing a perceptibility includes increasing a perceptibility of speech content of the separated source component.

21. The apparatus according to claim 20, wherein said increasing a perceptibility of said speech content includes at least one among (A) varying a speed of said speech content and (B) translating said speech content from a first spoken language to a second spoken language that is different from the first spoken language.

22. The apparatus according to claim 20, wherein said increasing a perceptibility of said speech content includes producing said target component to have a greater number of pitch periods than said separated source component.

23. The apparatus according to claim 13, wherein said apparatus comprises means for driving a loudspeaker to produce an acoustic signal that is based on the audio output signal, wherein the loudspeaker is disposed to be worn at an ear of a user and directed at an ear canal of the user, and

wherein said active noise cancellation operation is based on a signal that is produced by an error microphone, wherein the error microphone is disposed to be directed at the ear canal.

24. The apparatus according to claim 13, wherein said performing a spatially selective processing operation on the multichannel signal to separate the source component from the background component includes separating the voice of a person other than the user from the background component.

25. An apparatus for processing a multichannel signal that includes a first audio input channel and a second audio input channel, said apparatus comprising:

an active noise cancellation filter configured to perform an active noise cancellation operation on the first audio input channel to produce an antinoise signal;

an augmented reality processing module configured to process the multichannel signal to obtain a source component, said module including;

a spatially selective filter configured to perform a spatially selective processing operation on the multichannel signal to separate the source component from a background component, wherein said spatially selective processing operation is based on at least gain differences between the first and second audio input channels; and

an augmentation module configured (A) to process the obtained source component to produce a target component and (B) to output an augmented signal that is based on the target component; and

an audio output stage configured to combine the antinoise signal and the augmented signal to produce an audio output signal.

26. The apparatus according to claim 25, wherein said augmentation module is configured to combine the target component and the separated background component to produce the augmented signal.

27. The apparatus according to claim 25, wherein said spatially selective processing operation includes applying a beam in a source direction to produce the source component.

28. The apparatus according to claim 25, wherein said spatially selective processing operation includes applying a null beam in the source direction to produce the background component.

29. The apparatus according to claim 28, wherein said spatially selective processing operation includes applying a second null beam in a direction of a noise source to produce the background component.

30. The apparatus according to claim 25, wherein said spatially selective processing operation is based on at least phase differences between the first and second audio input channels.

31. The apparatus according to claim 25, wherein said augmentation module is configured to increase a perceptibility of the obtained source component by at least one among amplifying the separated source component, compressing a dynamic range of the separated source component, dereverberating the separated source component, and altering a frequency spectrum of the separated source component.

32. The apparatus according to claim 25, wherein said augmentation module is configured to increase a perceptibility of speech content of the separated source component.

33. The apparatus according to claim 32, wherein said increasing a perceptibility of said speech content includes at least one among (A) varying a speed of said speech content and (B) translating said speech content from a first spoken language to a second spoken language that is different from the first spoken language.

34. The apparatus according to claim 32, wherein said increasing a perceptibility of said speech content includes producing said target component to have a greater number of pitch periods than said separated source component.

35. The apparatus according to claim 25, wherein said audio output stage is configured to drive a loudspeaker to produce an acoustic signal that is based on the audio output signal, wherein the loudspeaker is disposed to be worn at an ear of a user and directed at an ear canal of the user, and

wherein said active noise cancellation operation is based on a signal that is produced by an error microphone, wherein the error microphone is disposed to be directed at the ear canal.

36. The apparatus according to claim 25, wherein said performing a spatially selective processing operation on the multichannel signal to separate the source component from the background component includes separating the voice of a person other than the user from the background component.

37. A non-transitory computer-readable storage medium having tangible features that cause a machine reading the features to:

perform an active noise cancellation operation on the first audio input channel to produce an antinoise signal;

process the multichannel signal to obtain a source component, said processing including performing a spatially selective processing operation on a multichannel signal that includes a first audio input channel and a second audio input channel to separate the source component from a background component, wherein said spatially selective processing operation is based on at least gain differences between the first and second audio input channels;

process the obtained source component to produce a target component; and

combine the antinoise signal and an augmented signal that is based on the target component to produce an audio output signal.

38. The medium according to claim 37, wherein said medium includes tangible features that cause a machine reading the features to combine the target component and the separated background component to produce the augmented signal.

39. The medium according to claim 37, wherein said spatially selective processing operation includes applying a beam in a source direction to produce the source component.

40. The medium according to claim 37, wherein said spatially selective processing operation includes applying a null beam in the source direction to produce the background component.

41. The medium according to claim 40, wherein said spatially selective processing operation includes applying a second null beam in a direction of a noise source to produce the background component.

42. The medium according to claim 37, wherein said spatially selective processing operation is based on at least phase differences between the first and second audio input channels.

43. The medium according to claim 37, wherein said processing the obtained source component comprises increasing a perceptibility of the obtained source component by at least one among amplifying the separated source component, compressing a dynamic range of the separated source component, dereverberating the separated source component, and altering a frequency spectrum of the separated source component.

44. The medium according to claim 43, wherein said increasing a perceptibility includes increasing a perceptibility of speech content of the separated source component.

45. The medium according to claim 44, wherein said increasing a perceptibility of said speech content includes at least one among (A) varying a speed of said speech content and (B) translating said speech content from a first spoken language to a second spoken language that is different from the first spoken language.

46. The medium according to claim 44, wherein said increasing a perceptibility of said speech content includes producing said target component to have a greater number of pitch periods than said separated source component.

47. The medium according to claim 37, wherein said medium includes tangible features that cause a machine reading the features to drive a loudspeaker to produce an acoustic signal that is based on the audio output signal, wherein the loudspeaker is disposed to be worn at an ear of a user and directed at an ear canal of the user, and

wherein said active noise cancellation operation is based on a signal that is produced by an error microphone, wherein the error microphone is disposed to be directed at the ear canal.

48. The medium according to claim 37, wherein said performing a spatially selective processing operation on the multichannel signal to separate the source component from the background component includes separating the voice of a person other than the user from the background component.