Pitch-based frequency domain voice removal

• US 8,219,390 B1
• Filed: 09/16/2003
• Issued: 07/10/2012
• Est. Priority Date: 09/16/2003
• Status: Active Grant

• Current Assignee: Creative Technology Ltd.
• Original Assignee: Creative Technology Ltd.
• Inventors: Laroche, Jean
• Primary Examiner(s): Saint Cyr, Leonard
• Application Number:
  US10/663,446
• Time in Patent Office: 3,220 Days
• Field of Search: 704/205, 704/207, 704/208, 704/209, 704/214, 704/216, 704/217
• US Class Current: 704/207
• CPC Class Codes: G10H 1/361 Recording/reproducing of accompaniment for use with an external source, e.g. karaoke systems
• CPC Class Codes: G10H 2210/066 for pitch analysis as part of wider processing for musical purposes, e.g. transcription, musical performance evaluation; Pitch recognition, e.g. in polyphonic sounds; Estimation or use of missing fundamental
• CPC Class Codes: G10H 2250/235 Fourier transform; Discrete Fourier Transform [DFT]; Fast Fourier Transform [FFT]
• CPC Class Codes: G10L 21/0272 Voice signal separating
• CPC Class Codes: G10L 25/90 Pitch determination of speech signals

Abstract

A system and method are disclosed for modifying an audio signal. A pitch associated with the audio signal is detected. A portion of the audio signal that is associated with the detected pitch is modified. Controlling the modification of a primary audio signal is disclosed. The level of a secondary audio signal is monitored. Modification of the primary audio signal is enabled if the level of the secondary audio signal rises above a first prescribed threshold at a time when the primary audio signal is not being modified. Modification of the primary audio signal is disabled if the level of the secondary audio signal drops below a second prescribed threshold at a time when the primary audio signal is being modified.

Claims

1. A method for modifying an audio signal with a pitch-based signal component removal device, comprising:

detecting a first most prominent pitch associated with the audio signal including;

transforming the audio signal into a time frequency domain to generate short time frequency spectra for the audio signal;

obtaining a plurality of pitch candidate frequency domain combs associated with a plurality of pitch candidates;

performing a cross-correlation in the frequency domain using information associated with the short time frequency spectra and the plurality of pitch candidate frequency domain combs in generating cross-correlation values; and

identifying the pitch candidate associated with a first maximum value among the cross-correlation values as the first most prominent pitch;

detecting a second most prominent pitch associated with the audio signal by removing cross-correlation values associated with the first most prominent pitch from consideration and identifying the pitch candidate associated with a second maximum value among the remaining cross-correlation values as the second most prominent pitch; and

in the event the second most prominent pitch is associated with voice, modifying in the audio signal a portion that is associated with the second most prominent pitch.

2. The method of claim 1, wherein modifying comprises removing from the signal said portion that is associated with the detected pitch.

3. The method of claim 2, wherein removing comprises attenuating said portion that is associated with the detected pitch.

4. The method of claim 3, wherein attenuating comprises partially attenuating said portion that is associated with the detected pitch.

5. The method of claim 1, wherein transforming the audio signal into a time frequency domain comprises processing the audio signal using a subband filter bank.

6. The method of claim 5, wherein transforming the audio signal into a time frequency domain comprises applying the short time Fourier transform to the signal.

7. The method of claim 5, wherein transforming the audio signal into a time frequency domain comprises performing a wavelet transform.

8. The method of claim 1, wherein transforming the audio signal into a time frequency domain comprises generating short time frequency spectra for the audio signal and wherein the step of modifying comprises modifying portions of the short time frequency spectra that are associated with the detected pitch to generate modified short time frequency spectra.

9. The method of claim 8, further comprising using said modified short time frequency spectra to synthesize a modified time-domain audio signal.

10. The method of claim 1, wherein detecting a pitch comprises detecting a pitch believed to be associated with a voice component.

11. The method of claim 1, wherein detecting a pitch comprises detecting a plurality of pitches.

12. The method of claim 1, wherein detecting a pitch comprises extracting a center-panned signal from the audio signal and processing said extracted center-panned signal to detect the pitch.

13. The method of claim 1, further comprising deciding whether the detected pitch is associated with a voice component.

14. The method of claim 13, further comprising performing said step of modifying only if it is decided that the detected pitch is associated with a voice component.

15. The method of claim 13, wherein deciding whether the detected pitch is associated with a voice component comprises comparing a value associated with the detected pitch with a prescribed threshold value.

16. The method of claim 15, wherein the value comprises a cross-correlation value.

17. The method of claim 16, wherein the cross-correlation value comprises a measure of the extent to which the audio signal correlates with a signal associated with a pitch candidate.

18. The method of claim 1, wherein detecting a pitch comprises:

determining spectral magnitude values for said short time frequency spectra; and

cross-correlating includes cross-correlating the spectral magnitude values with a plurality of pitch candidate frequency domain combs.

19. The method of claim 18, wherein cross-correlating the spectral magnitude values with a plurality of pitch candidate frequency domain combs yields a cross-correlation value associated with each pitch candidate and the pitch candidate having the highest cross-correlation value is identified as the detected pitch.

20. The method of claim 1, wherein modifying comprises removing from the signal said portion that is associated with the detected pitch by applying a gain to said portion that is associated with the detected pitch.

21. The method of claim 1, wherein modifying comprises identifying said portion associated with the detected pitch.

22. The method of claim 21, wherein identifying said portion associated with the detected pitch comprises:

transforming the audio signal into a time frequency domain; and

selecting for modification a frequency bin associated with the detected pitch.

23. The method of claim 22, wherein:

modifying further comprises applying a gain to said portion that is associated with the detected pitch; and

the gain is determined at least in part based on the extent to which the frequency bin is associated with a center-panned component of the audio signal.

24. The method of claim 23, wherein the audio signal comprises a left channel signal and a right channel signal and wherein the extent to which the frequency bin is associated with a center-panned component of the audio signal is determined by comparing the left channel frequency spectra associated with the frequency bin with the right channel frequency spectra associated with the frequency bin.

25. The method of claim 22, wherein selecting for modification comprises selecting a frequency bin closest to the detected pitch.

26. The method of claim 22, wherein selecting for modification comprises selecting a frequency bin closest to a harmonic of the detected pitch.

27. The method of claim 22, wherein selecting for modification comprises selecting a frequency bin closest to a subharmonic of the detected pitch.

28. The method of claim 23, wherein selecting for modification comprises selecting a range of frequency bins comprising a frequency bin closest to a harmonic of the detected pitch.

29. The method of claim 28, wherein the harmonic is the first harmonic.

30. The method of claim 28, wherein modifying comprises applying a gain to each frequency bin in said range of frequency bins.

31. The method of claim 30, wherein a separate gain is calculated for each frequency bin in the range of frequency bins.

32. The method of claim 31, wherein the gain for each respective frequency bin is determined at least in part based on the extent to which the frequency bin is associated with a center-panned component of the audio signal.

33. The method of claim 32, wherein the audio signal comprises a left channel signal and a right channel signal and wherein the extent to which a frequency bin is associated with a center-panned component of the audio signal is determined by comparing the left channel frequency spectra associated with the frequency bin with the right channel frequency spectra associated with the frequency bin.

34. The method of claim 30, wherein the same gain is applied to each frequency bin in the range of frequency bins.

35. The method of claim 34, wherein the gain is determined based on a selected frequency bin in the range of frequency bins.

36. The method of claim 35, wherein the selected frequency bin is the frequency bin closest to the harmonic of the detected pitch.

37. The method of claim 35, wherein the gain is determined at least in part based on the extent to which the selected frequency bin is associated with a center-panned component of the audio signal.

38. The method of claim 1, wherein modifying comprises amplifying said portion that is associated with the detected pitch relative to portions not associated with the detected pitch.

39. The method of claim 38, wherein amplifying said portion that is associated with the detected pitch relative to portions not associated with the detected pitch comprises enhancing said portion that is associated with the detected pitch while leaving said portions not associated with the detected pitch unchanged.

40. The method of claim 38, wherein amplifying said portion that is associated with the detected pitch relative to portions not associated with the detected pitch comprises leaving said portion that is associated with the detected pitch unchanged while attenuating said portions not associated with the detected pitch.

41. The method of claim 1, wherein the audio signal is a primary audio signal;

the method further includes monitoring the level of a secondary audio signal;

the method further includes enabling modification of the primary audio signal if the level of the secondary audio signal rises above a first prescribed threshold at a time when the primary audio signal is not being modified; and

the method further includes disabling modification of the primary audio signal if the level of the secondary audio signal drops below a second prescribed threshold at a time when the primary audio signal is being modified.

42. The method of claim 41, wherein the secondary audio signal comprises a signal generated by a microphone.

43. The method of claim 41, wherein the first prescribed threshold and the second prescribed threshold are the same.

44. The method of claim 41, wherein disabling processing comprises bypassing a system configured to perform the modification.

45. The method of claim 41, wherein disabling processing comprises bypassing or disabling a component of a system configured to perform the modification.

46. The method of claim 41, wherein the modification comprises:

detecting a pitch associated with the primary audio signal; and

modifying in the primary audio signal a portion that is associated with the detected pitch.

47. The method of claim 46, wherein:

detecting a pitch comprises detecting a pitch believed to be associated with a voice component; and

modifying the audio signal comprises removing said voice component from the audio signal.

48. The method of claim 46, wherein disabling modification comprises bypassing the step of detecting a pitch associated with an audio signal.

49. The method of claim 46, wherein disabling modification comprises bypassing the step of modifying in the audio signal a portion that is associated with the detected pitch.

50. The method of claim 1, wherein removing cross-correlation values associated with the first most prominent pitch includes:

zeroing out the cross-correlation values around the first most prominent pitch, its multiples and submultiples.

51. The method of claim 1 further comprising determining whether the second most prominent pitch is associated with voice.

52. The method of claim 51, wherein determining whether the second most prominent pitch is associated with voice is based at least in part on whether the second most prominent pitch is within a predefined frequency range.

53. The method of claim 51, wherein determining whether the second most prominent pitch is associated with voice is based at least in part on a comparison between a maximum correlation value and a predefined threshold.

54. The method of claim 1 further comprising:

detecting a third most prominent pitch associated with the audio signal; and

in the event the third most prominent pitch is associated with voice, modifying in the audio signal a portion that is associated with the third most prominent pitch.

55. The method of claim 1, wherein the audio signal comprises duophonic or polyphonic pitches.

56. A system for modifying an audio signal, comprising:

an input connection configured to receive the audio signal; and

a processor configured to;

detect a first most prominent pitch associated with the audio signal, including by;

transforming the audio signal into a time frequency domain to generate short time frequency spectra for the audio signal;

obtaining a plurality of pitch candidate frequency domain combs associated with a plurality of pitch candidates;

performing a cross-correlation in the frequency domain using information associated with the short time frequency spectra and the plurality of pitch candidate frequency domain combs in generating cross-correlation values; and

identifying the pitch candidate associated with a first maximum value among the cross-correlation values as the first most prominent pitch;

detect a second most prominent pitch associated with the audio signal by removing cross-correlation values associated with the first most prominent pitch from consideration and identifying the pitch candidate associated with a second maximum value among the remaining cross-correlation values as the second most prominent pitch; and

in the event the second most prominent pitch is associated with voice, modify in the audio signal a portion that is associated with the second most prominent pitch.

57. The system of claim 56, wherein:

the audio signal is a primary audio signal;

the input connection is further configured to receive a secondary audio signal; and

the processor is further configured to;

monitor the level of the secondary audio signal;

enable modification of the primary audio signal if the level of the secondary audio signal rises above a first prescribed threshold at a time when the primary audio signal is not being modified; and

disable modification of the primary audio signal if the level of the secondary audio signal drops below a second prescribed threshold at a time when the primary audio signal is being modified.

58. A system for modifying an audio signal, comprising:

means for detecting a first most prominent pitch associated with the audio signal, including;

transforming the audio signal into a time frequency domain to generate short time frequency spectra for the audio signal;

obtaining a plurality of pitch candidate frequency domain combs associated with a plurality of pitch candidates;

performing a cross-correlation in the frequency domain using information associated with the short time frequency spectra and the plurality of pitch candidate frequency domain combs in generating cross-correlation values; and

identifying the pitch candidate associated with a first maximum value among the cross-correlation values as the first most prominent pitch;

means for detecting a second most prominent pitch associated with the audio signal that includes means for removing cross-correlation values associated with the first most prominent pitch from consideration and identifying the pitch candidate associated with a second maximum value among the remaining cross-correlation values as the second most prominent pitch; and

means for modifying in the audio signal a portion that is associated with the second most prominent pitch in the event the second most prominent pitch is associated with voice.

59. The system of claim 58, wherein:

the audio signal is a primary audio signal;

the system further includes means for monitoring the level of a secondary audio signal;

the system further includes means for enabling modification of the primary audio signal if the level of the secondary audio signal rises above a first prescribed threshold at a time when the primary audio signal is not being modified; and

the system further includes means for disabling modification of the primary audio signal if the level of the secondary audio signal drops below a second prescribed threshold at a time when the primary audio signal is being modified.

60. A computer program product for modifying an audio signal, the computer program product being embodied in a non-transitory computer readable medium and comprising computer instructions for:

detecting a first most prominent pitch associated with the audio signal including;

transforming the audio signal into a time frequency domain to generate short time frequency spectra for the audio signal;

obtaining a plurality of pitch candidate frequency domain combs associated with a plurality of pitch candidates;

performing a cross-correlation in the frequency domain using information associated with the short time frequency spectra and the plurality of pitch candidate frequency domain combs in generating cross-correlation values; and

identifying the pitch candidate associated with a first maximum value among the cross-correlation values as the first most prominent pitch;

detecting a second most prominent pitch associated with the audio signal by removing cross-correlation values associated with the first most prominent pitch from consideration and identifying the pitch candidate associated with a second maximum value among the remaining cross-correlation values as the second most prominent pitch; and

in the event the second most prominent pitch is associated with voice, modifying in the audio signal a portion that is associated with the second most prominent pitch.

61. The computer program product of claim 60, wherein:

the audio signal is a primary audio signal;

the computer program product includes further computer instructions for monitoring the level of a secondary audio signal;

the computer program product includes further computer instructions for enabling modification of the primary audio signal if the level of the secondary audio signal rises above a first prescribed threshold at a time when the primary audio signal is not being modified; and

the computer program product includes further computer instructions for disabling modification of the primary audio signal if the level of the secondary audio signal drops below a second prescribed threshold at a time when the primary audio signal is being modified.