Audio signal processing methods and systems

• US 9,570,057 B2
• Filed: 07/20/2015
• Issued: 02/14/2017
• Est. Priority Date: 07/21/2014
• Status: Active Grant

• Current Assignee: Matthew Brown
• Original Assignee: Matthew Brown
• Inventors: Brown, Matthew
• Primary Examiner(s): Tran, Thang
• Application Number:
  US14/804,042
• Publication Number:
  US 20160019878A1
• Time in Patent Office: 575 Days
• US Class Current: 1/1
• CPC Class Codes: G10H 1/125 using a digital filter
• CPC Class Codes: G10H 1/383 Chord detection and/or recognition, e.g. for correction, or automatic bass generation
• CPC Class Codes: G10H 2210/041 based on mfcc [mel -frequency spectral coefficients]
• CPC Class Codes: G10H 2210/066 for pitch analysis as part of wider processing for musical purposes, e.g. transcription, musical performance evaluation; Pitch recognition, e.g. in polyphonic sounds; Estimation or use of missing fundamental
• CPC Class Codes: G10H 2210/081 for automatic key or tonality recognition, e.g. using musical rules or a knowledge base
• CPC Class Codes: G10H 2210/086 for transcription of raw audio or music data to a displayed or printed staff representation or to displayable MIDI-like note-oriented data, e.g. in pianoroll format
• CPC Class Codes: G10H 2250/215 Transforms, i.e. mathematical transforms into domains appropriate for musical signal processing, coding or compression
• CPC Class Codes: G10H 2250/225 MDCT [Modified discrete cosine transform], i.e. based on a DCT of overlapping data
• CPC Class Codes: G10H 2250/235 Fourier transform; Discrete Fourier Transform [DFT]; Fast Fourier Transform [FFT]
• CPC Class Codes: G10H 2250/251 Wavelet transform, i.e. transform with both frequency and temporal resolution, e.g. for compression of percussion sounds; Discrete Wavelet Transform [DWT]
• CPC Class Codes: G10H 2250/285 Hann or Hanning window
• CPC Class Codes: G10L 25/18 the extracted parameters being spectral information of each sub-band
• CPC Class Codes: G10L 25/90 Pitch determination of speech signals
• CPC Class Codes: H04R 3/04 for correcting frequency response

Abstract

Described are methods and systems of identifying one or more fundamental frequency component(s) of an audio signal. The methods and systems may include any one or more of an audio event receiving step, a signal discretization step, a masking step, and/or a transcription step.

Claims

1. A method of identifying at least one fundamental frequency component of an audio signal, the method comprising receiving and recording an audio event, and converting the recorded audio event into an audio signal, the method further comprising:

(a) filtering the audio signal to produce a plurality of sub-band time domain signals;

(b) transforming the plurality of sub-band time domain signals into a plurality of sub-band frequency domain signals by mathematical operators;

(c) summing together the plurality of sub-band frequency domain signals to yield a single spectrum;

(d) calculating the bispectrum of each of the plurality of sub-band time domain signals;

(e) summing together the bispectra calculated in (d);

(f) calculating the diagonal of the summed bispectra;

(g) multiplying the single spectrum and the diagonal of the summed bispectra to produce a product spectrum; and

(h) identifying at least one fundamental frequency component of the audio signal from the product spectrum or information contained in the product spectrum.

2. The method according to claim 1, wherein at least one identifiable fundamental frequency component is matched with a known audio event such that identification of the at least one fundamental frequency component enables identification of the known audio event.

3. The method according to claim 1, wherein the method further comprises visually representing on a screen or other display means at least one selected from the group consisting of:

the product spectrum;

information contained in the product spectrum;

identifiable fundamental frequency components; and

a representation of identifiable known audio events in the audio signal.

4. The method according to claim 1, wherein the product spectrum includes a plurality of peaks, and wherein at least one fundamental frequency component of the audio signal is identifiable from the locations of the peaks in the product spectrum.

5. The method according to claim 1, wherein filtering of the audio signal is carried out using a constant-Q filterbank applying a constant ratio of frequency to bandwidth across frequencies of the audio signal.

6. The method according to claim 5, wherein the filterbank comprises a plurality of spectrum analyzers and a plurality of filter and decimate blocks.

7. The method according to claim 1, wherein the mathematical operators for transforming the plurality of sub-band time domain signals into the plurality of sub-band frequency domain signals comprise fast Fourier transforms.

8. The method according to claim 1, wherein the audio signal comprises a plurality of audio signal segments, and wherein fundamental frequency components of the audio signal are identifiable from product spectra produced by the operation of steps (a) to (g) on the audio signal segments, or from the information contained in such product spectra for the audio signal segments.

9. The method according to claim 1, wherein the audio event comprises a plurality of audio event segments, each being converted into a plurality of audio signal segments, wherein fundamental frequency components of the audio event are identifiable from product spectra produced by operation of steps (a) to (g) of claim 1 on the audio signal segments, or from the information contained in such product spectra for the audio signal segments.

10. The method according to claim 1, wherein the method includes a signal discretization step, and wherein the signal discretization step enables discretizing the audio signal into time-based segments of varying sizes.

11. The method according to claim 10, wherein the segment size of the time-based segment is determinable by the energy characteristics of the audio signal.

12. The method according to claim 1, wherein the method includes a masking step, and wherein the masking step comprises applying a quantizing algorithm to map the frequency spectra of the product spectrum produced in step (g), and a mask bank consisting of a plurality of masks to be applied to the mapped frequency spectra.

13. The method according to claim 12, wherein the quantizing algorithm effects mapping the frequency spectra of the product spectrum to a series of audio event-specific frequency ranges, the mapped frequency spectra together constituting an array.

14. The method according to claim 12, wherein at least one mask in the mask bank contains fundamental frequency spectra associated with at least one known audio event.

15. The method according to claim 14, wherein the fundamental frequency spectra of a plurality of masks in the mask bank is set in accordance with the identified fundamental frequency component.

16. The method according to claim 13, wherein the mask bank operates by applying at least one mask to the array such that the frequency spectra of the at least one mask is subtracted from the frequency spectra in the array, in an iterative fashion from the lowest applicable fundamental frequency spectra mark to the highest applicable fundamental frequency spectra mark, until there is no frequency spectra left in the array below a minimum signal amplitude threshold.

17. The method according to claim 13, wherein the particular masks of the mask bank to be applied to the array are chosen based on which at least one fundamental frequency component(s) are identifiable in the product spectrum of the audio signal.

18. The method according to claim 13, further comprising iterative application of the masking step, wherein iterative application of the masking step comprises performing cross-correlation between the diagonal of the summed bispectra and masks in the mask bank, then selecting the mask having the highest cross-correlation value, the high correlation mask is then subtracted from the array, and this process continues iteratively until no frequency content below a minimum threshold remains in the array.

19. The method according to claim 14, wherein the masking step comprises producing a final array identifying each of the at least one known audio event present in the audio signal, wherein the at least one known audio event identifiable in the final array is determinable by observing which of the masks in the masking step are applied.

20. The method according to claim 19, wherein the method includes a transcription step, and wherein the transcription step comprises converting known audio events, identifiable by the masking step or by the product spectrum, into a visually representable transcription of the identified known audio events.

21. A non-transitory computer-readable medium for identifying at least one fundamental frequency component of an audio signal or audio event, the non-transitory computer-readable medium comprising:

code components configured to enable a computer to perform a method of identifying at least one fundamental frequency component of an audio signal, the method comprising receiving and recording an audio event, and converting the recorded audio event into an audio signal, the method further comprising;

(a) filtering the audio signal to produce a plurality of sub-band time domain signals;

(b) transforming the plurality of sub-band time domain signals into a plurality of sub-band frequency domain signals by mathematical operators;

(c) summing together the plurality of sub-band frequency domain signals to yield a single spectrum;

(d) calculating the bispectrum of each of the plurality of sub-band time domain signals;

(e) summing together the bispectra calculated in (d);

(f) calculating the diagonal the summed bispectra;

(g) multiplying the single spectrum and the diagonal of the summed bispectra to produce a product spectrum; and

(h) identifying at least one fundamental frequency component of the audio signal from the product spectrum or information contained in the product spectrum.