Method and apparatus for synchronizing audio and video components of multimedia presentations by identifying beats in a music signal

• US 7,301,092 B1
• Filed: 04/01/2004
• Issued: 11/27/2007
• Est. Priority Date: 04/01/2004
• Status: Active Grant

• Current Assignee: Corel Corporation (Corel Holdings LP)
• Original Assignee: Pinnacle Systems Incorporated (Avid Technology Incorporated)
• Inventors: McNally, Guy W. W., Zamara, Christopher J., Tung, Charmine S.
• Primary Examiner(s): Donels; Jeffrey
• Application Number:
  US10/816,737
• Time in Patent Office: 1,335 Days
• Field of Search: 846/12, 846/36
• US Class Current: 84/612
• CPC Class Codes: G10H 1/368 displaying animated or moving pictures synchronized with the music or audio part
• CPC Class Codes: G10H 1/40 Rhythm
• CPC Class Codes: G10H 2210/076 for extraction of timing, tempo; Beat detection

Abstract

Methods and systems for detecting a fundamental beat frequency in a predetermined time interval of a music signal are disclosed. The frequency may be detected by processing a music signal with the discrete wavelet transform to obtain a set of coefficients. A subset of the coefficients may be processed to obtain a plurality of candidate beat frequencies contained in the corresponding portion of the music signal. Harmonic relationships between the candidate beat frequencies may be determined, and the fundamental beat frequency may then be determined based upon the determined harmonic relationships.

Claims

1. A computer-implemented method of detecting a fundamental beat frequency in a predetermined time interval of a music signal, comprising:

a) processing a music signal with the discrete wavelet transform to obtain a set of coefficients;

b) processing a subset of the coefficients to obtain a plurality of candidate beat frequencies contained in the corresponding portion of the music signal;

c) determining the harmonic relationships between the candidate beat frequencies;

d) determining the fundamental beat frequency based upon the determined harmonic relationships; and

e) storing information about the determined fundamental beat frequency in a memory for use in production of a multimedia composition.

2. The computer-implemented method of claim 1, wherein determining the fundamental beat frequency comprises selecting one of the candidate beat frequencies having a non-ambiguous harmonic structure.

3. The computer-implemented method of claim 1, wherein determining harmonic relationships comprises determining integer relationships between the candidate beat frequencies.

4. The computer-implemented method of claim 1, wherein:

the candidate beat frequencies each comprise a range of frequencies;

processing a subset of the coefficients comprises calculating autocorrelation values; and

determining the fundamental beat frequency comprises;

identifying the candidate beat frequency having a non-ambiguous harmonic structure and the strongest relative amplitude value calculated to model human auditory perception;

determining the harmonic relationship between the candidate beat frequency having a non-ambiguous harmonic structure and the strongest relative amplitude value calculated to model human auditory perception, and the lowest candidate frequency having a non-ambiguous harmonic structure; and

selecting the fundamental beat frequency as the frequency range of the lowest candidate beat frequency having a non-ambiguous harmonic structure multiplied by the harmonic relationship.

5. The computer-implemented method of claim 1, wherein

processing a subset of the coefficients to obtain a plurality of candidate beat frequencies comprises calculating autocorrelation values of a subset of the coefficients;

and

determining the fundamental beat frequency comprises determining the fundamental beat frequency based upon the determined harmonic relationships and the relative amplitude values calculated to model human auditory perception.

6. The computer-implemented method of claim 1, wherein processing a subset of the coefficients to obtain a plurality of beat frequencies comprises creating a buffer of a predetermined number of coefficients.

7. The computer-implemented method of claim 6, wherein processing a subset of the coefficients comprises creating a dynamic and weighted histogram of beat frequencies.

8. The computer-implemented method of claim 7, wherein creating a dynamic and weighted histogram comprises consolidating values in adjacent bins of the histogram.

9. The computer-implemented method of claim 8, wherein consolidating values in adjacent bins of the histogram comprises using a mathematical window function.

10. An apparatus for analyzing the beat of a music signal comprising:

a fundamental beat frequency identifier generating a fundamental beat frequency signal from the music signal;

a time domain envelope analyzer comprising a peak generator generating a peak signal from the music signal, the peak signal comprising amplitude and time values of amplitude peaks of the music signal;

a comparator and beat identifier, coupled to the fundamental beat frequency identifier and the time domain envelope analyzer, and generating, from the peak signal and fundamental beat frequency signal, a series of time values identifying the amplitude peaks corresponding to onset times of beats within periods based on the fundamental beat frequency signal; and

a memory for storing information about the determined fundamental beat frequency for use in production of a multimedia composition.

15. The apparatus of claim 10, further comprising a preprocessor to convert an input music signal having a number of channels and a sampling rate to a music signal having one channel and a 22.05 kHz sampling rate.

16. The apparatus of claim 10, further comprising a time stamp adjuster to set the time values of the series of time values generated by the comparator and beat identifier to the time difference between the beat onset and the start of the music signal.

11. A computer-implemented method of detecting the localized fundamental beat frequency of a digital music signal comprising:

detecting time period peaks above a threshold in at least two, consecutive, predetermined-sized buffers of an autocorrelation function of a decomposition of a digital music signal using a discrete wavelet transform;

determining which one or more of the detected time period peaks is heard most often in the at least two, consecutive, predetermined-sized buffers, thereby creating a set of “

often-heard”

beat frequencies in a localized portion of the digital music signal and wherein one or more beat frequencies in the set has a magnitude representing how often it is heard;

determining the harmonic structure between each beat frequency in the set and the remaining beat frequencies in the set;

selecting one of the “

often heard”

beat frequencies as the localized fundamental beat frequency, wherein the criteria for selection comprise the greatest magnitude and a non-ambiguous harmonic structure; and

storing information about the determined fundamental beat frequency in a memory for use in production of a multimedia composition.

12. The computer-implemented method of claim 11, wherein detecting time period peaks above a threshold comprises:

half-wave rectifying the autocorrelation values of the at least two consecutive predetermined-size buffers of the autocorrelation function;

identifying time period peaks based on the rectified autocorrelation values; and

comparing the rectified autocorrelation values of the identified time period peaks to a threshold, thereby detecting time period peaks above a threshold.

13. The computer-implemented method of claim 12, wherein identifying time period peaks based on the rectified autocorrelation values comprises:

determining the maximum rectified autocorrelation value and an average noise value of the rectified autocorrelation values;

indicating the start of a peak as a time period whose rectified autocorrelation value is greater than the previous autocorrelation value and greater than the average noise value of the rectified autocorrelation values; and

identifying the time period corresponding to the turnover point after the start of a peak as a time period peak; and

wherein the threshold equals a predetermined percentage of the maximum rectified autocorrelation value.

14. The computer-implemented method of claim 11, wherein determining which one or more of the detected time period peaks is heard most often comprises:

providing a histogram bin for each frequency corresponding to a time period in the autocorrelation function;

creating a dynamic and weighted histogram of integrated autocorrelation values of detected time period peaks in two or more consecutive buffers of the at least two consecutive, predetermined-sized buffers, wherein creating the dynamic and weighted histogram comprises;

integrating the autocorrelation values of detected time period peaks in the two or more consecutive buffers by multiplying them by a predetermined integration value;

increasing the corresponding histogram bin'"'"'s value by the integrated autocorrelation value of a detected time period peak; and

decreasing the corresponding histogram bin'"'"'s value by the predetermined integration value to a minimum of zero if the time period of the auto correlation function is not a detected time period peak,

thereby creating a dynamic and weighted histogram; and

picking the one or more frequencies corresponding to the histogram bins with peak values as the set of “

often heard”

beat frequencies in the localized portion of the music signal, wherein each frequency in the set has a magnitude represented by the histogram bin value.

17. A computer-implemented method of identifying beats in a music signal that correspond to a fundamental beat frequency comprising:

determining a fundamental beat frequency in a music signal using a discrete wavelet transform;

obtaining an envelope signal of the music signal, wherein the envelope signal contains amplitude peaks of the music signal that represent beats in the music signal;

identifying one or more peaks in the envelope signal as beats in a music signal that correspond to a fundamental beat frequency; and

storing information about the determined fundamental beat frequency in a memory for use in production of a multimedia composition.