Music similarity function based on signal analysis

• US 7,031,980 B2
• Filed: 10/31/2001
• Issued: 04/18/2006
• Est. Priority Date: 11/02/2000
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Logan, Beth T., Salomon, Ariel
• Primary Examiner(s): Gaffin, Jeffrey A.
• Assistant Examiner(s): Mahmoudi, Hassan
• Application Number:
  US10/004,157
• Publication Number:
  US 20020181711A1
• Time in Patent Office: 1,630 Days
• Field of Search: 707/1, 707/3, 707/5, 707/7, 707/10, 707/104.1, 846/09, 704/223, 704/243, 360/13, 348/722, 348/180, 324/536
• US Class Current: 1/1
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04S 1/007 in which the audio signals are in digital form data reduction aspects thereof based on psychoacoustics G10L19/02
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

The present invention computer method and apparatus determines music similarity by generating a K-means (instead of Gaussian) cluster signature and a beat signature for each piece of music. The beat of the music is included in the subsequent distance measurement.

Claims

1. A method of processing a database of musical works, comprising:

obtaining a digitized audio file for each musical work;

obtaining a spectral signature based on spectral features for each audio file;

obtaining a rhythmic beat representation for each audio file;

for each musical work, summing the spectral signature and the rhythmic beat representation; and

determining a similarity of the plurality of musical works based on the summed results.

2. The method of claim 1, further comprising weighting the summed results.

3. The method of claim 1, further comprising visually displaying on a display device the plurality of musical works including indicating the determined similarities.

4. A computer program product for determining similarity between a plurality of musical works, the computer program product including a computer usable medium having computer readable code thereon, including program code which:

obtains respective digitized audio files of the plurality, and forms (i) a spectral signature based on spectral features from the corresponding audio file and (ii) a rhythmic beat representation from the corresponding audio file; and

for a given musical work of interest;

(a) compares its spectral signature to the spectral signatures of the musical works in the plurality;

(b) compares its rhythmic beat representation to the rhythmic beat representations of the musical works in the plurality; and

(c) sums, including respective weighting of results of the comparison in (a) and (b), the summed results providing an indication of which musical works in the plurality are similar to the given musical work of interest.

5. A computer data signal embodied in a carrier wave for determining similarity between a plurality of musical works, comprising:

program code for obtaining digitized audio files of the plurality, and for each musical work in the obtained digitized audio files, the program code forms (i) a spectral signature based on spectral features from the corresponding audio file and (ii) a rhythmic beat representation from the corresponding audio file; and

for a given musical work of interest, program code that;

(a) compares its spectral signature to the spectral signatures of the musical works in the plurality;

(b) compares its rhythmic beat representation to the rhythmic beat representations of the musical works in the plurality; and

(c) sums, including respective weighting of results of the comparison in (a) and (b), the summed results providing an indication of which musical works in the plurality are similar to the given musical work of interest.

6. A method for determining similarity between a plurality of musical works comprising the steps of:

obtaining respective digitized audio files of the plurality of musical works;

for each musical work in the plurality, forming (i) a spectral signature based on spectral features from the corresponding audio file and (ii) a rhythmic beat representation from the corresponding audio file;

for a given musical work of interest;

(a) comparing its spectral signature to the spectral signatures of the musical works in the plurality;

(b) comparing its rhythmic beat representation to the rhythmic beat representations of the musical works in the plurality; and

(c) summing, including respective weighting of results of the comparisons in (a) and (b), said summed results providing an indication of which musical works in the plurality are similar to the given musical work of interest.

7. The method of claim 6, wherein forming a spectral signature includes dividing the corresponding audio file into a plurality of frames.

8. The method of claim 7, further comprising converting each frame to a spectral representation to obtain a plurality of spectral representations for the audio file.

9. The method of claim 8, further comprising clustering the spectral representations of the audio file to obtain the spectral signature for the audio file.

10. The method of claim 9, further comprising comparing the spectral signatures of two different audio files using an Earth Mover'"'"'s Distance.

11. The method of claim 8, wherein the spectral representation includes a vector of Mel-frequency cepstral coefficients.

12. The method of claim 8, wherein each spectral representation includes a plurality of Mel-frequency cepstral coefficients.

13. The method of claim 7, further comprising performing a windowing function on each frame.

14. The method of claim 7, further comprising applying a Hamming window on each frame.

15. The method of claim 7, further comprising applying a pre-emphasis on each frame.

16. The method of claim 15, further comprising subjecting data from each frame to a Fast Fourier Transform function to obtain a frequency domain signal for each frame.

17. The method of claim 16, further comprising warping a log amplitude of each frequency signal to a Mel-frequency scale.

18. The method of claim 17, further comprising subjecting the warped frequency function to a second Fast Fourier Transform to obtain a parameter set of Mel-frequency cepstral coefficients.

19. The method of claim 17, further comprising subjecting the frequency domain signal for each frame to a set of triangular filters to obtain a plurality of Mel-frequency spaced components.

20. The method of claim 19, further comprising subjecting the Mel-frequency spaced components to a discrete cosine transform function to obtain a plurality of Mel-frequency cepstral coefficients.

21. The method of claim 6, wherein forming a rhythmic beat representation includes dividing the corresponding audio file into a plurality of frames.

22. The method of claim 21, further comprising converting each frame to a spectral representation to obtain a plurality of spectral representations for the audio file.

23. The method of claim 22, wherein each spectral representation includes a plurality of Mel-frequency cepstral coefficients.

24. The method of claim 22, further comprising computing a similarity matrix for the audio file.

25. The method of claim 24, further comprising computing a beat spectrogram for the audio file.

26. The method of claim 25, further comprising constructing a histogram of the beat spectrogram.

27. The method of claim 26, further comprising normalizing the histogram to account for the total number of frames of the audio file.

28. The method of claim 27, further comprising calculating a distance between a pair of histograms.

29. The method of claim 28, wherein calculating the distance includes calculating the closest distance between the pair of histograms.

30. The method of claim 29, wherein the closest distance is the minimum of the sum of absolute differences between bins of each histogram calculated over a range of scalings of each histogram.

31. The method of claim 30, further comprising applying a function to each histogram to weight certain bins.

32. The method of claim 31, further comprising scaling each histogram at least twice to allow for slight differences between musical works.

33. The method of claim 32, wherein for each scale factor, one histogram is resampled by a factor and compared to the unscaled histogram.

34. The method of claim 6, further comprising generating a set of similar musical works for the given musical work of interest.

35. The method of claim 34, further comprising visually displaying on a display device, the musical works in a manner illustrating relative similarities or dissimilarities of the musical works.

36. The method of claim 6, further comprising calculating a relative distance between each pair of musical works.

37. The method of claim 36, further comprising constructing a matrix of song similarity based on the relative distance.

38. The method of claim 37, further comprising performing a Multi-dimensional scaling on the matrix to obtain coordinates in K-dimensional space for each musical work, one coordinate per song.

39. The method of claim 38, further comprising plotting the coordinates.

40. A method for determining similarity between a plurality of musical works comprising:

obtaining respective digitized audio files of the plurality of musical works;

for each musical work, forming at least two different representations from the corresponding audio file, the different representations representing respective different aspects of the musical work, one of the representations being a spectral signature based on spectral features in the audio file;

for a given musical work of interest;

(a) comparing one of its two different representations to respective ones of the two different representations of the musical works in the plurality;

(b) comparing the other of the two different representations of the given musical work to respective other ones of the two different representations of the musical works in the plurality; and

(c) summing results of the comparisons in (a) and (b), said summed results providing a quantitative indication of which musical works in the plurality are similar to the given musical work of interest.

41. The method of claim 40, wherein the step of forming at least two different representations includes forming a beat representation for each musical work, the spectral signature representing instrumentation of the musical work and the beat representation representing rhythmic frequencies of the musical work.

42. The method of claim 40, further comprising the step of preprocessing the audio files before forming the different representations for each musical work.

43. The method of claim 42, wherein the step of preprocessing includes omitting relatively long pauses.

44. The method of claim 40, further comprising providing a respective reliability measure associated with each representation.

45. The method of claim 40, wherein the step of summing includes weighting results of the comparisons as a function of reliability measures of the representations compared.

46. The method of claim 40, further comprising visually displaying the plurality of musical works on a display device, in a manner illustrating relative similarities and dissimilarities among the plurality.

47. A computer system comprising:

a processor;

a memory system connected to the processor; and

a computer program, in the memory, which determines similarity between a plurality of musical works by;

obtaining respective digitized audio files of the plurality of musical works;

for each musical work, forming (i) a spectral signature based on spectral features from the corresponding audio file and (ii) a rhythmic beat representation from the corresponding audio file;

for a given musical work of interest;

(a) comparing its spectral signature to the spectral signatures of the musical works in the plurality;

(b) comparing its rhythmic beat representation to the rhythmic beat representations of the musical works in the plurality; and

(c) summing, including respective weighting of results of the comparisons in (a) and (b), said summed results providing an indication of which musical works in the plurality are similar to the given musical work of interest.

48. A system for determining similarity between a plurality of musical works, the system comprising:

means for obtaining respective digitized files of the plurality of musical works;

for each musical work, means for forming (i) a spectral signature based on spectral features from the corresponding audio file and (ii) a rhythmic beat representation from the corresponding audio file;

for a given musical work of interest;

(a) means for comparing its spectral signature based on spectral features to the spectral signatures of the musical works in the plurality;

(b) means for comparing its rhythmic beat representation to the rhythmic beat representations of the musical works in the plurality; and

(c) means for summing, including respective weighting of results of the comparisons in (a) and (b), the summed results providing an indication of which musical works in the plurality are similar to the given musical work of interest.