System and methods for providing automatic classification of media entities according to sonic properties

• US 20030045953A1
• Filed: 08/21/2001
• Published: 03/06/2003
• Est. Priority Date: 08/21/2001
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Weare, Christopher B.
• Granted Patent: US 7,532,943 B2
• Time in Patent Office: Days
• US Class Current: 700/94
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G10H 1/0041 in coded form
• CPC Class Codes: G10H 2240/081 Genre classification, i.e. descriptive metadata for classification or selection of musical pieces according to style analysis genre G10H2210/036
• CPC Class Codes: G10H 2240/085 Mood, i.e. generation, detection or selection of a particular emotional content or atmosphere in a musical piece
• CPC Class Codes: G10H 2240/135 Library retrieval index, i.e. using an indexing scheme to efficiently retrieve a music piece
• CPC Class Codes: G10H 2250/235 Fourier transform; Discrete Fourier Transform [DFT]; Fast Fourier Transform [FFT]
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence

Abstract

In connection with a classification system for classifying media entities that merges perceptual classification techniques and digital signal processing classification techniques for improved classification of media entities, a system and methods are provided for automatically classifying and characterizing sonic properties of media entities. Such a system and methods may be useful for the indexing of a database or other storage collection of media entities, such as media entities that are audio files, or have portions that are audio files. The methods also help to determine media entities that have similar sonic properties by utilizing classification chain techniques that test distances between media entities in terms of their properties. For example, a neighborhood of songs may be determined within which each song has similar sonic properties.

Claims

1. A method for automatically classifying spectral properties of audio data, comprising:

applying input audio data (1) to a critical band filtering process to form first output data and (2) to an entropy calculation process to form second output data;

applying the first output data to a first derivative process to form third output data; and

inputting said first, second and third output data to an averaging process to form a spectral feature vector representing the input audio data.

2. A method according to claim 1, wherein the audio data is divided into frames, and the method is performed frame by frame.

3. A method according to claim 1, further including calculating root mean squared values of the input audio data.

4. A method according to claim 2, wherein said entropy calculation process includes calculating:

5. A method according to claim 2, wherein said first output data output from said critical band filtering process includes, for each critical band located due to the critical band filtering process, data resulting from summing the value of the square of the magnitude of each bin of data in the frame.

6. A method according to claim 2, wherein said first derivative process that forms said output data includes, for each critical band located due to the critical band filtering process, calculating the first derivative of data for each bin of data in the frame to gain information about the rate of change of the spectral frequencies represented by the frame.

7. The method of claim 1, further comprising converting the input audio data from the time domain to the frequency domain.

8. A method according to claim 7, wherein said converting of the input audio data signal from the time domain to the frequency domain includes performing a fast fourier transform on the audio data.

9. A method according to claim 2, wherein the averaging process is an averaging process over all of the frames to form the spectral feature vector for the audio data.

10. A method according to claim 1, further comprising applying the output vector to a classification process which determines at least one of (1) at least one spectral properties value and (2) at least one spectral properties class that describes the audio data.

11. A method according to claim 1, wherein the spectral feature vector for the audio data is a vector that is 1×

25.

12. A method according to claim 1, wherein the audio data is formatted according to pulse code modulated format.

13. A method according to claim 12, wherein the audio data is previously in a format other than pulse code modulated format, and the method further comprises converting the audio data to pulse code modulated format from the other format.

14. The method of claim 1, further comprising converting the input audio data from the time domain to the frequency domain.

15. A method according to claim 12, wherein said converting of the input audio data signal from the time domain to the frequency domain includes performing a fast fourier transform on the audio data.

16. A method according to claim 2, wherein the averaging process is an averaging process over all of the frames to form the spectral feature vector for the audio data.

17. A method according to claim 1, further comprising performing a principal component analysis process on the spectral feature vector.

18. A computer readable medium bearing computer executable instructions for carrying out the method of claim 1.

19. A modulated data signal carrying computer executable instructions for performing the method of claim 1.

20. At least one computing device comprising means for performing the method of claim 1.

21. A method of classifying data according to spectral properties of the data, comprising:

assigning to each media entity of a plurality of media entities in a data set to at least one spectral properties class;

processing each media entity of said data set to extract at least one spectral properties characteristic based on digital signal processing of each media entity;

generating a plurality of spectral properties vectors for said plurality of media entities, wherein each spectral properties vector includes said at least one spectral properties class and at least one spectral properties characteristic based on digital signal processing; and

forming a classification chain based upon said plurality of feature vectors.

22. A method according to claim 21, further comprising:

processing an unclassified media entity to extract at least one spectral properties characteristic based on digital signal processing of the unclassified media entity;

generating a vector for the unclassified media entity including said at least one digital signal processing spectral properties characteristic;

presenting the vector for the unclassified media entity to the classification chain; and

classifying the unclassified entry with an estimate of the spectral properties class by calculating the representative spectral properties class of the subset of the plurality of vectors of the classification chain located in the neighborhood of the vector for the unclassified entity.

23. A method according to claim 22, further including calculating a neighborhood distance that defines a distance within which two vectors in the classification chain space are in the same neighborhood for purposes of being in the same spectral properties class.

24. A method according to claim 22, wherein said classifying of the unclassified entry includes classifying the unclassified entry with a median spectral properties class represented by the neighborhood.

25. A method according to claim 22, wherein said spectral properties class is described by a numerical value and said classifying of the unclassified entry includes classifying the unclassified entry with a mean of numerical spectral properties values found in the neighborhood.

26. A method according to claim 22, wherein said classifying includes returning at least one number indicating the level of confidence of the spectral properties class estimate.

27. A computer readable medium bearing computer executable instructions for carrying out the method of claim 21.

28. A modulated data signal carrying computer executable instructions for performing the method of claim 21.

29. At least one computing device comprising means for performing the method of claim 21.

30. A computing system, comprising:

a computing device including;

a classification chain data structure stored thereon having a plurality of classification vectors, wherein each vector includes data representative of a spectral properties class as classified by humans and spectral properties characteristics as determined by digital signal processing; and

processing means for comparing an unclassified media entity to the classification chain data structure to determine an estimate of the spectral properties class of the unclassified media entity.

31. A computing system according to claim 30, wherein said determining of an estimate of the spectral properties class includes returning at least one number indicating the level of confidence of the spectral properties class assignment.

32. A method according to claim 31, wherein the performance level of the classification chain improves over time due to the examination of unclassified media entities that have a low confidence level associated with the spectral properties class assignment.

33. A classification chain data structure utilized in connection with the classification of spectral properties of new unclassified media entities, comprising:

a plurality of classification vectors, wherein each vector includes;

spectral properties data as classified by humans; and

spectral properties data determined by digital signal processing techniques.

34. A method for classifying data according to its spectral properties, comprising:

a construction phase, comprising;

classifying by human experts a representative set of sounds according to their spectral perceptual qualities;

assigning each entry in the representative set at least one value;

reducing the results of the construction phase to a set of numbers called the characteristic vector of each sound; and

storing the characteristic vector in a classification chain for later calculations; and

a classification phase, comprising;

presenting an unclassified sound for classification;

calculating the characteristic vector of the unknown sound; and

presenting the characteristic vector to the classification chain, which returns an estimate of the spectral properties of the unknown sound.

35. The method of claim 34, wherein the reducing of the results of the construction phase to a set of numbers called the characteristic vector of each sound includes breaking up the sound into a plurality of frames of a fixed number of pulse code modulation values, each value representing a sample in the frame, determining the energy of the frame by calculating the root mean squared value of the frame, taking a fast fourier transform of the root mean squared value, and calculating the entropy content by normalizing the sum of the magnitudes of the bins of the fast fourier transform to unity for each frame and then calculating: