Automatic music mood detection

• US 7,115,808 B2
• Filed: 11/02/2005
• Issued: 10/03/2006
• Est. Priority Date: 03/25/2004
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Hong-Jiang, Lu, Lie
• Primary Examiner(s): DONELS, JEFFREY
• Application Number:
  US11/265,685
• Publication Number:
  US 20060054007A1
• Time in Patent Office: 335 Days
• Field of Search: 846/11, 846/22, 700/94
• US Class Current: 84/611
• CPC Class Codes: G10H 1/00 Details of electrophonic musical instruments
• CPC Class Codes: G10H 2210/071 for rhythm pattern analysis or rhythm style recognition rhythm pattern per se G10H2210/341
• CPC Class Codes: G10H 2240/085 Mood, i.e. generation, detection or selection of a particular emotional content or atmosphere in a musical piece

Abstract

A system and methods use music features extracted from music to detect a music mood within a hierarchical mood detection framework. A two-dimensional mood model divides music into four moods which include contentment, depression, exuberance, and anxious/frantic. A mood detection algorithm uses a hierarchical mood detection framework to determine which of the four moods is associated with a music clip based on the extracted features. In a first tier of the hierarchical detection process, the algorithm determines one of two mood groups to which the music clip belongs. In a second tier of the hierarchical detection process, the algorithm then determines which mood from within the selected mood group is the appropriate, exact mood for the music clip. Benefits of the mood detection system include automatic detection of music mood which can be used as music metadata to manage music through music representation and classification.

Claims

1. A computer readable medium containing instructions that when executed by a computing device perform actions, the instructions including:

instructions for extracting an intensity feature, a timbre feature, and a rhythm feature from a music clip;

instructions for classifying the music clip into a mood group based on the intensity feature; and

instructions for classifying the music clip into an exact music mood from the mood group based on the timbre feature and the rhythm feature.

2. The computer readable medium as recited in claim 1, wherein the instructions for extracting comprise:

instructions for converting the music clip into a uniform music clip having a uniform format;

instructions for dividing the uniform music clip into a plurality of frames; and

instructions for dividing each frame into a plurality of octave-based frequency sub-bands.

3. The computer readable medium as recited in claim 2, wherein the instructions for extracting an intensity feature comprise:

instructions for calculating a root mean-square (RMS) signal amplitude for each sub-band of each frame;

instructions for summing the RMS signal amplitudes across the sub-bands of each frame to determine a frame intensity for each frame; and

instructions for averaging the frame intensities to determine the intensity feature for the music clip.

4. The computer readable medium as recited in claim 2, wherein the instructions for extracting a timbre feature comprise:

instructions for calculating spectral shape features for each frame;

instructions for calculating spectral contrast features for each frame; and

instructions for representing the timbre feature with one or more of the spectral shape features and/or the spectral contrast features.

5. The computer readable medium as recited in claim 2, wherein the instructions for extracting a rhythm feature comprise:

instructions for extracting an amplitude envelope from the lowest sub-band and the highest sub-band of each frame across the uniform music clip;

instructions for estimating a difference curve of the amplitude envelope; and

instructions for detecting peaks above a threshold within the difference curve, the peaks being instrumental onsets.

6. The computer readable medium as recited in claim 5, wherein the instructions for extracting a rhythm feature further comprise:

instructions for extracting an average rhythm strength of the instrumental onsets;

instructions for extracting a rhythm regularity value based on the average of the maximum three peaks in the difference curve; and

instructions for extracting a rhythm tempo based on a common divisor of peaks in the difference curve.

7. The computer readable medium as recited in claim 1, wherein the instructions for classifying the music clip into a mood group comprise:

instructions for determining the probability of a first mood group based on the intensity feature;

instructions for determining the probability of a second mood group based on the intensity feature;

instructions for selecting the first mood group if the probability of the first mood group is greater than or equal to the probability of the second mood group; and

instructions for otherwise selecting the second mood group.

8. The computer readable medium as recited in claim 1, wherein the instructions for classifying the music clip into a mood group comprise instructions for classifying the music clip into one of:

a contentment and depression mood group;

or

an exuberance and anxiousness mood group.

9. The computer readable medium as recited in claim 1, wherein the mood group includes a first mood and a second mood, and the instructions for classifying the music clip into an exact music mood comprise:

instructions for determining the probability of the first mood based on the timbre feature and the rhythm feature;

instructions for determining the probability of the second mood based on the timbre feature and the rhythm feature;

instructions for selecting the first mood as the exact mood if the probability of the first mood is greater than or equal to the probability of the second mood; and

instructions for otherwise selecting the second mood as the exact mood.

10. The computer readable medium as recited in claim 9, wherein the mood group is selected from the group comprising:

a first mood group that includes a contentment mood and a depression mood; and

a second mood group that includes an exuberance mood and an anxiousness mood.

11. A system, comprising:

a music clip;

a mood detection algorithm configured to classify the music clip as a music mood according to music features extracted from the music clip;

a music feature extraction tool configured to extract the music features; and

a hierarchical music mood detection process configured to determine a mood group based on a first music feature and an exact music mood from within the mood group based on a second and third music feature.

12. The system as recited in claim 11, wherein the music feature extraction tool:

converts the music clip into a uniform music clip having a uniform format;

divides the uniform music clip into a plurality of frames; and

divides each frame into a plurality of octave-based frequency sub-bands.

13. The system as recited in claim 12, wherein the music feature extraction tool calculates a root mean-square (RMS) signal amplitude for each sub-band of each frame;

sums the RMS signal amplitudes across the sub-bands of each frame to determine a frame intensity for each frame; and

averages the frame intensities to determine the intensity feature for the music clip.

14. The system as recited in claim 12, wherein the music feature extraction tool extracts a timbre feature by:

calculating spectral shape features for each frame;

calculating spectral contrast features for each frame; and

representing the timbre feature with one or more of the spectral shape features and/or the spectral contrast features.

15. The system as recited in claim 12, wherein the music feature extraction tool extracts a rhythm feature by:

extracting an amplitude envelope from the lowest sub-band and the highest sub-band of each frame across the uniform music clip;

estimating a difference curve of the amplitude envelope; and

detecting peaks above a threshold within the difference curve, the peaks being instrumental onsets.

16. The system as recited in claim 15, wherein the music feature extraction tool extracts the rhythm feature further:

extracts an average rhythm strength of the instrumental onsets;

extracts a rhythm regularity value based on the average of the maximum three peaks in the difference curve; and

extracts a rhythm tempo based on a common divisor of peaks in the difference curve.

17. The system as recited in claim 11, wherein the mood detection algorithm:

determines the probability of a first mood group based on the intensity feature;

determines the probability of a second mood group based on the intensity feature;

selects the first mood group if the probability of the first mood group is greater than or equal to the probability of the second mood group; and

otherwise selects the second mood group.

18. The system as recited in claim 11, wherein the mood detection algorithm classifies the music clip into one of:

a contentment and depression mood group;

or,

an exuberance and anxiousness mood group.

19. The system as recited in claim 11, wherein the mood group includes a first mood and a second mood, and the mood detection algorithm classifies the music clip into an exact music mood by:

determining the probability of the first mood based on the timbre feature and the rhythm feature;

determining the probability of the second mood based on the timbre feature and the rhythm feature;

selecting the first mood as the exact mood if the probability of the first mood is greater than or equal to the probability of the second mood; and

otherwise selecting the second mood as the exact mood.

20. The system as recited in claim 19, wherein the mood group is selected from the group comprising:

a first mood group that includes a contentment mood and a depression mood; and

a second mood group that includes an exuberance mood and an anxiousness mood.