Automatic music mood detection

• US 7,396,990 B2
• Filed: 12/09/2005
• Issued: 07/08/2008
• Est. Priority Date: 12/09/2005
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Hong-Jiang, Lu, Lie
• Primary Examiner(s): Donels; Jeffrey
• Application Number:
  US11/275,100
• Publication Number:
  US 20070131096A1
• Time in Patent Office: 942 Days
• Field of Search: 846/00, 846/11, 846/22, 700/94
• US Class Current: 84/611
• CPC Class Codes: G10H 1/0008 Associated control or indicating means
• CPC Class Codes: G10H 2210/071 for rhythm pattern analysis or rhythm style recognition rhythm pattern per se G10H2210/341
• CPC Class Codes: G10H 2210/076 for extraction of timing, tempo; Beat detection tempo display G10H2220/081; tempo control G10H2210/375
• CPC Class Codes: G10H 2210/081 for automatic key or tonality recognition, e.g. using musical rules or a knowledge base
• CPC Class Codes: G10H 2240/061 MP3, i.e. MPEG-1 or MPEG-2 Audio Layer III, lossy audio compression
• CPC Class Codes: G10H 2240/081 Genre classification, i.e. descriptive metadata for classification or selection of musical pieces according to style analysis genre G10H2210/036
• CPC Class Codes: G10H 2240/085 Mood, i.e. generation, detection or selection of a particular emotional content or atmosphere in a musical piece
• CPC Class Codes: G10H 2240/091 Info, i.e. juxtaposition of unrelated auxiliary information or commercial messages with or between music files metadata G10H2240/075
• CPC Class Codes: G10H 2240/135 Library retrieval index, i.e. using an indexing scheme to efficiently retrieve a music piece
• CPC Class Codes: G10H 2240/155 Library update, i.e. making or modifying a musical database using musical parameters as indices data structures involving musical parameters G10H2240/005
• CPC Class Codes: G10H 2250/031 Spectrum envelope processing

Abstract

A system and methods use music features extracted from music to detect a music mood within a hierarchical mood detection framework. A two-dimensional mood model divides music into four moods which include contentment, depression, exuberance, and anxious/frantic. A mood detection algorithm uses a hierarchical mood detection framework to determine which of the four moods is associated with a music clip based on the extracted features. In a first tier of the hierarchical detection process, the algorithm determines one of two mood groups to which the music clip belongs. In a second tier of the hierarchical detection process, the algorithm then determines which mood from within the selected mood group is the appropriate, exact mood for the music clip. Benefits of the mood detection system include automatic detection of music mood which can be used as music metadata to manage music through music representation and classification.

Claims

1. A computer-readable storage medium including instructions, which when executed by a computer implement a mood detection module to classify a music clip as a music mood according to music features extracted from the music clip, comprising:

an extraction tool to extract the music features; and

a hierarchical music mood detection module to determine a mood group based on a first music feature and to determine an exact music mood from within the mood group based on a second and third music feature; and

wherein the mood detection module classifies the music clip according to the determined mood for storage and retrieval of the music clip.

2. The computer-readable storage medium as recited in claim 1, wherein the extraction tool extracts an intensity feature, a timbre feature, and a rhythm feature from the music clip;

the hierarchical music mood detection module classifies the music clip into a mood group based on the intensity feature; and

the hierarchical music mood detection module classifies the music clip into an exact music mood from the mood group based on the timbre feature and the rhythm feature.

3. The computer-readable storage medium as recited in claim 1, wherein the extraction tool:

converts the music clip into a uniform music clip having a uniform format;

divides the uniform music clip into a plurality of frames; and

divides each frame into a plurality of octave-based frequency sub-bands.

4. The computer-readable storage medium as recited in claim 3, wherein the extraction tool:

calculates a root mean-square (RMS) signal amplitude for each sub-band of each frame;

sums the RMS signal amplitudes across the sub-bands of each frame to determine a frame intensity for each frame; and

averages the frame intensities to determine the intensity feature for the music clip.

5. The computer-readable storage medium as recited in claim 3, wherein the extraction tool extracts a timbre feature by:

calculating spectral shape features for each frame;

calculating spectral contrast features for each frame; and

representing the timbre feature with one or more of the spectral shape features or one or more of the spectral contrast features.

6. The computer-readable storage medium as recited in claim 3, wherein the extraction tool extracts a rhythm feature by:

extracting an amplitude envelope from the lowest sub-band and the highest sub-band of each frame across the uniform music clip;

estimating a difference curve of the amplitude envelope; and

detecting peaks above a threshold within the difference curve, the peaks being instrumental onsets.

7. The computer-readable storage medium as recited in claim 6, wherein the extraction toot extracts a rhythm feature by:

extracting an average rhythm strength of the instrumental onsets;

extracting a rhythm regularity value based on the average of the maximum three peaks in the difference curve; and

extracting a rhythm tempo based on a common divisor of peaks in the difference curve.

8. The computer-readable storage medium as recited in claim 1, wherein the hierarchical music mood detection module classifies the music clip into a mood group by:

determining the probability of a first mood group based on the intensity feature;

determining the probability of a second mood group based on the intensity feature;

selecting the first mood group if the probability of the first mood group is greater than or equal to the probability of the second mood group; and

otherwise selecting the second mood group.

9. The computer-readable storage medium as recited in claim 1, wherein the hierarchical music mood detection module classifies the music clip into a mood group selected from the group of mood groups consisting of:

a contentment and depression mood group; and

an exuberance and anxious mood group.

10. The computer-readable storage medium as recited in claim 1, wherein the mood group includes a first mood and a second mood, and the hierarchical music mood detection module classifies the music clip into an exact music mood by:

determining the probability of the first mood based on the timbre feature and the rhythm feature;

determining the probability of the second mood based on the timbre feature and the rhythm feature;

selecting the first mood as the exact mood if the probability of the first mood is greater than or equal to the probability of the second mood; and

otherwise selecting the second mood as the exact mood.

11. The computer-readable storage medium as recited in claim 10, wherein the mood group is selected from the group of mood groups consisting of:

a first mood group that includes a contentment mood and a depression mood; and

a second mood group that includes an exuberance mood and an anxious mood.

12. A mood detection system to determine a mood of a music clip, comprising:

a computing device;

a music feature extraction tool running on the computing device to extract music features from the music clip;

a hierarchical mood detector to determine a mood group of the music clip based on a first music feature extracted by the music feature extraction tool and to determine an exact music mood from within the mood group based on a second and third music feature extracted by the music feature extraction tool; and

wherein the mood detection system indexes the music clip according to the determined mood for storage and retrieval of the music clip.

13. The mood detection system as recited in claim 12, wherein the music features comprise an intensity feature, a timbre feature, and a rhythm feature.

14. The mood detection system as recited in claim 13, further comprising:

a first classifier that classifies the music clip into a mood group based on the intensity feature; and

a second classifier that classifies the music clip into an exact music mood from the mood group based on the timbre feature and the rhythm feature.

15. The mood detection system as recited in claim 14, wherein the first classifier classifies the music clip into a mood group by:

determining the probability of a first mood group based on the intensity feature;

determining the probability of a second mood group based on the intensity feature;

selecting the first mood group if the probability of the first mood group is greater than or equal to the probability of the second mood group; and

otherwise selecting the second mood group.

16. The mood detection system as recited in claim 15, wherein the first classifier classifies the music clip into a mood group selected from the group of mood groups consisting of:

a contentment and depression mood group; and

an exuberance and anxious mood group.

17. The mood detection system as recited in claim 14, wherein the second classifier classifies the music clip into an exact music mood by:

determining the probability of the first mood based on the timbre feature and the rhythm feature;

determining the probability of the second mood based on the timbre feature and the rhythm feature

selecting the first mood as the exact mood if the probability of the first mood is greater than or equal to the probability of the second mood; and

otherwise selecting the second mood as the exact mood.

18. A system, comprising:

means for extracting an intensity feature, a timbre feature, and a rhythm feature from a music clip;

means for classifying the music clip into a mood group based on the intensity feature;

means for classifying the music clip into an exact music mood from the mood group based on the timbre feature and the rhythm feature; and

means for indexing the music clip according to the determined mood for storage and retrieval of the music clip.