Music searching methods based on human perception

• US 8,805,657 B2
• Filed: 11/02/2012
• Issued: 08/12/2014
• Est. Priority Date: 09/14/1999
• Status: Expired due to Fees

• Current Assignee: Gracenote, Inc. (RR Donnelley & Sons Company)
• Original Assignee: Gracenote, Inc. (RR Donnelley & Sons Company)
• Inventors: Wells, Maxwell J., Dhillon, Navdeep S., Waller, David
• Primary Examiner(s): JONES, HUGH M
• Application Number:
  US13/667,683
• Publication Number:
  US 20130191088A1
• Time in Patent Office: 648 Days
• Field of Search: 703/2
• US Class Current: 703/2
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/433 using audio data
• CPC Class Codes: G06F 16/634 Query by example, e.g. query by humming
• CPC Class Codes: G06F 16/68 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content
• CPC Class Codes: G06F 17/10 Complex mathematical operations function generation by table look-up G06F1/03; evaluation of elementary functions by calculation G06F7/544
• CPC Class Codes: G06F 2213/0038 System on Chip
• CPC Class Codes: G06F 2216/01 Automatic library building

Abstract

A method for characterizing a musical recording as a set of scalar descriptors, each of which is based on human perception. A group of people listens to a large number of musical recordings and assigns to each one many scalar values, each value describing a characteristic of the music as judged by the human listeners. Typical scalar values include energy level, happiness, danceability, melodicness, tempo, and anger. Each of the pieces of music judged by the listeners is then computationally processed to extract a large number of parameters which characterize the electronic signal within the recording. Algorithms are empirically generated which correlate the extracted parameters with the judgments based on human perception to build a model for each of the scalars of human perception. These models can then be applied to other music which has not been judged by the group of listeners to give to each piece of music a set of scalar values based on human perception. The set of scalar values can be used to find other pieces that sound similar to humans or vary in a dimension of one of the scalars.

Claims

1. A method comprising:

analyzing a musical recording by performing digital signal processing to obtain a mathematical analysis of sounds recorded in the musical recording;

calculating a first derivative parameter of the musical recording from the mathematical analysis of the sounds recorded in the musical recording;

using a computer, determining a second derivative parameter of the musical recording based on the calculated first derivative parameter of the musical recording,

the second derivative parameter being a scalar that represents an extent to which a descriptor is humanly perceivable in music represented by the musical recording;

storing the determined second derivative parameter of the musical recording as a characteristic of the musical recording;

receiving a search query that references a portion of a representative recording for which a similar recording is sought;

analyzing the portion of the representative recording by performing digital signal processing to obtain a further mathematical analysis of sounds recorded in the portion;

calculating a further first derivative parameter of the portion from the further mathematical analysis of the sounds recorded in the portion;

determining a further second derivative parameter of the portion,

the further second derivative parameter being a further scalar that represents a different extent to which the descriptor is humanly perceivable in the portion; and

in response to the search query, providing a search result that references the musical recording based on a comparison of the scalar to the further scalar.

2. The method of claim 1, wherein:

the determining of the second derivative parameter includes multiplying the first derivative parameter by a weight factor.

3. The method of claim 2, wherein:

the weight factor by which the first derivative parameter is multiplied is nonlinear with respect to the first derivative parameter.

4. The method of claim 2, wherein:

the weight factor by which the first derivative parameter is multiplied is accessed from a lookup table that correlates the weight factor with the first derivative parameter.

5. The method of claim 4, wherein:

the lookup table from which the weight factor is accessed includes values generated based on empirical data from human listeners of a plurality of musical recordings.

6. The method of claim 1, wherein:

the calculating of the first derivative parameter is performed as part of calculating a set of first derivative parameters from the mathematical analysis of the sounds recorded in the musical recording; and

the determining of the second derivative parameter is based on the calculated set of first derivative parameters.

7. The method of claim 6, wherein:

the determining of the second derivative parameter includes individually weighting each first derivative parameter among the set of first derivative parameters.

8. The method of claim 7, wherein:

the weighting of each first derivative parameter is based on an empirically generated lookup table that correlates each first derivative parameter with a corresponding weight factor.

9. The method of claim 1, wherein:

the calculated first derivative parameter represents a mathematical measurement extracted from the mathematical analysis of the sounds recorded in the musical recording,

the mathematical measurement being selected from a group consisting of harmonicity, loudness, dynamic range, rhythm strength, rhythm complexity, articulation, attack, note duration, tempo, and key.

10. The method of claim 1, wherein:

the determined second derivative parameter quantifies the descriptor that is humanly perceivable in the music represented by the musical recording,

the descriptor being a descriptive word selected from a group consisting of happiness, anger, and danceability.

11. The method of claim 1 further comprising:

receiving a search query that specifies a range that corresponds to the descriptor that is humanly perceivable in the music;

in response to the search query, providing a search result that references the musical recording based on the scalar being within the range specified by the search query.

12. The method of claim 1 further comprising:

receiving a search query that identifies a representative recording;

accessing a further second derivative parameter of the representative recording,

the further second derivative parameter being a further scalar that represents a different extent to which the descriptor is humanly perceivable in the representative recording; and

in response to the search query, providing a search result that references the musical recording based on a comparison of the scalar to the further scalar.

13. The method of claim 1, wherein:

the received search query includes the portion of the representative recording.

14. The method of claim 1, wherein:

the received search query identifies a location of the portion of the representative recording; and

the analyzing of the portion of the representative recording includes accessing the portion based on the identified location.

15. The method of claim 1, wherein:

the calculated first derivative parameter of the musical recording characterizes the sounds that are recorded in the musical recording.

16. The method of claim 1, wherein:

the determined second derivative parameter of the musical recording characterizes the music that is represented by the musical recording.

17. The method of claim 1, wherein:

the calculated first derivative parameter a machine-derived value extracted from the mathematical analysis of the musical recording;

the determined second derivative parameter represents the extent to which the descriptor is perceivable by a human in the music represented by the analyzed musical recording.

18. A non-transitory computer-readable medium comprising a computer program that, when executed by a computer, causes the computer to perform operations comprising:

analyzing a musical recording by performing digital signal processing to obtain a mathematical analysis of sounds recorded in the musical recording;

calculating a first derivative parameter of the musical recording from the mathematical analysis of the sounds recorded in the musical recording;

determining a second derivative parameter of the musical recording based on the calculated first derivative parameter of the musical recording,

the second derivative parameter being a scalar that represents an extent to which a descriptor is humanly perceivable in music represented by the musical recording;

storing the determined second derivative parameter of the musical recording as a characteristic of the musical recording;

receiving a search query that references a portion of a representative recording for which a similar recording is sought;

analyzing the portion of the representative recording by performing digital signal processing to obtain a further mathematical analysis of sounds recorded in the portion;

calculating a further first derivative parameter of the portion from the further mathematical analysis of the sounds recorded in the portion;

determining a further second derivative parameter of the portion,

the further second derivative parameter being a further scalar that represents a different extent to which the descriptor is humanly perceivable in the portion; and

in response to the search query, providing a search result that references the musical recording based on a comparison of the scalar to the further scalar.

19. A computer system comprising:

a computer; and

a computer program that, when executed by the computer, causes the computer to perform operations comprising;

analyzing a musical recording by performing digital signal processing to obtain a mathematical analysis of sounds recorded in the musical recording;

calculating a first derivative parameter of the musical recording from the mathematical analysis of the sounds recorded in the musical recording;

determining a second derivative parameter of the musical recording based on the calculated first derivative parameter of the musical recording,

the second derivative parameter being a scalar that represents an extent to which a descriptor is humanly perceivable in music represented by the musical recording;

storing the determined second derivative parameter of the musical recording as a characteristic of the musical recording;

receiving a search query that references a portion of a representative recording for which a similar recording is sought;

analyzing the portion of the representative recording by performing digital signal processing to obtain a further mathematical analysis of sounds recorded in the portion;

calculating a further first derivative parameter of the portion from the further mathematical analysis of the sounds recorded in the portion;

determining a further second derivative parameter of the portion,

the further second derivative parameter being a further scalar that represents a different extent to which the descriptor is humanly perceivable in the portion; and

in response to the search query, providing a search result that references the musical recording based on a comparison of the scalar to the further scalar.