Music searching methods based on human perception

• US 8,326,584 B1
• Filed: 04/21/2000
• Issued: 12/04/2012
• Est. Priority Date: 09/14/1999
• Status: Active Grant

• Current Assignee: CDDB, Inc. (RR Donnelley & Sons Company)
• Original Assignee: Gracenote, Inc. (RR Donnelley & Sons Company)
• Inventors: Dhillon, Navdeep S., Wells, Maxwell J., Waller, David
• Primary Examiner(s): JONES, HUGH M
• Application Number:
  US09/556,086
• Time in Patent Office: 4,610 Days
• Field of Search: 707/1, 704/222, 846/07, 703/6, 703/2
• US Class Current: 703/2
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/433 using audio data
• CPC Class Codes: G06F 16/634 Query by example, e.g. query by humming
• CPC Class Codes: G06F 16/68 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content
• CPC Class Codes: G06F 17/10 Complex mathematical operations function generation by table look-up G06F1/03; evaluation of elementary functions by calculation G06F7/544
• CPC Class Codes: G06F 2213/0038 System on Chip
• CPC Class Codes: G06F 2216/01 Automatic library building

Abstract

A method for characterizing a musical recording as a set of scalar descriptors, each of which is based on human perception. A group of people listens to a large number of musical recordings and assigns to each one many scalar values, each value describing a characteristic of the music as judged by the human listeners. Typical scalar values include energy level, happiness, danceability, melodicness, tempo, and anger. Each of the pieces of music judged by the listeners is then computationally processed to extract a large number of parameters which characterize the electronic signal within the recording. Algorithms are empirically generated which correlate the extracted parameters with the judgments based on human perception to build a model for each of the scalars of human perception. These models can then be applied to other music which has not been judged by the group of listeners to give to each piece of music a set of scalar values based on human perception. The set of scalar values can be used to find other pieces that sound similar to humans or vary in a dimension of one of the scalars.

Claims

1. A method for building a computational model of human perception of a descriptor of music, comprising:

a) extracting from each of at least five electronic representations of musical recordings at least two numeric parameters;

b) for each recording, combining the numeric parameters with a weighting for each parameter to compute a single number representing the descriptor for that recording;

c) adjusting the weightings for the parameters to find a set of weightings where each computed descriptor for each recording most closely matches perceptions reported for the recording by one or more human listeners.

3. A method as recited in claim 1, wherein the at least two numeric parameters are selected from dynamic range, loudness, harmonicity, rhythm strength, rhythm complexity, articulation, attack, note duration, tempo, sound salience and key.

4. A method as recited in claim 1, wherein the at least two numeric parameters include at least one of harmonicity, rhythm strength, rhythm complexity, articulation, attack, note duration, sound salience and key.

2. A computer readable medium containing a computer program which causes

a computer to perform a method for building a computational model of human perception of a descriptor of music, comprising:

a) extracting from each of at least five electronic representations of musical recordings at least two numeric parameters;

b) for each recording, combining the numeric parameters with a weightingfor each parameter to compute a single number representing the descriptor for that recording

c) adjusting the weightings for the parameters to find a set of weightings where each computed descriptor for each recording most closely matches perceptions reported for the recording by one or more human listeners.

5. A method for generating a data record associated with a music recording, the record comprising two or more scalar descriptors, each descriptor numerically describing the recording of music with which the data record is associated, comprising:

a) extracting from an electronic representation of the recording of music at least two numeric parameters;

b) combining the numeric parameters with a weighting for each parameter to compute a single number representing the descriptor for that recording, where the weightings were previously determined by;

c) extracting from an electronic representation of each of at least five musical recordings the same at least two numeric parameters;

d) for each recording, combining the numeric parameters with a weighting for each parameter to compute a single number representing the descriptor for that recording;

e) adjusting the weightings for the parameters to find a set of weightings where each computed descriptor for each recording most closely matches perceptions reported for the recording by one or more human listeners.

7. A method as recited in claim 5, wherein the at least two numeric parameters are selected from dynamic range, loudness, harmonicity, rhythm strength, rhythm complexity, articulation, attack, note duration, tempo, sound salience and key.

8. A method as recited in claim 5, wherein the at least two numeric parameters include at least one of harmonicity, rhythm strength, rhythm complexity, articulation, attack, note duration, sound salience and key.

6. A computer readable medium containing a computer program which causes a computer to perform a method for generating a data record associated with a music recording, the record comprising two or more scalar descriptors, each descriptor for numerically describing the recording of music with which the data record is associated, comprising:

a) extracting from an electronic representation of the recording of music at least two numeric parameters;

b) combining the numeric parameters with a weighting for each parameter to compute a single number representing the descriptor for that recording, where the weightings were previously determined by;

c) extracting from an electronic representation of each of at least five musical recordings the same at least two numeric parameters;

d) for each recording, combining the numeric parameters with a weighting for each parameter to compute a single number representing the descriptor for that recording;

e) adjusting the weightings for the parameters to find a set of weightings where each computed descriptor for each recording most closely matches perceptions reported for the recording by one or more human listeners.

9. A computer readable medium containing a computer extracted data record associated with a music recording, the record comprising:

two or more scalar descriptors, each descriptor numerically describing the recording of music with which the data record is associated, where each descriptor was generated by;

a) extracting from an electronic representation of the recording of music at least two numeric parameters;

b) combining the numeric parameters with a weighting for each parameter to compute a single number representing the descriptor for that recording, where the weightings were previously determined by;

c) extracting from an electronic representation of each of at least five musical recordings the same at least two numeric parameters;

d) for each recording, combining the numeric parameters with a weighting for each parameter to compute a single number representing the descriptor for that recording;

e) adjusting the weightings for the parameters to find a set of weightings where each computed descriptor for each recording most closely matches perceptions reported for the recording by one or more human listeners.

10. A method as recited in claim 9, wherein the at least two numeric parameters are selected from dynamic range, loudness, harmonicity, rhythm strength, rhythm complexity, articulation, attack, note duration, tempo, sound salience and key.

11. A method as recited in claim 9, wherein the at least two numeric parameters include at least one of harmonicity, rhythm strength, rhythm complexity, articulation, attack, note duration, sound salience and key.

12. A method for searching a database of data records associated with music recordings to find a desired recording, comprising:

a) identifying a comparison data record associated with a music recording in a computer readable database containing a plurality of data records, each associated with a music recording, the data records each comprising two or more scalar descriptors, each descriptor numerically describing the recording of music with which the data record is associated, where each descriptor was generated by;

1) extracting from an electronic representation of the recording of music at least two numeric parameters;

2) combining the numeric parameters with a weighting for each parameter to compute a single number representing the descriptor for that recording, where the weightings were previously determined by;

3) extracting from an electronic representation of each of at least five musical recordings the same at least two numeric parameters;

4) for each recording, combining the numeric parameters with a weighting for each parameter to compute a single number representing the descriptor for that recording;

5) adjusting the weightings for the parameters to find a set of weightings where each computed descriptor for each recording most closely matches perceptions reported for the recording by one or more human listeners; and

b) searching the database to find data records with descriptors that are similar to the descriptors to the comparison record.

13. The method of claim 12 further including, prior to searching the database, specifying that one of the descriptors of the comparison data record should be adjusted with an increase or a decrease, and the searching step is based on the descriptors of the comparison data record as adjusted.

16. A method as recited in claim 12, wherein the at least two numeric parameters are selected from dynamic range, loudness, harmonicity, rhythm strength, rhythm complexity, articulation, attack, note duration, tempo, sound salience and key.

17. A method as recited in claim 12, wherein the at least two numeric parameters include at least one of harmonicity, rhythm strength, rhythm complexity, articulation, attack, note duration, sound salience and key.

14. A computer readable medium containing a computer program which causes a computer to perform a method for searching a database of data records associated with music recordings to find a desired recording, comprising:

a) identifying a comparison data record associated with a music recording in a computer readable database containing a plurality of data records, each associated with a music recording. the data records each comprising two or more scalar descriptors, each descriptor numerically describing the recording of music with which the data record is associated, where each descriptor was generated by;

1) extracting from an electronic representation of the recording of music at least two numeric parameters;

2) combining the numeric parameters with a weighting for each parameter to compute a single number representing the descriptor for that recording, where the weightings were previously determined by;

3) extracting from an electronic representation of each of at least five musical recordings the same at least two numeric parameters;

4) for each recording, combining the numeric parameters with a weighting for each parameter to compute a single number representing the descriptor for that recording;

5) adjusting the weightings for the parameters to find a set of weightings where each computed descriptor for each recording most closely matches perceptions reported for the recording by one or more human listeners; and

b) searching the database to find data records with descriptors that are similar to the descriptors to the comparison record, and further including, prior to searching the database, specifying that one of the descriptors of the comparison data record should be adjusted with an increase or a decrease, and the searching step is based on the descriptors of the comparison data record as adjusted.

15. A computer readable medium containing a computer program which causes a computer to perform the a method for searching a database of data records associated with music recordings to find a desired recording, comprising:

a) identifying a comparison data record associated with a music recording in a computer readable database containing a plurality of data records, each associated with a music recording. the data records each comprising two or more scalar descriptors, each descriptor numerically describing the recording of music with which the data record is associated, where each descriptor was generated by;

1) extracting from an electronic representation of the recording of music at least two numeric parameters;

2) combining the numeric parameters with a weighting for each parameter to compute a single number representing the descriptor for that recording, where the weightings were previously determined by;

3) extracting from an electronic representation of each of at least five musical recordings the same at least two numeric parameters;

4) for each recording, combining the numeric parameters with a weighting for each parameter to compute a single number representing the descriptor for that recording;

5) adjusting the weightings for the parameters to find a set of weightings where each computed descriptor for each recording most closely matches perceptions reported for the recording by one or more human listeners; and

b) searching the database to find data records with descriptors that are similar to the descriptors to the comparison record.

18. A method for building a computational model of human perception of likeness between musical recordings, comprising:

a) extracting from each of at least five electronic representations of musical recordings at least two numeric parameters;

b) receiving from one or more human listeners who compare pairs of the musical recordings an indication of the human'"'"'s perception of likeness for each compared pair of recordings;

c) for each compared pair of the recordings, comparing each numeric parameter of one recording in the pair with the corresponding parameter of the second recording of the pair using an algorithm which produces a parameter comparison number representing the parameter comparison;

d) for each compared pair of the recordings, combining the parameter comparison numbers with a weighting for each parameter comparison number to compute a single difference number representing the difference between the two recordings of the pair;

e) adjusting the weightings for the comparison numbers to find a set of weightings where each computed difference number for each pair of recordings most closely matches perceptions reported for the pair of recordings by the one or more human listeners.

19. The method of claim 18 where the algorithm includes subtraction of parameter values.

21. The method of claim 18 where the algorithm includes computing a correlation between parameter values.

23. The method of claim 18 where, prior to the step of comparing the numeric parameters:

a) the parameters for each recording are combined with a weighting for each parameter to compute a single number representing a descriptor for that recording, where

b) the weightings were previously determined by adjusting the weightings to find a set of weightings where each computed descriptor for each recording most closely matches perceptions reported for the recording by one or more human listeners, and

c) the descriptors are then used in the step of comparing the numeric parameters in place of the parameters.

26. A method as recited in claim 18, wherein the at least two numeric parameters are selected from dynamic range, loudness, harmonicity, rhythm strength, rhythm complexity, articulation, attack, note duration, tempo, sound salience and key.

27. A method as recited in claim 18, wherein the at least two numeric parameters include at least one of harmonicity, rhythm strength, rhythm complexity, articulation, attack, note duration, sound salience and key.

20. A computer readable medium containing a computer program which causes a computer to perform a method for building a computational model of human perception of likeness between musical recordings, comprising:

a) extracting from each of at least five electronic representations of musical recordings at least two numeric parameters;

b) receiving from one or more human listeners who compare pairs of the musical recordings an indication of the human'"'"'s perception of likeness for each compared pair of recordings;

c) for each compared pair of the recordings, comparing each numeric parameter of one recording in the pair with the corresponding parameter of the second recording of the pair using an algorithm which produces a parameter comparison number representing the parameter comparison;

d) for each compared pair of the recordings, combining the parameter comparison numbers with a weighting for each parameter comparison number to compute a single difference number representing the difference between the two recordings of the pair;

e) adjusting the weightings for the comparison numbers to find a set of weights where each computed difference number for each pair of recordings most closely matches perceptions reported for the pair of recordings by the one or more human listeners where the algorithm includes subtraction of parameter values.

22. A computer readable medium containing a computer program which causes a computer to perform a method for building a computational model of human perception of likeness between musical recordings, comprising:

a) extracting from each of at least five electronic representations of musical recordings at least two numeric parameters;

b) receiving from one or more human listeners who compare pairs of the musical recordings an indication of the human'"'"'s perception of likeness for each compared pair of recordings;

c) for each compared pair of the recordings, comparing each numeric parameter of one recording in the pair with the corresponding parameter of the second recording of the pair using an algorithm which produces a parameter comparison number representing the parameter comparison;

d) for each compared pair of the recordings, combining the parameter comparison numbers with a weighting for each parameter comparison number to compute a single difference number representing the difference between the two recordings of the pair;

e) adjusting the weightings for the comparison numbers to find a set of weightings where each computed difference number for each pair of recordings most closely matches perceptions reported for the pair of recordings by the one or more human listeners where the algorithm includes computing a correlation between parameter values.

24. A computer readable medium containing a computer program which causes a computer to perform a method for building a computational model of human perception of likeness between musical recordings, comprising

a) extracting from each of at least five electronic representations of musical recordings at least two numeric parameters;

b) receiving from one or more human listeners who compare pairs of the musical recordings an indication of the human'"'"'s perception of likeness for each compared pair of recordings;

c) for each compared pair of the recordings, comparing each numeric parameter of one recording in the pair with the corresponding parameter of the second recording of the pair using an algorithm which produces a parameter comparison number representing the parameter comparison;

d) for each compared pair of the recordings, combining the parameter comparison numbers with a weighting for each parameter comparison number to compute a single difference number representing the difference between the two recordings of the pair;

e) adjusting the weightings for the comparison numbers to find a set of weightings where each computed difference number for each pair of recordings most closely matches perceptions reported for the pair of recordings by the one or more human listeners, where, prior to the step of comparing the numeric parameters;

a) the parameters for each recording are combined with a weighting for each parameter to compute a single number representing a descriptor for that recording, where

b) the weightings were previously determined by adjusting the weightings to find a set of weightings where each computed descriptor for each recording most closely matches perceptions reported for the recording by one or more human listeners and

c) the descriptors are then used in the step of comparing the numeric parameters in place of the parameters.

25. A computer readable medium containing a computer program which causes a computer to perform a method for building a computational model of human perception of likeness between musical recordings, comprising:

a) extracting from each of at least five electronic representations of musical recordings at least two numeric parameters;

b) receiving from one or more human listeners who compare pairs of the musical recordings an indication of the human'"'"'s perception of likeness for each compared pair of recordings;

c) for each compared pair of the recordings, comparing each numeric parameter of one recording in the pair with the corresponding parameter of the second recording of the pair using an algorithm which produces a parameter comparison number representing the parameter comparison;

d) for each compared pair of the recordings, combining the parameter comparison numbers with a weighting for each parameter comparison number to compute a single difference number representing the difference between the two recordings of the pair;

e) adjusting the weightings for the comparison numbers to find a set of weightings where each computed difference number for each pair of recordings most closely matches perceptions reported for the pair of recordings by the one or more human listeners.

28. A method for creating a database of differences between music recordings, comprising:

a) associating an identifier with each recording of a plurality of music recordings;

b) extracting from each recording of the plurality of recordings at least two numeric parameters;

c) computing from the extracted parameters for each of a plurality of pairs of the recordings a number which represents the difference between the recordings of the pair; and

d) assembling the computed difference numbers into a database where each computed difference is associated with the identifier for each of the two recordings from which the difference was computed.

29. The method of claim 28 where the computing step includes subtraction of parameter values.

30. The method of claim 28 where the computing step includes computing correlation between parameter values.

31. A computer readable medium containing a computer program which causes a computer to perform a method for creating a database of differences between music recordings, comprising:

a) associating an identifier with each recording of a plurality of music recordings;

b) extracting from each recording of the plurality of recordings at least two numeric parameters;

c) computing from the extracted parameters for each of a plurality of pairs of the recordings a number which represents the difference between the recordings of the pair; and

d) assembling the computed difference numbers into a database where each computed difference is associated with the identifier for each of the two recordings from which the difference was computed,

where the computing step includes computing correlation between parameter values.

32. A computer readable medium containing a computer program which causes a computer to perform a method for creating a database of differences between music recordings, comprising:

a) associating an identifier with each recording of a plurality of music recordings;

b) extracting from each recording of the plurality of recordings at least two numeric parameters;

c) computing from the extracted parameters for each of a plurality of pairs of the recordings a number which represents the difference between the recordings of the pair; and

d) assembling the computed difference numbers into a database where each computed difference is associated with the identifier for each of the two recordings from which the difference was computed.

33. A computer readable medium containing a computer program which causes a computer to perform a method for creating a database of differences between music recordings, comprising

a) associating an identifier with each recording of a plurality of music recordings;

b) extracting from each recording of the plurality of recordings at least two numeric parameters;

c) computing from the extracted parameters for each of a plurality of pairs of the recordings a number which represents the difference between the recordings of the pair; and

d) assembling the computed difference numbers into a database where each computed difference is associated with the identifier for each of the two recordings from which the difference was computed,

where the computing step includes subtraction of parameter values.