Method and system of representing musical information in a digital representation for use in content-based multimedia information retrieval

• US 20040093354A1
• Filed: 09/23/2003
• Published: 05/13/2004
• Est. Priority Date: 03/23/2001
• Status: Abandoned Application

• Current Assignee: Kent Ridge Digital Labs
• Original Assignee: Kent Ridge Digital Labs
• Inventors: Xu, Changsheng, Zhu, Yongwei
• Application Number:
  US10/670,083
• Publication Number:
  US 20040093354A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/634 Query by example, e.g. query by humming
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content

Abstract

The invention relates to content-based audio/music retrieval and other content-based multimedia information retrieval. In one aspect the present invention provides a method of representing audio/musical information in a digital representation suitable for use in content-based information indexing and retrieval including the steps of: determining a first representation including a set of peaks and valley corresponding to maximum and minimum values respectively of at least one characteristic of the audio/music, and; determining a second representation including values representing relative differences between peaks and valleys. The invention presents a method and a system for content-based music retrieval. A music score database is constructed to provide a unique representation of real music songs. Score keywords are extracted from the music score as the features of the musing songs.

Claims

1. A method of representing audio/musical information in a digital representation suitable for use in content-based information indexing and retrieval, the method comprising:

a) determining a first representation including a set of peaks and valleys corresponding to maximum and minimum values respectively of at least one characteristic of the audio/music; and

b) determining a second representation including values representing relative differences between the determined peaks and valleys.

2. A method as claimed in claim 1, further including:

c) determining a histogram of the first representation.

3. A method as claimed in claim 2, wherein the histogram of the first representation includes a representation of, the population, or duration, of peaks or valleys in a given time interval.

4. A method as claimed in claim 1, wherein the relative difference value for a peak is given by:

the difference between the magnitude of a valley immediately following the peak and the magnitude of the peak, and;

the relative difference value of a valley is given by;

the difference between the magnitude of a peak immediately following the valley and the magnitude of the valley.

5. A method as claimed in claim 1, further including:

d) determining a histogram of the second representation.

6. A method as claimed in claim 1, wherein the audio/musical information is a music score.

7. A method as claimed in claim 6, further including pre-processing the music score before performing a), wherein the pre-processing includes:

removing zero notes from the music score, and;

adjoining the remaining nonzero notes to fill any gaps left by the removed zero notes.

8. A method as claimed in claim 1, wherein the audio/musical information is an acoustic signal.

9. A method as claimed in claim 8, wherein the acoustic signal is a vocal or humming signal.

10. A method as claimed in claim 8, further including preprocessing the acoustic signal before performing a), wherein the pre-processing includes:

converting the acoustic signal to a digital signal;

removing noise from the digital signal;

subjecting the noise free digital signal to pitch detection; and

subjecting the pitch detected digital signal to interval or note detection.

11. A method as claimed in claim 10, wherein the pitch detection includes a windowed Fourier transform and auto-correlation of the noise free digital signal.

12. A method as claimed in claim 10, wherein the interval or note detection includes logarithmically scaling the pitch detected digital signal.

13. A method as claimed in claim 1, wherein the characteristic of the audio/music is any one or more of the following:

volume level;

pitch; and

interval information.

14. A method of creating a music score database, comprising:

representing an actual music track uniquely with a music score such that there is a link between the music score and the actual music track;

representing the music score in accordance with a representing method to form search keywords, wherein the representing method is adapted to represent audio/musical information in a digital representation suitable for use in content-based information indexing and retrieval, the representing method comprising;

determining a first representation including a set of peaks and valleys corresponding to maximum and minimum values respectively of at least one characteristic of the audio/music; and

determining a second representation including values representing relative differences between the determined peaks and valleys, whrein the audio/musical information is the music score; and

storing the search keywords in a database.

15. A method as claimed in claim 14, further including:

creating at least one index for storage with the database, the at least one index including a global feature corresponding to an entire music score wherein the global feature includes the histogram of the second representation.

16. A method of creating a query keyword from an acoustic input for retrieval of music information in a music score database, the method comprising:

representing the acoustic input in a digital representation in accordance with a representing method, wherein the representing method is adapted to represent audio/musical information in a digital representation suitable for use in content-based information indexing and retrieval,

wherein the representing method comprises;

determining a first representation including a set of peaks and valleys corresponding to maximum and minimum values respectively of at least one characteristic of the audio/music; and

determining a second representation including values representing relative differences between the determined peaks and valleys, whrein the audio/musical information is an acoustic signal.

17. A method of retrieving audio/music information from a music score database, by matching query keywords with database keywords, the method comprising:

a) comparing a query keyword, created from an acoustic input for retrieval of music information in a music score database, with a global feature corresponding to each music score to eliminate non-relevant database keywords;

b) comparing the second representation of the query with the second representation of each database keyword; and

c) comparing the histogram of the first representation of the query with the histogram of the first representation of each database keyword.

18. A method of creating a music score database, comprising:

a) using a music score to uniquely represent an actual music song such that there is a link provided between a music score database and a music database;

b) using a curve including a set of digital values to represent the music score information and;

c) using peaks and valleys of the curve so as to index the music score database.

19. A method of converting a music score into score keywords, comprising:

a) preprocessing a score curve so as to remove zero notes, the score curve including a set of digital values representing musical notes;

b) detecting peaks and valleys of the score curve;

c) calculating the distance between each peak/valley and valley/peak pair; and

d) using the peaks and valleys as reference points, and a note histogram of the peaks and valleys to serve as score keywords.

20. A method of creating indexes to organise a music score database created in accordance with a method, comprising:

constructing a global feature for the complete actual music song, wherein the global feature is the histogram of the values of the distances between each peak/valley and valley/peak pair,

wherein the music score database creating method comprises;

using a music score to uniquely represent an actual music song such that there is a link provided between a music score database and a music database;

using a curve including a set of digital values to represent the music score information and;

using peaks and valleys of the curve so as to index the music score database.

21. A method of automatically converting acoustic input in the form of humming into query keywords, comprising:

a) converting the acoustic input into digital signal;

b) detecting the pitch from the digital signal;

c) converting the pitch into notes;

d) representing the acoustic input by a pitch curve;

e) smoothing of the pitch curve by removing small peaks and valleys;

f) detecting peaks and valleys of the pitch curve; and

g) generating the query keywords using the peaks and valleys in accordance with a method, wherein the method comprises calculating the distance between each peak/valley and valley/peak pair; and

using the peaks and valleys as reference points, and a note histogram of the peaks and valleys to serve as score keywords.

22. A method of matching query keywords with music score keywords, comprising:

a) checking a global feature for the complete actual music song, wherein the global feature is the histogram of the values of the distances between each peak/valley and valley/peak pair;

b) matching the sequence of peak/valley distance values of the query and the peak/valley distance values of the music score keywords; and

c) matching the note histogram by histogram intersection.

23. A system for representing audio/musical information in a digital representation suitable for use in content-based information indexing and retrieval, the system comprising:

means for determining a first representation including a set of peaks and valleys corresponding to maximum and minimum values respectively of at least one characteristic of the audio/music; and

means for determining a second representation including values representing relative differences between the determined peaks and valleys.

24. A system for creating a music score database, comprising:

means for using a music score to uniquely represent an actual music song such that there is a link provided between a music score database and a music database;

means for using a curve including a set of digital values to represent the music score information, and;

means for using peaks and valleys of the curve so as to index the music score database.

25. A system for converting a music score into score keywords, comprising:

means for preprocessing a score curve to remove zero notes, the score curve including a set of digital values representing musical notes;

means for detecting peaks and valleys of the score curve;

means for calculating the distance between each peak/valley and valley/peak pair; and

means for using the peaks and valleys as reference points, and a note histogram of the peaks and valleys to serve as score keywords.