Systems and methods for detecting musical features in audio content

• US 10,546,566 B2
• Filed: 04/12/2019
• Issued: 01/28/2020
• Est. Priority Date: 11/08/2016
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Girardot, Agnes, Noel, Jean-Baptiste
• Primary Examiner(s): Tsang, Fan S
• Assistant Examiner(s): Siegel, David
• Application Number:
  US16/382,579
• Publication Number:
  US 20190237050A1
• Time in Patent Office: 291 Days
• CPC Class Codes: G10H 1/0008 Associated control or indicating means
• CPC Class Codes: G10H 1/40 Rhythm
• CPC Class Codes: G10H 2210/061 for extraction of musical phrases, isolation of musically relevant segments, e.g. musical thumbnail generation, or for temporal structure analysis of a musical piece, e.g. determination of the movement sequence of a musical work
• CPC Class Codes: G10H 2210/066 for pitch analysis as part of wider processing for musical purposes, e.g. transcription, musical performance evaluation; Pitch recognition, e.g. in polyphonic sounds; Estimation or use of missing fundamental
• CPC Class Codes: G10H 2220/005 Non-interactive screen display of musical or status data graphical user interfaces specifically adapted for electrophonic musical instruments G10H2220/091; fingering displays G10H2220/041
• CPC Class Codes: G10H 2250/015 Markov chains, e.g. hidden Markov models [HMM], for musical processing, e.g. musical analysis or musical composition

Abstract

Systems and methods for identifying musical features in audio content are presented. Audio content information may be obtained from a digital audio file, the information providing a duration for playback of the audio content and a representation of sound frequencies associated with various moments throughout the duration of the audio content. Sound frequencies associated with one or more of the moments throughout the duration of the audio content may be identified, and characteristics or patterns of the identified sound frequencies may be recognized as being indicative of one or more musical features (e.g., parts, phrases, hits, bars, onbeats, beats, quavers, semiquavers, etc.). Some implementations of the present technology define display objects for display on a digital display, the display objects provided with visual features in an arrangement that distinguishes one musical feature from another across the duration of the audio content.

Claims

1. A system for identifying musical features in digital audio content, comprising:

one or more physical computer processors configured by computer readable instructions to;

obtain a digital audio file, the digital audio file including information representing audio content having a duration and sound frequencies associated with one or more moments in the audio content;

identify sound frequencies associated with a first moment and a second moment in the duration of the audio content;

identify one or more frequency characteristics associated with the first moment based on one or more of the sound frequencies associated with the first moment and the second moment;

identify one or more musical features associated with the first moment based on the one or more identified frequency characteristics associated with the first moment;

identify a transition in the audio content from a first part to a second part, the transition identified at a third moment in the duration of the audio content; and

adjust the identification of the transition from the third moment to a different moment in the duration of the audio content based on at least one of the one or more identified musical features.

2. The system of claim 1, wherein the one or more of the frequency characteristics include amplitude associated with the first moment.

3. The system of claim 1, wherein the identification of the transition is based on using a Hidden Markov Model.

4. The system of claim 1, wherein the identification of the one or more musical features is based on a match between one or more of the one or more identified frequency characteristics and a predetermined frequency pattern template corresponding to a particular musical feature.

5. The system of claim 1, wherein the identification of the transition is adjusted to the different moment to coincide with one of the one or more identified musical features.

6. The system of claim 5, wherein the one of the one or more identified musical features is selected for the adjustment of the identification of the transition based on a hierarchy of musical features, the hierarchy of musical features including an order of different types of musical features from a highest priority to a lowest priority.

7. The system of claim 6, wherein the one of the one or more identified musical features has the highest priority among the one or more identified musical features.

8. The system of claim 6, wherein the order includes, from the highest priority to the lowest priority, a phrase musical feature, a drop musical feature, a hit musical feature, a bar musical feature, an onbeat musical feature, a beat musical feature, a quaver musical feature, and a semiquaver musical feature.

9. The system of claim 1, wherein the identification of the transition is adjusted to the different moment to occur between two of the one or more identified musical features.

10. The system of claim 1, wherein the identification of the transition is adjusted further based on a first duration of the first part and/or a second duration of the second part being shorter than a threshold duration.

11. A method for identifying musical features in digital audio content, the method comprising the steps of:

obtaining a digital audio file, the digital audio file including information representing audio content having a duration and sound frequencies associated with one or more moments in the audio content;

identifying sound frequencies associated with a first moment and a second moment in the duration of the audio content;

identifying one or more frequency characteristics associated with the first moment based on one or more of the sound frequencies associated with the first moment and the second moment;

identifying one or more musical features associated with the first moment based on the one or more identified frequency characteristics associated with the first moment;

identifying a transition in the audio content from a first part to a second part, the transition identified at a third moment in the duration of the audio content; and

adjusting the identification of the transition from the third moment to a different moment in the duration of the audio content based on at least one of the one or more identified musical features.

12. The method of claim 11, wherein the one or more of the frequency characteristics include amplitude associated with the first moment.

13. The method of claim 11, wherein identifying the transition is based on using a Hidden Markov Model.

14. The method of claim 11, wherein the identification of the one or more musical features is based on a match between one or more of the one or more identified frequency characteristics and a predetermined frequency pattern template corresponding to a particular musical feature.

15. The method of claim 11, wherein the identification of the transition is adjusted to the different moment to coincide with one of the one or more identified musical features.

16. The method of claim 15, wherein the one of the one or more identified musical features is selected for the adjustment of the identification of the transition based on a hierarchy of musical features, the hierarchy of musical features including an order of different types of musical features from a highest priority to a lowest priority.

17. The method of claim 16, wherein the one of the one or more identified musical features has the highest priority among the one or more identified musical features.

18. The method of claim 16, wherein the order includes, from the highest priority to the lowest priority, a phrase musical feature, a drop musical feature, a hit musical feature, a bar musical feature, an onbeat musical feature, a beat musical feature, a quaver musical feature, and a semiquaver musical feature.

19. The method of claim 11, wherein the identification of the transition is adjusted to the different moment to occur between two of the one or more identified musical features.

20. The method of claim 11, wherein the identification of the transition is adjusted further based on a first duration of the first part and/or a second duration of the second part being shorter than a threshold duration.