Audio processing techniques for semantic audio recognition and report generation

• US 9,812,109 B2
• Filed: 10/16/2015
• Issued: 11/07/2017
• Est. Priority Date: 12/21/2012
• Status: Active Grant

• Current Assignee: The Nielsen Company LLC (MOOD Media Corporation)
• Original Assignee: The Nielsen Company LLC (MOOD Media Corporation)
• Inventors: Neuhauser, Alan, Stavropoulos, John
• Primary Examiner(s): ABEBE, DANIEL DEMELASH
• Application Number:
  US14/885,216
• Publication Number:
  US 20160035332A1
• Time in Patent Office: 753 Days
• Field of Search: 704243
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G10H 1/40 Rhythm
• CPC Class Codes: G10H 2210/036 of musical genre, i.e. analysing the style of musical pieces, usually for selection, filtering or classification
• CPC Class Codes: G10H 2210/066 for pitch analysis as part of wider processing for musical purposes, e.g. transcription, musical performance evaluation; Pitch recognition, e.g. in polyphonic sounds; Estimation or use of missing fundamental
• CPC Class Codes: G10H 2210/071 for rhythm pattern analysis or rhythm style recognition rhythm pattern per se G10H2210/341
• CPC Class Codes: G10H 2210/076 for extraction of timing, tempo; Beat detection tempo display G10H2220/081; tempo control G10H2210/375
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 19/018 Audio watermarking, i.e. embedding inaudible data in the audio signal

Abstract

System, apparatus and method for determining semantic information from audio, where incoming audio is sampled and processed to extract audio features, including temporal, spectral, harmonic and rhythmic features. The extracted audio features are compared to stored audio templates that include ranges and/or values for certain features and are tagged for specific ranges and/or values. Extracted audio features that are most similar to one or more templates from the comparison are identified according to the tagged information. The tags are used to determine the semantic audio data that includes genre, instrumentation, style, acoustical dynamics, and emotive descriptor for the audio signal.

Claims

1. An apparatus for forming an audio template for determining semantic audio information, the apparatus comprising:

a processor to;

extract a plurality of audio features from audio, at least one of the plurality of audio features including at least one of a temporal feature, a spectral feature, a harmonic feature, or a rhythmic feature;

determine a range for each of the plurality of audio features; and

store a set of ranges of the plurality of audio features to compare against other audio features from subsequent audio to generate a tag for the set of ranges signifying semantic audio information for the subsequent audio, wherein the set of ranges includes more than one range and the tag is associated with an audio timbre range, a beat range, a loudness range and a spectral histogram range.

2. The apparatus of claim 1, wherein the tag is associated with at least one of a genre descriptor, an instrumentation descriptor, a style descriptor, an acoustical dynamics descriptor, or an emotive descriptor for the audio.

3. The apparatus of claim 1, wherein the harmonic feature includes at least one of a pitch, a tonality, a pitch class profile, harmonic changes, a main pitch class, an octave range of dominant pitch, a main tonal interval relation, or an overall pitch strength of at least some of the audio, the rhythmic feature includes at least one of a rhythmic structure, a beat period, a rhythmic fluctuation, or an average tempo for at least some of the audio, the spectral feature includes at least one of a spectral centroid, a spectral rolloff, a spectral flux, a spectral flatness measure, a spectral crest factor, Mel-frequency cepstral coefficients, Daubechies wavelet coefficients, a spectral dissonance, a spectral irregularity, or a spectral inharmonicity of at least some of the audio, and the temporal feature includes at least one of amplitude, power, or zero crossing of at least some of the audio.

4. The apparatus of claim 1, wherein the tag is associated with timber and the set of ranges includes a range for a mean of a spectral centroid, a range for a variance of the spectral centroid, and a range of a percentage of low/high energy frames.

5. The apparatus of claim 1, wherein the tag is associated with beat and the set of ranges includes a range for an amplitude of peaks in a beat histogram, a range for periods of peaks in the beat histogram, and a range for a ratio between a peak and a sum of all peaks in the beat histogram.

6. The apparatus of claim 1, wherein the tag is associated with pitch and the set of ranges includes a range for an amplitude of prominent peaks in a pitch histogram, and a range for periods of peaks in the pitch histogram, wherein the pitch histogram is on a full semitone scale or an octave independent scale.

7. An apparatus for forming an audio template for determining semantic audio information, the apparatus comprising:

a processor to;

extract a plurality of audio features from audio, at least one of the plurality of audio features including at least one of a temporal feature, a spectral feature, a harmonic feature, or a rhythmic feature;

determine a range for each of the plurality of audio features; and

store a set of ranges of the plurality of audio features to compare against other audio features from subsequent audio to generate a tag for the set of ranges signifying semantic audio information for the subsequent audio, wherein the set of ranges includes more than one range and the tag is associated with timber and includes a range for a mean of a spectral centroid, a range for a variance of the spectral centroid, and a range of a percentage of low/high energy frames.

8. The apparatus of claim 7, wherein the tag is associated with at least one of a genre descriptor, an instrumentation descriptor, a style descriptor, an acoustical dynamics descriptor, or an emotive descriptor for the audio.

9. The apparatus of claim 7, wherein the tag is further associated with beat and the set of ranges further includes a range for an amplitude of peaks in a beat histogram, a range for periods of peaks in the beat histogram, and a range for a ratio between a peak and a sum of all peaks in the beat histogram.

10. The apparatus of claim 7, wherein the tag is further associated with pitch and the set of ranges further includes a range for an amplitude of prominent peaks in a pitch histogram, a range for periods of peaks in the pitch histogram, wherein the pitch histogram is on a full semitone scale or an octave independent scale.

11. The apparatus of claim 7, wherein the harmonic feature includes at least one of a pitch, a tonality, a pitch class profile, harmonic changes, a main pitch class, an octave range of dominant pitch, a main tonal interval relation, or an overall pitch strength of at least some of the audio, the rhythmic feature includes at least one of a rhythmic structure, a beat period, a rhythmic fluctuation, or an average tempo for at least some of the audio, the spectral feature includes at least one of a spectral centroid, a spectral rolloff, a spectral flux, a spectral flatness measure, a spectral crest factor, Mel-frequency cepstral coefficients, Daubechies wavelet coefficients, a spectral dissonance, a spectral irregularity, or a spectral inharmonicity of at least some of the audio, and the temporal feature includes at least one of amplitude, power, or zero crossing of at least some of the audio.

12. An apparatus for forming an audio template for determining semantic audio information, the apparatus comprising:

a processor to;

extract a plurality of audio features from audio, at least one of the plurality of audio features including at least one of a temporal feature, a spectral feature, a harmonic feature, or a rhythmic feature;

determine a range for each of the plurality of audio features; and

store a set of ranges of the plurality of audio features to compare against other audio features from subsequent audio to generate a tag for the set of ranges signifying semantic audio information for the subsequent audio, wherein the set of ranges includes more than one range and the tag is associated with beat and includes a range for an amplitude of peaks in a beat histogram, a range for periods of peaks in the beat histogram, and a range for a ratio between a peak and a sum of all peaks in the beat histogram.

13. The apparatus of claim 12, wherein the tag is associated with at least one of a genre descriptor, an instrumentation descriptor, a style descriptor, an acoustical dynamics descriptor, or an emotive descriptor for the audio.

14. The apparatus of claim 12, wherein the tag is further associated with timber and the set of ranges further includes a range for a mean of a spectral centroid, a range for a variance of the spectral centroid, and a range of a percentage of low/high energy frames.

15. The apparatus of claim 12, wherein the tag is further associated with pitch and the set of ranges further includes a range for an amplitude of prominent peaks in a pitch histogram, and a range for periods of peaks in the pitch histogram, wherein the pitch histogram is on a full semitone scale or an octave independent scale.

16. The apparatus of claim 12, wherein the harmonic feature includes at least one of a pitch, a tonality, a pitch class profile, harmonic changes, a main pitch class, an octave range of dominant pitch, a main tonal interval relation, or an overall pitch strength of at least some of the audio, the rhythmic feature includes at least one of a rhythmic structure, a beat period, a rhythmic fluctuation, or an average tempo for at least some of the audio, the spectral feature includes at least one of a spectral centroid, a spectral rolloff, a spectral flux, a spectral flatness measure, a spectral crest factor, Mel-frequency cepstral coefficients, Daubechies wavelet coefficients, a spectral dissonance, a spectral irregularity, or a spectral inharmonicity of at least some of the audio, and the temporal feature includes at least one of amplitude, power, or zero crossing of at least some of the audio.

17. An apparatus for forming an audio template for determining semantic audio information, the apparatus comprising:

a processor to;

extract a plurality of audio features from audio, at least one of the plurality of audio features including at least one of a temporal feature, a spectral feature, a harmonic feature, or a rhythmic feature;

determine a range for each of the plurality of audio features; and

store a set of ranges of the plurality of audio features to compare against other audio features from subsequent audio to generate a tag for the set of ranges signifying semantic audio information for the subsequent audio, wherein the set of ranges includes more than one range and the tag is associated with pitch and includes a range for an amplitude of prominent peaks in a pitch histogram, and a range for periods of peaks in the pitch histogram, wherein the pitch histogram is on a full semitone scale or an octave independent scale.

18. The apparatus of claim 17, wherein the tag is associated with at least one of a genre descriptor, an instrumentation descriptor, a style descriptor, an acoustical dynamics descriptor, or an emotive descriptor for the audio.

19. The apparatus of claim 17, wherein the tag is further associated with timber and the set of ranges further includes a range for a mean of a spectral centroid, a range for a variance of the spectral centroid, and a range of a percentage of low/high energy frames.

20. The apparatus of claim 17, wherein the tag is further associated with beat and the set of ranges further includes a range for an amplitude of peaks in a beat histogram, a range for periods of peaks in the beat histogram, and a range for a ratio between a peak and a sum of all peaks in the beat histogram.

21. The apparatus of claim 17, wherein the harmonic feature includes at least one of a pitch, a tonality, a pitch class profile, harmonic changes, a main pitch class, an octave range of dominant pitch, a main tonal interval relation, or an overall pitch strength of at least some of the audio, the rhythmic feature includes at least one of a rhythmic structure, a beat period, a rhythmic fluctuation, or an average tempo for at least some of the audio, the spectral feature includes at least one of a spectral centroid, a spectral rolloff, a spectral flux, a spectral flatness measure, a spectral crest factor, Mel-frequency cepstral coefficients, Daubechies wavelet coefficients, a spectral dissonance, a spectral irregularity, or a spectral inharmonicity of at least some of the audio, and the temporal feature includes at least one of amplitude, power, or zero crossing of at least some of the audio.