Audio matching with semantic audio recognition and report generation

• US 10,360,883 B2
• Filed: 08/29/2017
• Issued: 07/23/2019
• Est. Priority Date: 12/21/2012
• Status: Active Grant

• Current Assignee: The Nielsen Company LLC (MOOD Media Corporation)
• Original Assignee: Nielsen Company (US) LLC (RR Donnelley & Sons Company)
• Inventors: Neuhauser, Alan, Stavropoulos, John
• Primary Examiner(s): Abebe, Daniel
• Application Number:
  US15/689,225
• Publication Number:
  US 20170358283A1
• Time in Patent Office: 693 Days
• Field of Search: 704203
• CPC Class Codes: G10H 1/0008 Associated control or indicating means
• CPC Class Codes: G10H 2210/036 of musical genre, i.e. analysing the style of musical pieces, usually for selection, filtering or classification
• CPC Class Codes: G10H 2210/041 based on mfcc [mel -frequency spectral coefficients]
• CPC Class Codes: G10H 2210/056 for extraction or identification of individual instrumental parts, e.g. melody, chords, bass; Identification or separation of instrumental parts by their characteristic voices or timbres
• CPC Class Codes: G10H 2210/066 for pitch analysis as part of wider processing for musical purposes, e.g. transcription, musical performance evaluation; Pitch recognition, e.g. in polyphonic sounds; Estimation or use of missing fundamental
• CPC Class Codes: G10H 2210/071 for rhythm pattern analysis or rhythm style recognition rhythm pattern per se G10H2210/341
• CPC Class Codes: G10H 2210/076 for extraction of timing, tempo; Beat detection tempo display G10H2220/081; tempo control G10H2210/375
• CPC Class Codes: G10H 2210/081 for automatic key or tonality recognition, e.g. using musical rules or a knowledge base
• CPC Class Codes: G10H 2250/235 Fourier transform; Discrete Fourier Transform [DFT]; Fast Fourier Transform [FFT]
• CPC Class Codes: G10H 2250/251 Wavelet transform, i.e. transform with both frequency and temporal resolution, e.g. for compression of percussion sounds; Discrete Wavelet Transform [DWT]
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence
• CPC Class Codes: G10L 25/51 for comparison or discrimination
• CPC Class Codes: G10L 25/81 for discriminating voice from music
• CPC Class Codes: G10L 25/84 for discriminating voice from noise

Abstract

Example articles of manufacture and apparatus disclosed herein for producing supplemental information for audio signature data obtain the audio signature data of a first time period including data relating to at least one of time or frequency components representing a first characteristic of media. Disclosed examples also obtain first semantic audio signature data, for the first time period, that is a measure of generalized information representing characteristics of the media. Disclosed examples further store the audio signature data of the first time period in association with a second time period when it is determined that second semantic audio signature data for the second time period substantially matches the first semantic audio signature data for the first time period.

Claims

1. An article of manufacture comprising instructions that, when executed, cause a processor to at least:

obtain audio signature data of a first time period, the audio signature data including data relating to at least one of time or frequency components representing a first characteristic of media;

obtain first semantic audio signature data for the first time period, the first semantic audio signature data being a measure of generalized information representing characteristics of the media; and

store the audio signature data of the first time period in association with a second time period when the processor determines that second semantic audio signature data for the second time period substantially matches the first semantic audio signature data for the first time period.

2. The article of manufacture of claim 1, wherein the measure of generalized information representing characteristics of the media includes a timber feature having at least one of a mean of a spectral centroid, a variance of the spectral centroid, or a percentage of low/high energy frames.

3. The article of manufacture of claim 1, wherein the measure of generalized information representing characteristics of the media includes a beat feature having at least one of an amplitude of peaks in a beat histogram, a period of peaks in the beat histogram, or a ratio between a peak and a sum of all peaks in the beat histogram.

4. The article of manufacture of claim 1, wherein the measure of generalized information representing characteristics of the media includes a pitch feature having at least one of an amplitude of prominent peaks in a pitch histogram, or a period of peaks in the pitch histogram, where the pitch histogram is on a full semitone scale or an octave independent scale.

5. The article of manufacture of claim 1, wherein the first semantic audio signature data includes at least one of a temporal feature, a spectral feature, a harmonic feature, or a rhythmic feature.

6. The article of manufacture of claim 1, wherein the audio signature data is generated by transforming an audio signal of the media from a time domain to a frequency domain.

7. The article of manufacture of claim 1, wherein the first semantic audio signature data is generated by transforming an audio signal of the media from a time domain to a frequency domain.

8. A apparatus for producing supplemental information for audio signature data, the apparatus including:

a processor to;

obtain the audio signature data at an input from a data network, the audio signature data received from a device, the audio signature data including data relating to at least one of time or frequency components representing a first characteristic of media;

obtain semantic audio signature data at the input from the data network, the semantic audio signature data received from the device, the semantic audio signature data being a measure of at least one of a timber feature, a beat feature or a pitch feature relating to a second characteristic of the media;

associate the semantic audio signature data to the audio signature data; and

process the associated semantic audio signature data and audio signature data to determine a change in the second characteristic relative to the first characteristic.

9. The apparatus of claim 8, wherein the timber feature includes one of a mean of a spectral centroid, a variance of the spectral centroid, or a percentage of low/high energy frames.

10. The apparatus of claim 8, wherein the beat feature includes one of an amplitude of peaks in a beat histogram, a period of peaks in the beat histogram, or a ratio between a peak and a sum of all peaks in the beat histogram.

11. The apparatus of claim 8, wherein the pitch feature includes one of an amplitude of prominent peaks in a pitch histogram, or a period of peaks in the pitch histogram, where the pitch histogram is on a full semitone scale or an octave independent scale.

12. The apparatus of claim 8, wherein the first semantic audio signature data includes at least one of a temporal feature, a spectral feature, a harmonic feature, or a rhythmic feature.

13. An article of manufacture comprising instructions that, when executed, cause a processor to at least:

obtain the audio signature data at an input from a data network, the audio signature data received from a device, the audio signature data including data relating to at least one of time or frequency components representing a first characteristic of media;

obtain semantic audio signature data at the input from the data network, the semantic audio signature data received from the device, the semantic audio signature data being a measure of at least one of a timber feature, a beat feature or a pitch feature relating to a second characteristic of the media;

associate the semantic audio signature data to the audio signature data; and

process the associated semantic audio signature data and audio signature data to determine a change in the second characteristic relative to the first characteristic.

14. The article of manufacture of claim 13, wherein the timber feature includes one of a mean of a spectral centroid, variance of the spectral centroid, or a percentage of low/high energy frames.

15. The article of manufacture of claim 13, wherein the beat feature includes one of an amplitude of peaks in a beat histogram, a period of peaks in the beat histogram, or a ratio between a peak and a sum of all peaks in the beat histogram.

16. The article of manufacture of claim 13, wherein the pitch feature includes one of an amplitude of prominent peaks in a pitch histogram, or a period of peaks in the pitch histogram, where the pitch histogram is on a full semitone scale or an octave independent scale.

17. The article of manufacture of claim 13, wherein the first semantic audio signature data includes at least one of a temporal feature, a spectral feature, a harmonic feature, or a rhythmic feature.