Audio matching with supplemental semantic audio recognition and report generation

• US 9,640,156 B2
• Filed: 09/23/2015
• Issued: 05/02/2017
• Est. Priority Date: 12/21/2012
• Status: Active Grant

• Current Assignee: The Nielsen Company LLC (MOOD Media Corporation)
• Original Assignee: The Nielsen Company LLC (MOOD Media Corporation)
• Inventors: Neuhauser, Alan, Stavropoulos, John
• Primary Examiner(s): GUERRA-ERAZO, EDGAR X
• Application Number:
  US14/862,508
• Publication Number:
  US 20160012807A1
• Time in Patent Office: 587 Days
• Field of Search: 704243, 704246, 704250
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G10H 1/0008 Associated control or indicating means
• CPC Class Codes: G10H 2210/041 based on mfcc [mel -frequency spectral coefficients]
• CPC Class Codes: G10H 2210/066 for pitch analysis as part of wider processing for musical purposes, e.g. transcription, musical performance evaluation; Pitch recognition, e.g. in polyphonic sounds; Estimation or use of missing fundamental
• CPC Class Codes: G10H 2210/071 for rhythm pattern analysis or rhythm style recognition rhythm pattern per se G10H2210/341
• CPC Class Codes: G10H 2210/076 for extraction of timing, tempo; Beat detection tempo display G10H2220/081; tempo control G10H2210/375
• CPC Class Codes: G10H 2210/081 for automatic key or tonality recognition, e.g. using musical rules or a knowledge base
• CPC Class Codes: G10H 2240/041 File watermark, i.e. embedding a hidden code in an electrophonic musical instrument file or stream for identification or authentification purposes audio watermarking G10L19/018
• CPC Class Codes: G10H 2240/081 Genre classification, i.e. descriptive metadata for classification or selection of musical pieces according to style analysis genre G10H2210/036
• CPC Class Codes: G10H 2240/085 Mood, i.e. generation, detection or selection of a particular emotional content or atmosphere in a musical piece
• CPC Class Codes: G10H 2240/095 Identification code, e.g. ISWC for musical works; Identification dataset
• CPC Class Codes: G10H 2250/031 Spectrum envelope processing
• CPC Class Codes: G10L 19/018 Audio watermarking, i.e. embedding inaudible data in the audio signal
• CPC Class Codes: G10L 25/03 characterised by the type of extracted parameters
• CPC Class Codes: G10L 25/09 the extracted parameters being zero crossing rates
• CPC Class Codes: G10L 25/18 the extracted parameters being spectral information of each sub-band
• CPC Class Codes: G10L 25/21 the extracted parameters being power information
• CPC Class Codes: G10L 25/24 the extracted parameters being the cepstrum
• CPC Class Codes: G10L 25/51 for comparison or discrimination

Abstract

System, apparatus and method for determining semantic information from audio, where incoming audio is sampled and processed to extract audio features, including temporal, spectral, harmonic and rhythmic features. The extracted audio features are compared to stored audio templates that include ranges and/or values for certain features and are tagged for specific ranges and/or values. The semantic information may be associated with audio codes to determine changing characteristics of identified media during a time period.

Claims

1. A processor-based method for producing supplemental information for media containing an embedded audio code, the method comprising:

obtaining an audio code from a device during a first time period, the audio code representing a first characteristic of the media, wherein the audio code is read from an audio portion of the media;

obtaining first semantic audio signature data from the device during the first time period, the first semantic audio signature data being a measure of at least one of a temporal feature, a spectral feature, a harmonic feature, or a rhythmic feature relating to a second characteristic of the media; and

associating the audio code of the first time period to a second time period when the processor determines that second semantic audio signature data of the second time period substantially matches the first semantic audio signature data for the first time period.

2. The method of claim 1, wherein:

the temporal feature includes at least one of an amplitude, a power, or a zero crossing of at least some of the audio of the media;

the spectral feature includes at least one of a spectral centroid, a spectral rolloff, a spectral flux, a spectral flatness measure, a spectral crest factor, Mel-frequency cepstral coefficients, Daubechies wavelet coefficients, a spectral dissonance, a spectral irregularity or a spectral inharmonicity of at least some of the audio of the media;

the harmonic feature includes at least one of a pitch, a tonality, a pitch class profile, harmonic changes, a main pitch class, an octave range of dominant pitch, a main tonal interval relation or an overall pitch strength of at least some of the audio of the media; and

the rhythmic feature includes at least one of a rhythmic structure, a beat period, a rhythmic fluctuation or an average tempo for at least some of the audio of the media.

3. The method of claim 1, wherein the audio code is decoded by transforming at least some of the audio from a time domain to a frequency domain.

4. The method of claim 1, wherein the first semantic audio signature data is formed by transforming at least some of the audio from a time domain to a frequency domain.

5. An apparatus for producing supplemental information for media containing an audio code, the apparatus comprising:

a processor to;

obtain an audio code from a device during a first time period, the audio code representing a first characteristic of the media, wherein the audio code is read from an audio portion of the media;

obtain first semantic audio signature data from the device during the first time period, the first semantic audio signature data being a measure of at least one of a temporal feature, a spectral feature, a harmonic feature, or a rhythmic feature relating to a second characteristic of the media; and

associate the audio code of the first time period to a second time period when second semantic audio signature data of the second time period substantially matches the first semantic audio signature data for the first time period.

6. The apparatus of claim 5, wherein:

the temporal feature includes at least one of an amplitude, a power and a zero crossing of at least some of the audio of the media;

the spectral feature includes at least one of a spectral centroid, a spectral rolloff, a spectral flux, a spectral flatness measure, a spectral crest factor, Mel-frequency cepstral coefficients, Daubechies wavelet coefficients, a spectral dissonance, a spectral irregularity, or a spectral inharmonicity of at least some of the audio of the media;

the harmonic feature includes at least one of a pitch, a tonality, a pitch class profile, harmonic changes, a main pitch class, an octave range of dominant pitch, a main tonal interval relation, or an overall pitch strength of at least some of the audio of the media; and

the rhythmic feature includes at least one of a rhythmic structure, a beat period, a rhythmic fluctuation, or an average tempo for at least some of the audio of the media.

7. The apparatus of claim 5, wherein the audio code is formed by transforming at least some of the audio from a time domain to a frequency domain.

8. The apparatus of claim 5, wherein the semantic audio signature data is formed by transforming at least some of the audio from a time domain to a frequency domain.

9. A tangible computer readable storage device comprising instructions, that when executed, cause a computing device to at least:

obtain an audio code from a device during a first time period, the audio code representing a first characteristic of media;

obtain first semantic audio signature data from the device during the first time period, the first semantic audio signature data being a measure of at least one of a temporal feature, a spectral feature, a harmonic feature, or a rhythmic feature relating to a second characteristic of the media; and

associate the audio code of the first time period to a second time period when second semantic audio signature data of the second time period substantially matches the first semantic audio signature data of the first time period.

10. The tangible computer readable storage device of claim 9, wherein:

the temporal feature includes at least one of an amplitude, a power and a zero crossing of at least some audio of the media;

the spectral feature includes at least one of a spectral centroid, a spectral rolloff, a spectral flux, a spectral flatness measure, a spectral crest factor, Mel-frequency cepstral coefficients, Daubechies wavelet coefficients, a spectral dissonance, a spectral irregularity, or a spectral inharmonicity of at least some of the audio of the media;

the harmonic feature includes at least one of a pitch, a tonality, a pitch class profile, harmonic changes, a main pitch class, an octave range of dominant pitch, a main tonal interval relation, or an overall pitch strength of at least some of the audio of the media; and

the rhythmic feature includes at least one of a rhythmic structure, a beat period, a rhythmic fluctuation, or an average tempo for at least some of the audio of the media.

11. The tangible computer readable storage device of claim 9, wherein the audio code is formed by transforming at least some of the audio from a time domain to a frequency domain.

12. The tangible computer readable storage device of claim 9, wherein the semantic audio signature data is formed by transforming at least some of the audio from a time domain to a frequency domain.

13. A processor-based method for producing supplemental information for media containing audio codes, the method comprising:

obtaining audio codes from a device, the audio codes representing a first characteristic of the media, wherein the audio codes are read from audio portions of the media;

obtaining semantic audio signature data from the device, the semantic audio signature data includes at least one of a temporal feature, a spectral feature, a harmonic feature, or a rhythmic feature relating to a second characteristic of the media;

associating the semantic audio signature data to the audio codes; and

processing the associated semantic audio signature data and audio codes to determine a change in the second characteristic relative to the first characteristic.

14. The method of claim 13, wherein:

the temporal feature includes at least one of an amplitude, a power, or a zero crossing of at least some of the audio of the media,

the spectral feature includes at least one of a spectral centroid, a spectral rolloff, a spectral flux, a spectral flatness measure, a spectral crest factor, Mel-frequency cepstral coefficients, Daubechies wavelet coefficients, a spectral dissonance, a spectral irregularity, or a spectral inharmonicity of at least some of the audio of the media,

the harmonic feature includes at least one of a pitch, a tonality, a pitch class profile, harmonic changes, a main pitch class, an octave range of dominant pitch, a main tonal interval relation, or an overall pitch strength of at least some of the audio of the media, and

the rhythmic feature includes at least one of a rhythmic structure, a beat period, a rhythmic fluctuation or an average tempo for at least some of the audio of the media.

15. The method of claim 13, wherein the first characteristic includes one of a media identification, a media distributor identification, or a media broadcaster identification.

16. The method of claim 13, wherein the second characteristic includes one of a genre, an instrumentation, a style, acoustical dynamics, or an emotive descriptor.

17. The method of claim 13, wherein the audio codes are formed by transforming at least some of the media from a time domain to a frequency domain.

18. The method of claim 13, wherein the semantic audio signature data is formed by transforming at least some of the media from a time domain to a frequency domain.

19. An apparatus for producing supplemental information for media containing audio codes, the apparatus comprising:

a processor to;

obtain audio codes from a device, the audio codes representing a first characteristic of the media, wherein the audio codes are read from audio portions of the media;

obtain semantic audio signature data from the device, the semantic audio signature data being a measure of at least one of a temporal feature, a spectral feature, a harmonic feature, or a rhythmic feature relating to a second characteristic of the media;

associate the semantic audio signature data to the audio codes; and

process the associated semantic audio signature data and audio codes to determine a change in the second characteristic relative to the first characteristic.

20. The apparatus of claim 19, wherein:

the temporal feature includes at least one of an amplitude, a power, or a zero crossing of at least some of the audio of the media,

the spectral feature includes at least one of a spectral centroid, a spectral rolloff, a spectral flux, a spectral flatness measure, a spectral crest factor, Mel-frequency cepstral coefficients, Daubechies wavelet coefficients, a spectral dissonance, a spectral irregularity, or a spectral inharmonicity of at least some of the audio of the media,

the harmonic feature includes at least one of a pitch, a tonality, a pitch class profile, harmonic changes, a main pitch class, an octave range of dominant pitch, a main tonal interval relation, or an overall pitch strength of at least some of the audio of the media, and

the rhythmic feature includes at least one of a rhythmic structure, a beat period, a rhythmic fluctuation or an average tempo for at least some of the audio of the media.

21. The apparatus of claim 19, wherein the first characteristic includes one of a media identification, a media distributor identification, or a media broadcaster identification.

22. The apparatus of claim 19, wherein the second characteristic includes one of a genre, an instrumentation, a style, acoustical dynamics, or an emotive descriptor.

23. The apparatus of claim 19, wherein the audio codes are formed by transforming at least some of the media from a time domain to a frequency domain.

24. The apparatus of claim 19, wherein the semantic audio signature data is formed by transforming at least some of the media from a time domain to a frequency domain.

25. A tangible computer readable storage device comprising instructions, that when executed, cause a computing device to at least:

obtain audio codes from a device, the audio codes representing a first characteristic of media;

obtain semantic audio signature data from the device, the semantic audio signature data being a measure of at least one of a temporal feature, a spectral feature, a harmonic feature, or a rhythmic feature relating to a second characteristic of the media;

associate the semantic audio signature data to the audio codes; and

process the associated semantic audio signature data and audio codes to determine a change in the second characteristic relative to the first characteristic.

26. The tangible computer readable storage device of claim 25, wherein:

the temporal feature includes at least one of an amplitude, a power, or a zero crossing of at least some audio of the media,

the spectral feature includes at least one of a spectral centroid, a spectral rolloff, a spectral flux, a spectral flatness measure, a spectral crest factor, Mel-frequency cepstral coefficients, Daubechies wavelet coefficients, a spectral dissonance, a spectral irregularity, or a spectral inharmonicity of at least some of the audio of the media,

the harmonic feature includes at least one of a pitch, a tonality, a pitch class profile, harmonic changes, a main pitch class, an octave range of dominant pitch, a main tonal interval relation, or an overall pitch strength of at least some of the audio of the media, and

the rhythmic feature includes at least one of a rhythmic structure, a beat period, a rhythmic fluctuation or an average tempo for at least some of the audio of the media.

27. The tangible computer readable storage device of claim 25, wherein the second characteristic includes one of a genre, an instrumentation, a style, acoustical dynamics, or an emotive descriptor.

28. The tangible computer readable storage device of claim 25, wherein the audio codes are formed by transforming at least some of the media from a time domain to a frequency domain.

29. The tangible computer readable storage device of claim 25, wherein the semantic audio signature data is formed by transforming at least some of the media from a time domain to a frequency domain.

30. The tangible computer readable storage device of claim 25, wherein the first characteristic includes one of a media identification, a media distributor identification, or a media broadcaster identification.