Audio decoding with supplemental semantic audio recognition and report generation

• US 9,158,760 B2
• Filed: 12/21/2012
• Issued: 10/13/2015
• Est. Priority Date: 12/21/2012
• Status: Expired due to Fees

• Current Assignee: The Nielsen Company LLC (MOOD Media Corporation)
• Original Assignee: The Nielsen Company LLC (MOOD Media Corporation)
• Inventors: Neuhauser, Alan, Stavropoulos, John
• Primary Examiner(s): GUERRA-ERAZO, EDGAR X
• Application Number:
  US13/725,021
• Publication Number:
  US 20140180675A1
• Time in Patent Office: 1,026 Days
• Field of Search: 704/243, 704/246, 704/250
• US Class Current: 1/1
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G10H 1/0008 Associated control or indicating means
• CPC Class Codes: G10H 2210/041 based on mfcc [mel -frequency spectral coefficients]
• CPC Class Codes: G10H 2210/066 for pitch analysis as part of wider processing for musical purposes, e.g. transcription, musical performance evaluation; Pitch recognition, e.g. in polyphonic sounds; Estimation or use of missing fundamental
• CPC Class Codes: G10H 2210/071 for rhythm pattern analysis or rhythm style recognition rhythm pattern per se G10H2210/341
• CPC Class Codes: G10H 2210/076 for extraction of timing, tempo; Beat detection tempo display G10H2220/081; tempo control G10H2210/375
• CPC Class Codes: G10H 2210/081 for automatic key or tonality recognition, e.g. using musical rules or a knowledge base
• CPC Class Codes: G10H 2240/041 File watermark, i.e. embedding a hidden code in an electrophonic musical instrument file or stream for identification or authentification purposes audio watermarking G10L19/018
• CPC Class Codes: G10H 2240/081 Genre classification, i.e. descriptive metadata for classification or selection of musical pieces according to style analysis genre G10H2210/036
• CPC Class Codes: G10H 2240/085 Mood, i.e. generation, detection or selection of a particular emotional content or atmosphere in a musical piece
• CPC Class Codes: G10H 2240/095 Identification code, e.g. ISWC for musical works; Identification dataset
• CPC Class Codes: G10H 2250/031 Spectrum envelope processing
• CPC Class Codes: G10L 19/018 Audio watermarking, i.e. embedding inaudible data in the audio signal
• CPC Class Codes: G10L 25/03 characterised by the type of extracted parameters
• CPC Class Codes: G10L 25/09 the extracted parameters being zero crossing rates
• CPC Class Codes: G10L 25/18 the extracted parameters being spectral information of each sub-band
• CPC Class Codes: G10L 25/21 the extracted parameters being power information
• CPC Class Codes: G10L 25/24 the extracted parameters being the cepstrum
• CPC Class Codes: G10L 25/51 for comparison or discrimination

Abstract

System, apparatus and method for determining semantic information from audio, where incoming audio is sampled and processed to extract audio features, including temporal, spectral, harmonic and rhythmic features. The extracted audio features are compared to stored audio templates that include ranges and/or values for certain features and are tagged for specific ranges and/or values. The semantic information may be associated with audio codes to determine changing characteristics of identified media during a time period.

Claims

1. A processor-based method for producing supplemental information for media containing embedded audio codes, wherein the codes are read from an audio portion of the media, the method comprising the steps of:

receiving the audio codes at an input from a data network, said audio codes being received from a device during a first time period, said audio codes representing a first characteristic of the audio portion;

receiving semantic audio signature data at the input from the data network, said semantic audio signature data being received from the device for the first time period, wherein the semantic audio signature comprises at least one of temporal, spectral, harmonic and rhythmic features relating to a second characteristic of the media content; and

successively associating the semantic audio signature data to the audio codes in a processor for the first time period.

2. The method of claim 1, wherein the semantic audio signature data temporal features comprise at least one of amplitude, power and zero crossing of at least some of the media content audio.

3. The method of claim 1, wherein the semantic audio signature data spectral features comprise at least one of spectral centroid, spectral rolloff, spectral flux, spectral flatness measure, spectral crest factor, Mel-frequency cepstral coefficients, Daubechies wavelet coefficients, sopectral dissonance, spectral irregularity and spectral inharmonicity of at least some of the media content audio.

4. The method of claim 1, wherein the semantic audio signature data harmonic features comprise at least one of pitch, tonality, pitch class profile, harmonic changes, main pitch class, octave range of dominant pitch, main tonal interval relation and overall pitch strength of at least some of the media content audio.

5. The method of claim 1, wherein the semantic audio signature data rhythmic features comprise at least one of rhythmic structure, beat period, rhythmic fluctuation and average tempo for at least some of the media content audio.

6. The method of claim 1, wherein the audio codes are formed by transforming at least some of the audio signals from a time domain to a frequency domain.

7. The method of claim 1, wherein the semantic audio signature data is formed by transforming at least some of the audio signals from a time domain to a frequency domain.

8. A system for producing supplemental information for media containing embedded audio codes, wherein the codes are read from an audio portion of the media, comprising:

an input configured to receive the audio codes from a data network, said audio codes being received from a device during a first time period, wherein the audio codes representing a first characteristic of the audio portion;

said input being further configured to receive semantic audio signature data from the data network, said semantic audio signature data being received from the device for the first time period, wherein the semantic audio signature comprises at least one of temporal, spectral, harmonic and rhythmic features relating to a second characteristic of the media content; and

a processor, operatively coupled to the input, said processor being configured to successively associate the semantic audio signature data to the audio codes in a processor for the first time period.

9. The system of claim 8, wherein the semantic audio signature data temporal features comprise at least one of amplitude, power and zero crossing of at least some of the media content audio.

10. The system of claim 8, wherein the semantic audio signature data spectral features comprise at least one of spectral centroid, spectral rolloff, spectral flux, spectral flatness measure, spectral crest factor, Mel-frequency cepstral coefficients, Daubechies wavelet coefficients, sopectral dissonance, spectral irregularity and spectral inharmonicity of at least some of the media content audio.

11. The system of claim 8, wherein the semantic audio signature data harmonic features comprise at least one of pitch, tonality, pitch class profile, harmonic changes, main pitch class, octave range of dominant pitch, main tonal interval relation and overall pitch strength of at least some of the media content audio.

12. The system of claim 8, wherein the semantic audio signature data rhythmic features comprise at least one of rhythmic structure, beat period, rhythmic fluctuation and average tempo for at least some of the media content audio.

13. The system of claim 8, wherein the audio codes are formed by transforming at least some of the audio signals from a time domain to a frequency domain.

14. The system of claim 8, wherein the semantic audio signature data is formed by transforming at least some of the audio signals from a time domain to a frequency domain.

15. A processor-based method for producing supplemental information for media containing embedded audio codes, wherein the codes are read from an audio portion of the media, the method comprising the steps of:

receiving the audio codes at an input from a data network, said audio codes being received from a device during a first time period, wherein the audio codes represent a first characteristic of audio portion;

receiving semantic audio signature data at the input from the data network, said semantic audio signature data being received from the device for the first time period, wherein the semantic audio signature comprises at least one of temporal, spectral, harmonic and rhythmic features relating to a second characteristic of the media content;

successively associating the semantic audio signature data to the audio codes in a processor for the first time period; and

processing the associated semantic audio signature data and audio codes data to determine changing second characteristics in relation to the first characteristic.

16. The method of claim 15, wherein the second characteristic comprises at least one of:

the temporal features comprising at least one of amplitude, power and zero crossing of at least some of the media content,

the spectral features comprising at least one of spectral centroid, spectral rolloff, spectral flux, spectral flatness measure, spectral crest factor, Mel-frequency cepstral coefficients, Daubechies wavelet coefficients, sopectral dissonance, spectral irregularity and spectral inharmonicity of at least some of the media content,

the harmonic features comprising at least one of pitch, tonality, pitch class profile, harmonic changes, main pitch class, octave range of dominant pitch, main tonal interval relation and overall pitch strength of at least some of the media content,

the rhythmic features comprising at least one of rhythmic structure, beat period, rhythmic fluctuation and average tempo for at least some of the audio signals.

17. The method of claim 15, wherein the first characteristic data comprises one of media content identification, media content distributor identification, and media content broadcaster identification.

18. The method of claim 15, wherein the second characteristic data comprises one of genre, instrumentation, style, acoustical dynamics and emotive descriptors.

19. The method of claim 15, wherein the audio codes are formed by transforming at least some of the media content from a time domain to a frequency domain.

20. The method of claim 15, wherein the semantic audio signature data is formed by transforming at least some of the media content from a time domain to a frequency domain.