Apparatus and method for automatic extraction of important events in audio signals

• US 20050102135A1
• Filed: 11/10/2004
• Published: 05/12/2005
• Est. Priority Date: 11/12/2003
• Status: Active Grant

• Current Assignee: Sony Deutschland GmbH (Sony Group Corp.)
• Original Assignee: Sony Deutschland GmbH (Sony Group Corp.)
• Inventors: Kompe, Ralf, Kemp, Thomas, Marasek, Krzysztof, Lam, Yin Hay, Tato, Raquel, Goronzy, Silke
• Granted Patent: US 8,635,065 B2
• Time in Patent Office: Days
• US Class Current: 704/213
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
• CPC Class Codes: G10L 25/00 Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 muting semiconductor-based amplifiers when some special characteristics of a signal are sensed by a speech detector, e.g. sensing when no signal is present, H03G3/34

Abstract

The present invention discloses an apparatus for automatic extraction of important events in audio signals comprising: signal input means for supplying audio signals; audio signal fragmenting means for partitioning audio signals supplied by the signal input means into audio fragments of a predetermined length and for allocating a sequence of one or more audio fragments to a respective audio window; feature extracting means for analysing acoustic characteristics of the audio signals comprised in the audio fragments and for analysing acoustic characteristics of the audio signals comprised in the audio windows; and important event extraction means for extracting important events in audio signals supplied by the audio signal fragmenting means based on predetermined important event classifying rules depending on acoustic characteristics of the audio signals comprised in the audio fragments and on acoustic characteristics of the audio signals comprised in the audio windows, wherein each important event extracted by the important event extraction means comprises a discrete sequence of cohesive audio fragments corresponding to an important event included in the audio signals.

Claims

1. Apparatus for automatic extraction of important events in audio signals comprising:

signal input means for supplying audio signals;

audio signal fragmenting means for partitioning audio signals supplied by the signal input means into audio fragments of a predetermined length and for allocating a sequence of one or more audio fragments to a respective audio window;

feature extracting means for analysing acoustic characteristics of the audio signals comprised in the audio fragments and for analysing acoustic characteristics of the audio signals comprised in the audio windows; and

important event extraction means for extracting important events in audio signals supplied by the audio signal fragmenting means based on predetermined important event classifying rules depending on acoustic characteristics of the audio signals comprised in the audio fragments and on acoustic characteristics of the audio signals comprised in the audio windows, wherein each important event extracted by the important event extraction means comprises a discrete sequence of cohesive audio fragments corresponding to an important event included in the audio signals.

2. Apparatus for automatic extraction of important events in audio signals according to claim 1,

characterised in that

the acoustic characteristics analysed in the audio signals comprised in the audio fragments include zero crossing rate and/or energy/loudness and/or sub-band energy rate and/or mel-cepstral frequency components and/or frequency centroid and/or bandwidth.

3. Apparatus for automatic extraction of important events in audio signals according to claim 1,

characterised in that

the acoustic characteristics analysed in the audio signals comprised in the audio windows include volume standard deviation and/or volume dynamic range and/or high zero crossing rate ratio and/or low short-term energy ratio and/or spectral flux.

4. Apparatus for automatic extraction of important events in audio signals according to claim 1,

characterised in that the apparatus for automatic extraction of important events in audio signals further comprises:

a classifier database comprising important event classifying rules, wherein an important event classifying rule is provided for each kind of programme included in the audio signal; and

classifier selection means for detecting the kind of programme the audio signals actually belong to and for choosing a suitable important event classifying rule from the important event classifying rules stored in the classifier database;

wherein the important event extraction means extracts important events in audio signals based on the respective important event classifying rule chosen by the classifier selection means.

5. Apparatus for automatic extraction of important events in audio signals according to claim 4,

characterised in that

the kind of programme the respective audio signals actually belong to is identified by the classifier selection means by using an information signal comprised in the raw signals.

6. Apparatus for automatic extraction of important events in audio signals according to claim 4,

characterised in that

the important event classifying rules are provided as hidden Markov models and/or Neuronal Networks and/or Gaussian Mixture Models and/or decision trees.

7. Apparatus for automatic extraction of important events in audio signals according to claim 1,

characterised in that

the feature extraction means detects a speech signal comprised in the audio signal of each audio window, recognises emotion in said speech signal based on acoustic characteristics of an audio signal comprised in said speech signal and outputs a corresponding emotion signal to the important event extraction means;

wherein the important event extraction means uses the emotion signal provided by the feature extraction means for extracting important events in audio signals.

8. Apparatus for automatic extraction of important events in audio signals according to claim 7,

characterised in that

the emotion signal output by the feature extraction means is normalised according to predetermined emotion classes.

9. Apparatus for automatic extraction of important events in audio signals according to claim 8,

characterised in that

the emotion classes comprise a class for each excited, disappointed, angry, happy and bored.

10. Apparatus for automatic extraction of important events in audio signals according to claim 7,

characterised in that

the acoustic characteristics used for emotion recognition comprise prosodic features including pitch and/or pitch range and/or intonation attitude and/or loudness and/or speaking rate and/or phone duration.

11. Apparatus for automatic extraction of important events in audio signals according to claim 7,

characterised in that

the acoustic characteristics used for emotion recognition comprise quality features including phonation type and/or articulatory manner and/or voice timbre and/or spectral tilt and/or difference of amplitude between harmonics and formants and/or formants bandwidth and/or jitter and/or harmonic-to-noise ratio.

12. Apparatus for automatic extraction of important events in audio signals according to claim 1,

characterised in that the apparatus for automatic extraction of important events in audio signals further comprises:

signal output means for generating an output file;

wherein the output file contains the raw signal supplied to the signal input means and an information signal comprising information regarding to the important events included in the raw signal.

13. Method for automatic extraction of important events in audio signals comprising the following steps:

partitioning audio signals into audio fragments of a predetermined length;

allocating a sequence of one or more audio fragments to a respective audio window of a plurality of audio windows;

analysing acoustic characteristics of the audio signals comprised in the audio fragments;

analysing acoustic characteristics of the audio signals comprised in the audio windows; and

extracting important events in audio signals based on predetermined important event classifying rules by using acoustic characteristics of the audio signals comprised in the audio fragments and acoustic characteristics of the audio signals comprised in the audio windows, wherein each important event comprises a discrete sequence of cohesive audio fragments corresponding to an important event included in the audio signals.

14. Method for automatic extraction of important events in audio signals according to claim 13,

characterised in that

the step of analysing acoustic characteristics in the audio signals comprised in the audio fragments includes analysis of zero crossing rate and/or energy/loudness and/or sub-band energy rate and/or mel-cepstral frequency components and/or frequency centroid and/or bandwidth.

15. Method for automatic extraction of important events in audio signals according to claim 13,

characterised in that

the step of analysing acoustic characteristics in the audio signals comprised in the audio windows includes analysis of volume standard deviation and/or volume dynamic range and/or high zero crossing rate ratio and/or low short-term energy ratio and/or spectral flux.

16. Method for automatic extraction of important events in audio signals according to claim 13,

characterised in that the method for automatic extraction of important events in audio signals further comprises the steps of:

providing a classifier database comprising important event classifying rules, wherein an important event classifying rule is provided for each kind of programme included in the audio signal;

detecting the kind of programme the audio signals actually belong to; and

choosing a suitable important event classifying rule from the important event classifying rules stored in the classifier database;

wherein the step of extracting important events in audio signals is performed based on said suitable important event classifying rule.

17. Method for automatic extraction of important events in audio signals according to claim 13,

characterised in that the method further comprises the steps of

detecting a speech signal comprised in the audio signals of each audio window,

recognising an emotion in said speech signal based on acoustic characteristics of an audio signal comprised in said speech signal;

and

outputting a corresponding emotion signal;

wherein the step of extracting important events in audio signals is performed by using said emotion signal.

18. Method for automatic extraction of important events in audio signals according to claim 17,

characterised in that the method further comprises the step of

normalising the emotion signal according to predetermined emotion classes.

19. Method for automatic extraction of important events in audio signals according to claim 17,

characterised in that

the acoustic characteristics used in the step of recognising an emotion in said speech signal comprise prosodic features including pitch and/or pitch range and/or intonation attitude and/or loudness and/or speaking rate and/or phone duration.

20. Method for automatic extraction of important events in audio signals according to claim 17,

characterised in that

the acoustic characteristics used in the step of recognising an emotion in said speech signal comprise quality features including phonation type and/or articulatory manner and/or voice timbre and/or spectral tilt and/or difference of amplitude between harmonics and formants and/or formants bandwidth and/or jitter and/or harmonic-to-noise ratio.

26. Software product comprising a series of state elements that are adapted to be processed by a data processing means of a terminal such, that a method according to claim 13 may be executed thereon.

21. Method for automatic generation of important event classifying rules for extraction of important events in audio signals comprising the following steps:

partitioning audio signals into audio fragments of a predetermined length;

allocating a temporal sequence of one or more audio fragments to a respective audio window of a plurality of audio windows;

tagging each audio window with a predefined important event label;

extracting fragment characteristics for the audio fragments of a given window by analysing acoustic characteristics of the audio signals comprised in the audio fragments of a given window;

extracting window characteristics for the respective audio window by analysing acoustic characteristics of the audio signal comprised in the respective audio window; and

generating an important event classifying rule for the respective important event label by using the fragment characteristics and the window characteristics.

22. Method for automatic generation of important event classifying rules according to claim 21,

characterised in that

the step of extracting fragment characteristics comprises calculation of a zero cross rate and/or volume and/or sub-band energy rate and/or mel-cepstral frequency components and/or frequency centroid and/or bandwidth of the audio signal comprised in the respective video fragment.

23. Method for automatic generation of important event classifying rules according to claim 21,

characterised in that

the step of extracting window characteristics comprises calculation of a volume standard deviation and/or a volume dynamic range and/or a high zero crossing rate ratio and/or a low short-term energy ratio and/or spectral flux of the audio signal comprised in the respective video window.

24. Method for automatic generation of important event classifying rules according to claim 21,

characterised in that the method further comprises the steps of:

detecting a speech signal in the audio signal comprised in the respective audio window; and

recognising emotion in the speech signal based on prosodic features and/or quality features of said speech signal;

wherein the step of generating an important event classifying rule for the respective important event label is performed by using said recognised emotion.

25. Method for automatic generation of important event classifying rules according to claim 24,

characterised in that

the prosodic features include pitch and/or pitch range and/or intonation attitude and/or loudness and/or speaking rate and/or phone duration;

and

the quality features include phonation type and/or articulatory manner and/or voice timbre and/or spectral tilt and/or difference of amplitude between harmonics and formants and/or formants bandwidth and/or jitter and/or harmonic-to-noise ratio.