System and method for indexing videos based on speaker distinction

• US 20030182118A1
• Filed: 03/25/2002
• Published: 09/25/2003
• Est. Priority Date: 03/25/2002
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Zhang, Tong, Obrador, Pere
• Granted Patent: US 7,184,955 B2
• Time in Patent Office: Days
• US Class Current: 704/246
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques

Abstract

A system and method for indexing multimedia files utilizes audio characteristics of predefined audio content contained in selected multimedia segments of the multimedia files to distinguish the selected multimedia segments. In the exemplary embodiment, the predefined audio content is speech contained in video segments of video files. Furthermore, the audio characteristics are speaker characteristics. The speech-containing video segments are detected by analyzing the audio contents of the video files. The audio contents of the speech-containing video segments are then characterized to distinguish the video segments according to speakers. The indexing of speech-containing video segments based on speakers allows users to selectively access video segments that contain speech from a particular speaker without having to manually search all the speech-containing video segments.

Claims

1. A method for indexing multimedia data comprising:

detecting particular multimedia segments of said multimedia data, said particular multimedia segments containing predefined audio content;

classifying said particular multimedia segments according to audio characteristics of said predefined audio content in said particular multimedia segments; and

generating indices of locations of said particular multimedia segments in said multimedia data, said indices including classification information of said particular multimedia segments according to said audio characteristics.

2. The method of claim 1 wherein said predefined audio content includes speech.

3. The method of claim 2 wherein said audio characteristics of said predefined audio content include speaker characteristics.

4. The method of claim 1 further comprising a step of generating spectral-based feature coefficients for audio portions of said particular multimedia segments.

5. The method of claim 4 wherein said step of generating said spectral-based feature coefficients includes generating mel-frequency cepstral coefficients.

6. The method of claim 4 further comprising a step of modeling said spectral-based feature coefficients of said particular multimedia segments into audio models.

7. The method of claim 6 wherein said step of classifying said particular multimedia segments includes clustering said audio models of said particular multimedia segments.

8. The method of claim 1 wherein said step of detecting said particular multimedia segments includes comparing energy values of audio portions of said multimedia data to an energy threshold.

9. The method of claim 1 wherein said step of detecting said particular multimedia segments includes comparing variance values of zero-crossing rates and amplitude span values of said zero-crossing rates for audio portions of said multimedia data to corresponding thresholds.

10. The method of claim 1 wherein said step of detecting said particular multimedia segments includes checking for harmonic speech relations in peaks of spectrums for audio portions of said multimedia data.

11. A system for indexing multimedia data comprising:

a detector configured to detect particular multimedia segments of said multimedia data, said particular multimedia segments containing predefined audio content;

a module configured to classify said particular multimedia segments according to audio characteristics of said predefined audio content in said particular multimedia segments; and

a processor configured to generate indices of locations of said particular multimedia segments in said multimedia data, said indices including classification information of said particular multimedia segments according to said audio characteristics.

12. The system of claim 11 wherein said predefined audio content includes speech.

13. The system of claim 12 wherein said audio characteristics of said predefined audio content include speaker characteristics.

14. The system of claim 11 wherein said module includes a feature generator configured to generate spectral-based feature coefficients for audio portions of said particular multimedia segments.

15. The system of claim 14 wherein said feature generator is configured to generate mel-frequency cepstral coefficients.

16. The system of claim 14 wherein said module includes a modeling unit configured to model said spectral-based feature coefficients of said particular multimedia segments into audio models.

17. The system of claim 16 wherein said module includes a clustering unit configured to cluster said audio models of said particular multimedia segments to classify said particular multimedia segments.

18. The system of claim 11 wherein said detector includes an energy analyzer configured to compare energies of audio portions of said multimedia data to an energy threshold.

19. The system of claim 11 wherein said detector includes a zero-crossing analyzer configured to compare variance values of zero-crossing rates and amplitude span values of said zero-crossing rates for audio portions of said multimedia data to corresponding thresholds.

20. The system of claim 11 wherein said detector includes a spectrum analyzer configured to check for harmonic speech relations in peaks of spectrums for audio portions of said multimedia data.

21. A method for indexing video data comprising:

detecting video segments of said video data that contain speech content;

associating said video segments based on speaker characteristics of said speech content in said video segments; and

generating indices corresponding to locations of said video segments in said video data, said indices including said speaker characteristics of said speech content in said video segments.

22. The method of claim 21 further comprising a step of generating spectral-based feature coefficients for audio portions of said video segments.

23. The method of claim 22 wherein said step of generating said spectral-based feature coefficients includes generating mel-frequency cepstral coefficients.

24. The method of claim 23 further comprising a step of modeling said spectral-based feature coefficients of said video segments into speaker models.

25. The method of claim 24 wherein said step of associating said video segments based on said speaker characteristics includes clustering said speaker models of said video segments.

26. The method of claim 21 wherein said step of detecting said video segments includes comparing energy values of audio portions of said video data to an energy threshold.

27. The method of claim 21 wherein said step of detecting said video segments includes comparing variance values of zero-crossing rates and amplitude span values of said zero-crossing rates for audio portions of said video data to corresponding thresholds.

28. The method of claim 21 wherein said step of detecting said video segments includes checking for harmonic speech relations in peaks of spectrums for audio portions of said video data.