System and method for audio hot spotting

• US 7,617,188 B2
• Filed: 03/24/2005
• Issued: 11/10/2009
• Est. Priority Date: 03/24/2005
• Status: Active Grant

• Current Assignee: Mitre Corporation
• Original Assignee: Mitre Corporation
• Inventors: Fish, Randall K., Greiff, Warren R., Hu, Qian, Goodman, Fred J., Boykin, Stanley M.
• Primary Examiner(s): LEWIS, CHERYL RENEA
• Application Number:
  US11/087,703
• Publication Number:
  US 20060217966A1
• Time in Patent Office: 1,692 Days
• Field of Search: 707 1- 5, 707/104.1, 707/100, 707/101, 345/728
• US Class Current: 1/1
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/685 using automatically derived transcript of audio data, e.g. lyrics speech recognition G10L15/00
• CPC Class Codes: G10L 25/00 Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 muting semiconductor-based amplifiers when some special characteristics of a signal are sensed by a speech detector, e.g. sensing when no signal is present, H03G3/34
• CPC Class Codes: Y10S 707/913 Multimedia
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

Audio hot spotting is accomplished by specifying query criterion to include a non-lexical audio cue. The non-lexical audio cue can be, e.g., speech rate, laughter, applause, vocal effort, speaker change or any combination thereof. The query criterion is retrieved from an audio portion of a file. A segment of the file containing the query criterion can be provided to a user. The duration of the provided segment can be specified by the user along with the files to be searched. A list of detections of the query criterion within the file can also be provided to the user. Searches can be refined by the query criterion additionally including a lexical audio-cue. A keyword index of topic terms contained in the file can also be provided to the user.

Claims

1. A method of searching for audio features within multimedia files, comprising:

specifying one or more classes of audio sounds, each of the classes comprising acoustic characteristics of a range of sounds;

searching an audio track of a multimedia file to detect occurrences of the classes using statistical attributes respectively associated with the classes, wherein the range of sounds of a given class satisfy statistical attributes of the given class, and wherein the searching step is effective irrespective of whether any occurrence of the audio track of the multimedia file was used to generate the statistical attributes of the classes;

retrieving at least a segment of the searched multimedia file, if the segment contains at least one occurrence of the classes specified in the specifying step; and

providing, to a user, the retrieved segment of the multimedia file,

wherein the user examines the retrieved segment of the multimedia file to determine at least one of a content of the file, a context of the file, and a relevance of the file.

2. A method of searching for audio features within multimedia files, comprising:

specifying a query criterion that comprises one or more classes of audio sounds, each of the classes comprising acoustic characteristics of a range of sounds;

querying an audio track of a multimedia file to detect occurrences of the classes using statistical attributes respectively associated with the classes, wherein the range of sounds of a given class satisfy statistical attributes of the given class, and wherein the searching step is effective irrespective of whether any occurrence of the audio track of the multimedia file was used to generate the statistical attributes of the classes;

retrieving at least a segment of the searched multimedia file, if the segment contains at least one occurrence of the classes specified in the specifying step; and

providing, to a user, the retrieved segment of the multimedia file, wherein the user examines the retrieved segment of the file to determine at least one of a content of the file, a context of the file, and a relevance of the file.

3. The method of claim 2, wherein the classes include one of speech rate, laughter, applause, vocal effort, and speaker change.

4. The method of claim 2, wherein the specifying step further comprises specifying a duration of the retrieved segment.

5. The method of claim 2, further comprising providing, to the user, a list of detections of the query criterion within the file.

6. The method of claim 2, wherein the specifying step comprises specifying a query criterion comprising a lexical audio feature.

7. The method of claim 6, wherein the specifying step further comprises specifying a lexical audio feature comprising one of a user-specified phoneme string, a user-specified keyword, and a user-specified keyphrase.

8. The method of claim 6, further comprising combining the classes and the lexical audio feature using Boolean operators.

9. The method of claim 6, wherein the querying step comprises querying an audio track of the file to detect a phonetically expanded lexical audio feature.

10. The method of claim 6, wherein the querying step comprises querying an audio track of the file to detect a semantically expanded lexical audio feature.

11. The method of claim 6, wherein the querying step comprises querying an audio track of the file to detect a morphologically expanded lexical audio feature.

12. The method of claim 6, wherein the querying step comprises querying an audio track of the file to detect a translingually expanded lexical audio feature.

13. The method of claim 2, wherein the multimedia file is an audio file.

14. The method of claim 2, wherein the multimedia file is a video file.

15. The method of claim 2, wherein the multimedia file is any of WAV, AVI, MPEG, MPA, MP3 and MP4 file.

16. The method of claim 2, further comprising providing the user with a list of topic terms detected in the audio track of the file.

17. A system for searching for audio features within multimedia files, comprising:

a user interface for specifying a query criterion comprising one or more classes of audio sounds, each of the classes comprising acoustic characteristics of a range of sounds; and

an audio feature extractor that (i) queries an audio track of a multimedia file to detect occurrences of the classes using statistical attributes respectively associated with the classes, wherein the range of sounds of a given class satisfy statistical attributes of the given class, and wherein the searching step is effective irrespective of whether any occurrence of the audio track of the multimedia file was used to generate the statistical attributes of the classes;

(ii) retrieves at least a segment of the searched multimedia file, if the segment contains at least one occurrence of the classes specified in the specifying step;

(iii) provides to a user, the retrieved segment of the multimedia file,

wherein the user examines the retrieved segment of the file to determine at least one of a content of the file, a context of the file, and a relevance of the file.

18. The system of claim 17, wherein the classes include one of speech rate, laughter, applause, vocal effort, and speaker change.

19. The system of claim 17, wherein the query criterion further comprises a segment duration.

20. The system of claim 17, wherein the audio feature extractor provides, to the user, a list of detections of the query criterion within the file.

21. The system of claim 17, wherein the audio feature extractor further provides a cross-file detection of the query criterion.

22. The system of claim 17, wherein the query criterion further comprises a lexical audio feature.

23. The system of claim 22, wherein the lexical audio feature is one of a user-specified phoneme string, a user-specified keyword, and a user-specified keyphrase.

24. The system of claim 22, wherein the query criterion further comprises a combination of the classes and the lexical audio feature using Boolean operators.

25. The system of claim 22, wherein the lexical audio feature is phonetically expanded.

26. The system of claim 22, wherein the lexical audio feature is semantically expanded.

27. The system of claim 22, wherein the lexical audio feature is morphologically expanded.

28. The system of claim 22, wherein the lexical audio feature is translingually expanded.

29. The system of claim 17, wherein the multimedia file is an audio file.

30. The system of claim 17, wherein the multimedia file is a video file.

31. The system of claim 17, wherein the multimedia file is any of WAV, AVI, MPEG, MPA, MP3 and MP4 file.

32. The system of claim 17, wherein the audio feature extractor further provides the user with a list of topic terms detected in the audio track of the file.

33. A system for searching for audio features within multimedia files, comprising:

user interface module for specifying a query criterion that comprises one or more classes of audio sounds, each of the classes comprising acoustic characteristics of a range of sounds; and

audio feature extracting module for,

(i) querying an audio track of a multimedia file to detect occurrences of the classes using statistical attributes respectively associated with the classes, wherein the range of sounds of a given class satisfy statistical attributes of the given class, and wherein the searching step is effective irrespective of whether any occurrence of the audio track of the multimedia file was used to generate the statistical attributes of the classes;

(ii) retrieving at least a segment of the searched multimedia file, if the segment contains at least one occurrence of the classes specified in the specifying step;

(iii) providing to a user, the retrieved segment of the multimedia file,

wherein the user examines the retrieved segment of the file to determine at least one of a content of the file, a context of the file, and a relevance of the file.

34. The system of claim 33, wherein the classes include one of speech rate, laughter, applause, vocal effort, and speaker change.

35. The system of claim 33, further comprising audio feature extracting module for providing, to the user, a list of detections of the query criterion within the file.

36. The system of claim 33, further comprising user interface module for specifying the query criterion to further comprise a lexical audio feature.

37. The system of claim 36, wherein the lexical audio feature is one of a user-specified phoneme, a user-specified keyword, and a user-specified keyphrase.