Method and article of manufacture for content-based analysis, storage, retrieval, and segmentation of audio information

• US 5,918,223 A
• Filed: 07/21/1997
• Issued: 06/29/1999
• Est. Priority Date: 07/22/1996
• Status: Expired due to Term

• Current Assignee: Audible Magic Corporation
• Original Assignee: Muscle Fish
• Inventors: Wold, Erling H., Keislar, Douglas F., Wheaton, James A., Blum, Thomas L.
• Primary Examiner(s): Amsbury, Wayne
• Assistant Examiner(s): LEWIS, CHERYL RENEA
• Application Number:
  US08/897,662
• Time in Patent Office: 708 Days
• Field of Search: 707/1, 707/104, 345/302
• US Class Current: 1/1
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/634 Query by example, e.g. query by humming
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content
• CPC Class Codes: G06F 2218/08 Feature extraction
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

A system that performs analysis and comparison of audio data files based upon the content of the data files is presented. The analysis of the audio data produces a set of numeric values (a feature vector) that can be used to classify and rank the similarity between individual audio files typically stored in a multimedia database or on the World Wide Web. The analysis also facilitates the description of user-defined classes of audio files, based on an analysis of a set of audio files that are members of a user-defined class. The system can find sounds within a longer sound, allowing an audio recording to be automatically segmented into a series of shorter audio segments.

Claims

1. A method for analyzing sound files for one or more of storing, comparing, and retrieving audio data, comprising the steps of:

(a) measuring a plurality of acoustical features of a sound file chosen from the group consisting of at least one of loudness, pitch, brightness, bandwidth, and MFCC coefficients thereof;

(b) computing measurements chosen from the group consisting of mean, standard deviation, autocorrelation and first derivative thereof, of the acoustical features of the audio files, forming a vector of the feature and measurements data, and storing the computed measurements and vector in a feature file with a linkage to the sound file; and

(c) grouping the feature files based on at least one of (i) similar measurements for the feature files, and (ii) distance between the vector for a sound file and a vector for a reference, to facilitate rapid classification, storage, and/or retrieval of sound files based on a predefined search criteria.

2. The method as recited in claim 1, wherein the predefined search criteria is based on the feature file of a particular audio file.

3. The method as recited in claim 1, wherein the predefined search criteria are similar to a user created sound.

4. The method as recited in claim 1, wherein the predefined search criteria are characteristics of a predefined set of example audio files.

5. The method as recited in claim 1, wherein the acoustical feature is a set of Mel-frequency cepstral coefficients (MFCCs).

6. The method as recited in claim 1, wherein the statistical measurement is the average value of the first derivative of each acoustical feature.

7. The method as recited in claim 1, wherein the statistical measurement is the standard deviation of the first derivative of each acoustical feature.

8. A computer program embodied on a computer-readable medium for analyzing audio files, comprising:

(a) a code segment for measuring a plurality of acoustical features of a sound file chosen from the group consisting of at least one of loudness, pitch, brightness, bandwidth, and MFCC coefficients thereof;

(b) a code segment for computing measurements chosen from the group consisting of mean, standard deviation, autocorrelation and first derivative thereof, of the acoustical features of the audio files, forming a vector of the feature and measurements data, and storing the computed measurements and vector in a feature file with a linkage to the audio file; and

(c) a code segment for grouping the feature files based on at least one of (i) similar measurements for the feature files, and (ii) distance between the vector for a sound file and a vector for a reference, to facilitate rapid classification, storage, and/or retrieval of sound files based on a predefined search criteria.

9. The computer program embodied on a computer-readable medium for analyzing sound files as recited in claim 8, wherein the predefined search criteria is based on the feature file of a particular audio file.

10. The computer program embodied on a computer-readable medium for analyzing sound files as recited in claim 8, wherein the predefined search criteria is similarity to a user-created created sound.

11. The computer program embodied on a computer-readable medium for analyzing sound files as recited in claim 8, wherein the predefined search criteria are characteristics of a predefined set of example audio files.

12. A content based comparison method for segmenting sound files, comprising the steps of:

(a) measuring a plurality of acoustical features of an sound file chosen from the group consisting of at least one of loudness, pitch, brightness, bandwidth, and MFCC coefficients thereof;

(b) computing measurements chosen from the group consisting of mean, standard deviation, autocorrelation and first derivative thereof, of the acoustical features of the audio files, forming a vector of the feature and measurements data, and storing the computed measurements and vector in a feature file with a linkage to the audio file; and

(c) segmenting the feature files based on at least one (i) similar measurements for the feature files, and (ii) distance between the vector for a sound file and a vector for a reference, to facilitate rapid classification, storage, and/or retrieval of sound files based on a predefined search criteria.

13. The method as recited in claim 12, wherein the content-based comparison compares a user-supplied sound to successive regions of the sound that is to be segmented to measure the similarity between the measurements of the two sounds'"'"' acoustical features.

14. The method as recited in claim 12, wherein the content-based comparison compares a user-supplied sound to successive regions of the sound that is to be segmented, measuring the frame-by-frame similarity between the two sounds'"'"' acoustical features.

15. The method as recited in claim 12, wherein the content-based comparison compares adjacent regions of the sound that is to be segmented, measuring the similarity between the regions'"'"' statistical measurements.

16. A computer program embodied on a computer-readable medium for segmenting sound files by using content-based statistical comparison of a plurality of acoustical features of a sound file chosen from the group consisting of at least one of loudness, pitch, brightness, bandwidth, and MFCC coefficients thereof to determine the segment boundaries.

17. A computer program according to claim 16 in which the sound files include a segment of digital audio on a storage device.

18. The computer program embodied on a computer-readable medium for segmenting sound files as recited in claim 16, wherein the content-based comparison compares a user-supplied sound to successive regions of the sound that is to be segmented, measuring the similarity between the statistical measurements of the two sounds'"'"' acoustical features.

19. The computer program embodied on a computer-readable medium for segmenting sound files as recited in claim 16, wherein the content-based comparison compares a user-supplied sound to successive regions of the sound that is to be segmented, measuring the frame-by-frame similarity between the two sounds'"'"' acoustical features.

20. The computer program embodied on a computer-readable medium for segmenting audio files as recited in claim 16, wherein the content-based comparison compares adjacent regions of the sound that is to be segmented, measuring the similarity between the regions'"'"' statistical measurements.