System and method for recognizing audio pieces via audio fingerprinting

• US 7,487,180 B2
• Filed: 01/31/2006
• Issued: 02/03/2009
• Est. Priority Date: 09/23/2003
• Status: Active Grant

• Current Assignee: Gracenote, Inc. (RR Donnelley & Sons Company)
• Original Assignee: Musicip Corporation (AmpliFIND Music Services, Inc.)
• Inventors: Hicken, Wendell T., Holm, Frode
• Primary Examiner(s): To; Baoquoc N
• Application Number:
  US11/345,548
• Publication Number:
  US 20060190450A1
• Time in Patent Office: 1,099 Days
• Field of Search: 707/3, 707/6, 707/9, 707/10, 707/104.1, 705/44, 713/176, 702/185
• US Class Current: 1/1
• CPC Class Codes: G06F 16/634 Query by example, e.g. query by humming
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: Y10S 707/916 Audio
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99939 Privileged access
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

An audio fingerprinting system and method. A server receives an audio fingerprint of a first audio piece, searches a database for the audio fingerprint, retrieves an audio profile vector associated with the audio fingerprint, updates user preference information based on the audio profile vector, and selects a second audio piece based on the user preference information. The audio fingerprint is generated by creating a matrix based on the frequency measurements of the audio piece, and performing a singular value decomposition of the matrix. To expedite the search of the database and to increase matching accuracy, a subset of candidates in the database is identified based on the most prominent musical notes of the audio piece, and the search is limited to the identified subset. One of the attributes of the audio profile vector is a particular audio class. An identifier for the audio class is generated based on an average of audio fingerprints of the audio pieces belonging to the audio class.

Claims

1. An audio recognition method comprising:

receiving an audio fingerprint of a musical piece from a client device;

comparing the received audio fingerprint against a plurality of stored audio fingerprints for a match;

determining if the received audio fingerprint corresponds to a particular one of the stored audio fingerprints;

if the received audio fingerprint corresponds to the particular one of the stored audio fingerprints;

retrieving an audio profile vector stored in association with the particular one of the stored audio fingerprints, the audio profile vector including at least N numerical values quantifying N acoustic attributes of the musical piece, wherein N>

0, and wherein at least one of the acoustic attributes is tempo, and the associated numerical value quantifies the tempo of the musical piece based on an automatic processing of audio signals of the musical piece by a music profiling engine; and

transmitting information stored in association with the retrieved audio profile vector to the client device for doing at least one of generating a music playlist, making music related recommendations, and making other music-related selections; and

if the received audio fingerprint does not correspond to the particular one of the stored audio fingerprints, prompting the client device for generating the audio profile vector.

2. The method of claim 1, wherein the audio profile vector is generated based on an automatic processing of audio signals of the audio piece.

3. The method of claim 1, wherein one of the plurality of acoustic attributes included in the audio profile vector is associated with a particular audio class, and the numerical value indicates a distance of the audio piece to the audio class.

4. The method of claim 3, wherein the audio class is identified based on an audio class fingerprint, the audio class fingerprint being an average of audio fingerprints of audio pieces associated with the audio class.

5. The method of claim 4, wherein the numerical value indicating the distance of the audio piece to the audio class is determined based on a distance calculation of the received audio fingerprint and the audio class fingerprint.

6. The method of claim 1 further comprising:

identifying an index of a subset of the plurality of stored audio fingerprints, the index identifying a plurality of musical notes determined to be most prominent for the audio fingerprints in the subset; and

searching the identified subset for the match.

7. The method of claim 1, wherein if none of the stored audio fingerprints correspond to the received audio fingerprint, invoking the client device to generate the audio profile vector of the audio piece.

8. The method of claim 1, wherein the received and stored audio fingerprints are each represented as a matrix of vectors.

9. The method of claim 8, wherein the received audio fingerprint corresponds to the particular one of the stored audio fingerprints if a distance computation between the matrix representing the received audio fingerprint and the matrix representing the particular one of the stored audio fingerprints results in a single scalar distance value that satisfies a threshold distance.

10. The method of claim 1, wherein the information transmitted to the client device is the retrieved audio profile vector.

11. The method of claim 1 further comprising:

recommending a music item based on the audio profile vector.

12. The method of claim 1 further comprising:

receiving from the client device a user rating with the received audio fingerprint;

modifying user preference information based on the user rating and the retrieved audio profile vector; and

recommending a music item based on the user preference information.

13. The method of claim 1, wherein none of the N numerical values quantifying the N acoustic attributes of the musical piece is determined based on human analysis of the musical piece.

14. An audio recognition method comprising:

receiving an audio fingerprint of an audio piece;

comparing the received audio fingerprint against a plurality of stored audio fingerprints for a match;

identifying the audio piece responsive to a match of the audio fingerprint; and

retrieving information stored in association with the identified audio piece, wherein the audio fingerprint is a representation of matrix V^T generated from a singular value decomposition (SVD) of an N×

M matrix A, the matrix A being built based on frequency measurements of audio signals associated with the audio piece, wherein SVD(A)=USV^T, where U is an N×

M orthogonal matrix, S in an M×

M diagonal matrix, and V^T is a transpose of an M×

M orthogonal matrix.

15. The method of claim 14, wherein rows of the matrix A represent time, and columns of the matrix A represent the frequency measurements.

16. An audio recognition system comprising:

a first data store storing a plurality of audio fingerprints for a plurality of audio pieces;

one or more processors;

one or more memory devices operably coupled to the one or more processors storing program instructions therein, each of the one or more processors being operable to execute one or more of the program instructions, the program instructions including;

receiving an audio fingerprint of a particular musical piece from a client device;

comparing the received audio fingerprint against the plurality of stored audio fingerprints for a match;

determining if the received audio fingerprint corresponds to a particular one of the stored audio fingerprints;

retrieving an audio profile vector stored in association with the particular one of the stored audio fingerprints if the received audio fingerprint corresponds to the particular one of the stored audio fingerprints, the audio profile vector including at least N numerical values quantifying N acoustic attributes of the musical piece, wherein N>

0, and wherein at least one of the acoustic attributes is tempo, and the associated numerical value quantifies the tempo of the musical piece based on an automatic processing of audio signals of the audio piece by a music profiling engine;

transmitting information stored in association with the retrieved audio profile vector to the client device for doing at least one of generating a music playlist, making music related recommendations, and making other music-related selections; and

prompting the client device for generating the audio profile vector if the received audio fingerprint does not correspond to the particular one of the stored audio fingerprints.

17. The system of claim 16 further comprising:

a second data store storing the audio profile vector in association with the particular one of the stored audio fingerprints, the audio profile vector being generated based on an automatic processing of audio signals of the audio piece.

18. The system of claim 17, wherein one of the plurality of acoustic attributes included in the audio profile vector is associated with a particular audio class, and the numerical value indicates a distance of the audio piece to the audio class.

19. The system of claim 18, wherein the audio class is identified based on an audio class fingerprint, the audio class fingerprint being an average of audio fingerprints of audio pieces associated with the audio class.

20. The system of claim 16, wherein the program instructions further include:

identifying an index of a subset of the plurality of stored audio fingerprints based on the audio fingerprint, the index identifying a plurality of musical notes determined to be most prominent for the audio fingerprints in the subset; and

searching the identified subset for the match.

21. An audio recognition system comprising:

a first data store storing a plurality of audio fingerprints for a plurality of audio pieces;

one or more processors;

one or more memory devices operably coupled to the one or more processors storing program instructions therein, each of the one or more processors being operable to execute one or more of the program instructions, the program instructions including;

receiving an audio fingerprint of a particular audio piece;

comparing the received audio fingerprint against the plurality of stored audio fingerprints for a match;

identifying the audio piece responsive to a match of the audio fingerprint; and

retrieving information stored in association with the identified audio piece, wherein the audio fingerprint is a representation of matrix V^T generated from a singular value decomposition (SVD) of an N×

M matrix A, the matrix A being built based on frequency measurements of audio signals associated with the audio piece, wherein SVD(A)=USV^T, where U is an N×

M orthogonal matrix, S in an M×

M diagonal matrix, and V^T is a transpose of an M×

M orthogonal matrix.

22. The system of claim 21, wherein rows of the matrix A represent time, and columns of the matrix A represent the frequency measurements.