Audio fingerprinting system and method

• US 20050065976A1
• Filed: 09/23/2003
• Published: 03/24/2005
• Est. Priority Date: 09/23/2003
• Status: Active Grant

• Current Assignee: Gracenote, Inc. (RR Donnelley & Sons Company)
• Original Assignee: Musicip Corporation (AmpliFIND Music Services, Inc.)
• Inventors: Hicken, Wendell T., Holm, Frode
• Granted Patent: US 7,013,301 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/634 Query by example, e.g. query by humming
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: Y10S 707/916 Audio
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99939 Privileged access
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

An audio fingerprinting system and method. A server receives an audio fingerprint of a first audio piece, searches a database for the audio fingerprint, retrieves an audio profile vector associated with the audio fingerprint, updates user preference information based on the audio profile vector, and selects a second audio piece based on the user preference information. The audio fingerprint is generated by creating a matrix based on the frequency measurements of the audio piece, and performing a singular value decomposition of the matrix. To expedite the search of the database and to increase matching accuracy, a subset of candidates in the database is identified based on the most prominent musical notes of the audio piece, and the search is limited to the identified subset. One of the attributes of the audio profile vector is a particular audio class. An identifier for the audio class is generated based on an average of audio fingerprints of the audio pieces belonging to the audio class.

Claims

1-11. -11. (Canceled)

12. An audio fingerprinting method comprising:

receiving an audio signal associated with an audio piece;

obtaining a plurality of frequency measurements of the audio signal;

building a matrix A based on the frequency measurements;

performing a singular value decomposition on the matrix A, wherein A=USV^T;

retrieving one or more rows of matrix V^T; and

storing the retrieved rows of matrix V^T in a data store in association with the audio piece, wherein a request including the retrieved rows of matrix V^T is received by an application program and information stored in a database for the audio piece identified and retrieved based on the rows of matrix V^T in the request.

13. The method of claim 12, wherein rows of the matrix A represent time, and columns of the matrix A represent the frequency measurements.

15. The method of claim 12, wherein the information is an audio profile vector storing acoustic analysis data for the audio piece.

56. The method of claim 15 further comprising recommending a second audio piece based on the acoustic analysis data.

57. The method of claim 15, wherein the audio profile vector quantifies a degree of similarity of the audio piece to audio pieces classified into a particular audio class.

58. The method of claim 57 further comprising generating an identifier for the particular audio class, the generating including:

selecting audio pieces associated with the particular audio class;

computing a second audio fingerprint for each selected audio piece;

calculating an average of the computed second audio fingerprints;

associating the calculated average to the particular audio class; and

storing the calculated average in a data store as the identifier of the particular audio class.

59. The method of claim 58, wherein computing of the second audio fingerprint comprises:

obtaining from a particular audio signal associated with the selected audio piece a plurality of frequency measurements;

building a matrix A based on the frequency measurements;

performing a singular value decomposition on the matrix A, wherein A=USV^T;

retrieving one or more rows of matrix V^T; and

associating the retrieved rows of matrix V^T with the selected audio piece.

60. The method of claim 59, wherein rows of the matrix A represent time, and columns of the matrix A represent the frequency measurements.

61. The method of claim 12, further comprising generating an index of the audio piece, the generating including:

automatically obtaining from the audio signal associated with the audio piece a list of musical notes included in the audio piece;

determining from the audio signal a prominence of the musical notes in the audio piece; and

selecting a pre-determined number of most prominent musical notes in the audio piece as the index.

62. The method of claim 61, wherein the selected musical notes are translated to musical note numbers, and the index includes the translated musical note numbers.

63. The method of claim 61, wherein data stored in the database is organized into one or more groups, wherein each group is identified by a particular index.

64. The method of claim 63, wherein a search of the database for the information for the audio piece limited to a group identified by the generated index.

14. (Canceled)

16. (Canceled)

17. An audio indexing method comprising:

receiving an audio signal of an audio piece;

automatically obtaining from the audio signal a list of musical notes included in the audio piece;

determining from the audio signal a prominence of the musical notes in the audio piece;

selecting a pre-determined number of most prominent musical notes in the audio piece;

generating an index based on the selected musical notes; and

searching a database based on the generated index.

18. The method of claim 17, wherein data stored in the database is organized into one or more groups, wherein each group is identified by a particular index.

19. The method of claim 18, wherein the search of the database is limited to a group identified by the generated index.

20. The method of claim 17, wherein the selected musical notes are translated to musical note numbers, and the index comprises the translated musical note numbers.

21. The method of claim 20, wherein data stored in the database is organized into one or more groups, wherein each group is identified by a particular index, the method further comprising:

comparing the translated musical note numbers in the generated index with note numbers included in an index identifying a group of data in the database; and

outputting a match indication if the index identifying the group includes at least a pre-determined amount of musical note numbers as in the generated index.

22. The method of claim 21, wherein the generated index includes four musical note numbers, and the match indication is output if the index identifying the group includes at least three of the four musical note numbers

23. The method of claim 17, wherein the database is an audio fingerprint database including compact representations of a plurality of audio pieces.

24. A method for generating an identifier for an audio class, the method comprising:

selecting a plurality of audio pieces associated with the audio class;

computing an audio fingerprint for each selected audio piece;

calculating an average of the computed audio fingerprints;

associating the calculated average to the audio class; and

storing the calculated average in a data store.

25. The method of claim 24, wherein computing of the audio fingerprint comprises:

obtaining from an audio signal associated with the audio piece a plurality of frequency measurements;

building a matrix A based on the frequency measurements;

performing a singular value decomposition on the matrix A, wherein A=USV^T;

retrieving one or more rows of matrix V^T;

associating the retrieved rows of matrix V^T with the audio piece; and

storing the retrieved rows of matrix V^T.

26. The method of claim 25, wherein rows of the matrix A represent time, and columns of the matrix A represent the frequency measurements.

27. The method of claim 24, wherein the audio class is defined by a particular orchestration or instrumentation.

28-38. -38. (Canceled)

39. An audio fingerprinting system comprising:

an audio file reader reading an audio file storing an audio piece;

a processor coupled to the audio file reader, the processor being configured to;

obtain a plurality of frequency measurements of audio signals associated with the audio piece;

build a matrix A based on the frequency measurements;

perform a singular value decomposition on the matrix A, wherein A=USV^T; and

retrieve one or more rows of matrix V^T;

a data store coupled to the processor for storing the retrieved rows of matrix V^T in association with the audio piece; and

a database storing information for a plurality of audio pieces, wherein a request including the retrieved rows of matrix V^T is received by an application program and information for the audio piece stored in the database identified and retrieved based on the rows of matrix V^T in the request.

40. The system of claim 39, wherein rows of the matrix A represent time, and columns of the matrix A represent the frequency measurements.

42. The system of claim 39, wherein the information is an audio profile vector storing acoustic analysis data for the audio piece.

65. The system of claim 39, wherein the processor is further configured to recommend a second audio piece based on the acoustic analysis data.

66. The system of claim 42, wherein the audio profile vector quantifies a degree of similarity of the audio piece to audio pieces classified into a particular audio class.

67. The system of claim 66, wherein the processor is further configured to generate an identifier for the particular audio class, the generating including:

selecting audio pieces associated with the particular audio class;

computing a second audio fingerprint for each selected audio piece;

calculating an average of the computed second audio fingerprints;

associating the calculated average to the particular audio class; and

storing the calculated average in a data store as the identifier of the particular audio class.

68. The system of claim 67, wherein computing of the second audio fingerprint comprises:

obtaining from a particular audio signal associated with the selected audio piece a plurality of frequency measurements;

building a matrix A based on the frequency measurements;

performing a singular value decomposition on the matrix A, wherein A=USV^T;

retrieving one or more rows of matrix V^T; and

associating the retrieved rows of matrix V^T with the selected audio piece.

69. The system of claim 68, wherein rows of the matrix A represent time, and columns of the matrix A represent the frequency measurements.

70. The system of claim 39, wherein the processor is further configured to generate an index of the audio piece, the generating including:

automatically obtaining from the audio signal associated with the audio piece a list of musical notes included in the audio piece;

determining from the audio signal a prominence of the musical notes in the audio piece; and

selecting a pre-determined number of most prominent musical notes in the audio piece as the index.

71. The system of claim 70, wherein the selected musical notes are translated to musical note numbers, and the index includes the translated musical note numbers.

72. The system of claim 70, wherein data stored in the database is organized into one or more groups, wherein each group is identified by a particular index.

73. The system of claim 72, wherein a search of the database for the information for the audio piece limited to a group identified by the generated index.

41. (Canceled)

43. (Canceled)

44. An audio indexing system comprising:

means for receiving an audio signal of an audio piece;

means for automatically obtaining from the audio signal a list of musical notes included in the audio piece;

means for determining from the audio signal a prominence of the musical notes in the audio piece;

means for selecting a pre-determined number of most prominent musical notes in the audio piece;

means for generating an index based on the selected musical notes; and

means for searching a database based on the generated index.

45. The system of claim 44, wherein data stored in the database is organized into one or more groups, wherein each group is identified by a particular index.

46. The system of claim 45, wherein the search of the database is limited to a group identified by the generated index.

47. The system of claim 44, wherein the selected musical notes are associated with musical note numbers, and the index comprises the musical note numbers.

48. The system of claim 44, wherein the database is an audio fingerprint database including compact representations of a plurality of audio pieces.

49. A system for generating an identifier for an audio class, the system comprising:

means for computing an audio fingerprint for each of a plurality of selected audio pieces;

means for calculating an average of the computed audio fingerprints;

means for associating the calculated average to the audio class; and

means for storing the calculated average in a data store.

50. The system of claim 49, wherein the means for computing the audio fingerprint comprises:

means for obtaining from an audio signal associated with the audio piece a plurality of frequency measurements;

means for building a matrix A based on the frequency measurements;

means for performing a singular value decomposition on the matrix A, wherein A=USV^T;

means for retrieving one or more rows of matrix V^T; and

means for associating the retrieved rows of matrix V^T with the audio piece.

51. The system of claim 50, wherein rows of the matrix A represent time, and columns of the matrix A represent the frequency measurements.

52. The system of claim 49, wherein the audio class is defined by a particular orchestration or instrumentation.

53. An article of manufacture comprising a computer readable medium having computer usable program code containing executable instructions that, when executed, cause a computer to perform the steps of:

obtaining a plurality of frequency measurements of an audio signal associated with an audio piece;

building a matrix A based on the frequency measurements;

performing a singular value decomposition on the matrix A, wherein A=USV^T;

retrieving one or more rows of matrix V^T; and

storing the retrieved rows of matrix V^T in a data store in association with the audio piece.

54. The article of manufacture of claim 53, wherein rows of the matrix A represent time, and columns of the matrix A represent the frequency measurements.

55. An article of manufacture comprising a computer readable medium having computer usable program code containing executable instructions that, when executed, cause a computer to perform the steps of:

automatically obtaining from an audio signal of an audio piece, a list of musical notes included in the audio piece;

determining from the audio signal a prominence of the musical notes in the audio piece;

selecting a pre-determined number of most prominent musical notes in the audio piece;

generating an index based on the selected musical notes; and

searching a database based on the generated index.