Systems and methods for recognizing sound and music signals in high noise and distortion

• US 8,700,407 B2
• Filed: 01/24/2013
• Issued: 04/15/2014
• Est. Priority Date: 07/31/2000
• Status: Expired due to Fees

• Current Assignee: Apple Inc.
• Original Assignee: Shazam Ivestments Ltd. (Apple Inc.)
• Inventors: Wang, Avery Li-Chun, Smith, Julius O.
• Primary Examiner(s): MCFADDEN, SUSAN IRIS
• Application Number:
  US13/749,243
• Publication Number:
  US 20130138442A1
• Time in Patent Office: 446 Days
• Field of Search: 704/270
• US Class Current: 704/270
• CPC Class Codes: G06F 16/634 Query by example, e.g. query by humming
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
• CPC Class Codes: G10L 19/018 Audio watermarking, i.e. embedding inaudible data in the audio signal
• CPC Class Codes: G10L 25/54 for retrieval
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence

Abstract

A method for recognizing an audio sample locates an audio file that closely matches the audio sample from a database indexing a large set of original recordings. Each indexed audio file is represented in the database index by a set of landmark timepoints and associated fingerprints. Landmarks occur at reproducible locations within the file, while fingerprints represent features of the signal at or near the landmark timepoints. To perform recognition, landmarks and fingerprints are computed for the unknown sample and used to retrieve matching fingerprints from the database. For each file containing matching fingerprints, the landmarks are compared with landmarks of the sample at which the same fingerprints were computed. If a large number of corresponding landmarks are linearly related, i.e., if equivalent fingerprints of the sample and retrieved file have the same time evolution, then the file is identified with the sample.

Claims

1. A method performed by a computing device, the method comprising:

receiving a media sample, the media sample being a rendition of a portion of an original recording; and

determining one or more characteristics of the media sample; and

determining, by the computing device, a time offset indicating a time position in the original recording corresponding to the one or more characteristics of the media sample.

2. The method of claim 1, further comprising identifying a media entity from a plurality of media entities which matches the media sample based on substantially matching features of the media sample and the media entity at a position in the media entity corresponding to the time offset.

3. The method of claim 1, further comprising determining, within a database, substantially matching characteristics of one or more media entities to the one or more characteristics of the media sample, wherein the substantially matching characteristics are associated with respective time offsets corresponding to the time offset.

4. The method of claim 1, further comprising:

receiving the media sample from a client device;

determining an identity of a media entity associated with substantially matching characteristics to the one or more characteristics; and

providing the identity of the media entity to the client device.

5. The method of claim 1, further comprising:

determining a fingerprint of the media sample based on the one or more characteristics;

associating the time offset with the fingerprint; and

searching for a substantially matching fingerprint and time offset within a database of fingerprint and offset pairs.

6. The method of claim 1, wherein the time offset is further indicative of a time difference between a beginning of the original recording and a beginning of the media sample.

7. The method of claim 1, further comprising determining one or more landmarks of the media sample, each landmark representing a distinctive and reproducible location in the media sample.

8. The method of claim 7, wherein each landmark represents a location at a local maxima of the media sample.

9. The method of claim 7, further comprising determining a fingerprint of the media sample at the landmark.

10. The method of claim 7, further comprising:

determining a fingerprint of the media sample based on the one or more characteristics; and

determining multiple fingerprints of the media sample for each landmark.

11. The method of claim 1, further comprising:

determining one or more landmarks of the media sample, each landmark representing a distinctive and reproducible location in the media sample;

providing a plurality of time offsets to a given landmark to determine a set of time-slices;

at each time-slice, determining a value that characterizes a feature in the media sample; and

combining the value for each time-slice to form a multi-slice fingerprint value.

12. The method of claim 1, further comprising performing a content identification of the media sample to determine an identity of content of the media sample.

13. The method of claim 12, further comprising based on a result of the content identification, determining the time offset.

14. The method of claim 1, further comprising determining, by the computing device, multiple time offsets, each time offset indicating a given time position in the media stream corresponding to one of a plurality of characteristics of the media sample.

15. The method of claim 1, wherein receiving the media sample comprises receiving a sound recording, and wherein determining the one or more characteristics of the media sample comprises determining a fingerprint value, and the method further comprising:

determining one or more landmarks of the sound recording, each landmark representing a distinctive and reproducible location in the sound recording, wherein the fingerprint value represents a number of features of the sound recording substantially at the distinctive and reproducible location of a given landmark;

providing an index set comprising the fingerprint value and an associated landmark; and

associating a sound identifier with the index set, wherein the sound identifier includes information indicative of an identity of the sound recording.

16. The method of claim 1, wherein the media stream comprises a song, and wherein determining, by the computing device, the time offset comprises using the determined one or more characteristics of the media sample to determine the time position of the media sample in the song.

17. A system, comprising:

a computing device; and

a memory, the memory configured to store instructions that when executed by the computing device cause the computing device to;

receiving a media sample, the media sample being a rendition of a portion of an original recording; and

determining one or more characteristics of the media sample; and

determining a time offset indicating a time position in the original recording corresponding to the one or more characteristics of the media sample.

18. The system of claim 17, wherein the instructions are further executable by the computing device to cause the computing device to:

determine, within a database, substantially matching characteristics of one or more media entities to the one or more characteristics of the media sample, wherein the substantially matching characteristics are associated with respective time offsets corresponding to the time offset.

19. The system of claim 17, wherein the instructions are further executable by the computing device to cause the computing device to:

determine one or more landmarks of the media sample, each landmark representing a distinctive and reproducible location in the media sample; and

determine a fingerprint of the media sample at the one or more landmarks.

20. The system of claim 17, wherein the instructions are further executable by the computing device to cause the computing device to:

perform a content identification of the media sample to determine an identity of content of the media sample; and

based on a result of the content identification, determining the time offset.

21. A method performed by a computing device, the method comprising:

determining one or more characteristics of each media sample of a plurality of media samples; and

determining, for each of the one or more characteristics of each media sample, a time offset indicating a time position in the media sample corresponding to the determined one or more characteristics of the media sample, wherein the one or more characteristics and a respective time offset form one or more characteristic/time offset pairs; and

generating, by the computing device, an index set for each media sample, wherein the index set includes information indicative of at least the one or more characteristic/time offset pairs.

22. The method of claim 21, wherein the media sample is a rendition of a portion of an original recording, and wherein the time offset is further indicative of a time difference between a beginning of the original recording and a beginning of the media sample.

23. The method of claim 21, wherein the index set is operable to be used for identification of unknown media samples.

24. The method of claim 21, further comprising associating information indicative of an identity of the media sample with the index set for each media sample.

25. The method of claim 21, further comprising:

determining one or more landmarks of the media sample, each landmark representing a distinctive and reproducible location in the media sample; and

determining the one or more characteristics approximately at the one or more landmarks.

26. The method of claim 21, wherein each landmark represents a location at a local maxima of the media sample.