Systems and methods for recognizing sound and music signals in high noise and distortion

• US 9,899,030 B2
• Filed: 07/19/2016
• Issued: 02/20/2018
• Est. Priority Date: 07/31/2000
• Status: Expired due to Fees

• Current Assignee: Apple Inc.
• Original Assignee: Shazam Ivestments Ltd. (Apple Inc.)
• Inventors: Wang, Avery Li-Chun, Smith, III, Julius O.
• Primary Examiner(s): McFadden, Susan
• Application Number:
  US15/213,969
• Publication Number:
  US 20160328473A1
• Time in Patent Office: 581 Days
• Field of Search: 704270
• CPC Class Codes: G06F 16/634 Query by example, e.g. query by humming
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
• CPC Class Codes: G10L 19/018 Audio watermarking, i.e. embedding inaudible data in the audio signal
• CPC Class Codes: G10L 25/54 for retrieval
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence

Abstract

A method for recognizing an audio sample locates an audio file that matches the sample from a database indexing original recordings. Each indexed file is represented in the database index by a set of landmark timepoints and associated fingerprints. Landmarks occur at reproducible locations within the file, while fingerprints represent features of the signal at or near the landmark timepoints, and the fingerprints are invariant with respect to time stretching of the sample. To perform recognition, landmarks and fingerprints are computed for the unknown sample and used to retrieve matching fingerprints from the database. For each file containing matching fingerprints, the landmarks are compared with landmarks of the sample at which the same fingerprints were computed. If a large number of corresponding landmarks are linearly related, i.e., if equivalent fingerprints of the sample and retrieved file have the same time evolution, then the file is identified with the sample.

Claims

1. A method performed by a computing device, the method comprising:

receiving a media sample of media subject to time stretching, wherein the time stretching of the media occurs based on the media being played at a speed different than an original recording speed;

generating a fingerprint of the media sample for content identification that is invariant with respect to time stretching of the media sample; and

performing a content identification of the media sample using the fingerprint by comparing the fingerprint to stored fingerprints in memory of a database index, wherein the stored fingerprints are based on media played at original recording speeds.

2. The method of claim 1, wherein receiving the media sample comprises:

receiving the media sample from a client device that has recorded the media sample by use of a microphone.

3. The method of claim 1, wherein generating the fingerprint of the media sample for content identification that is invariant with respect to time stretching of the media sample comprises:

determining spectrogram peaks of the media sample, and each spectrogram peak has an associated time;

determining respective time differences between selected spectrogram peaks; and

generating the fingerprint based on ratios of the respective time differences.

4. The method of claim 1, wherein generating the fingerprint of the media sample for content identification that is invariant with respect to time stretching of the media sample comprises:

generating the fingerprint based on ratios of frequencies of spectrogram peaks of the media sample.

5. The method of claim 1, wherein generating the fingerprint of the media sample for content identification that is invariant with respect to time stretching of the media sample comprises:

determining spectrogram peaks of the media sample, and each spectrogram peak has an associated time and frequency;

generating intermediate values of the media sample based on the associated times and frequencies of the spectrogram peaks; and

combining the intermediate values using a function to form a hashed fingerprint of the media sample that is independent of time stretching of the media sample.

6. The method of claim 1, wherein generating the fingerprint of the media sample for content identification that is invariant with respect to time stretching of the media sample comprises:

determining, by the computing device, a time-frequency representation of the media sample;

determining, by the computing device, salient points in the time-frequency representation of the media sample;

determining differences of times of the salient points; and

generating the fingerprint based at least on the differences of times of the salient points.

7. The method of claim 1, wherein generating the fingerprint of the media sample for content identification that is invariant with respect to time stretching of the media sample comprises:

determining spectrogram peaks of the media sample, and each spectrogram peak has an associated time and frequency;

determining respective time differences between selected spectrogram peaks; and

generating the fingerprint of the media sample by hashing together the respective time differences and corresponding frequencies of the selected spectrogram peaks for at least a portion of the spectrogram peaks.

8. The method of claim 1, wherein generating the fingerprint of the media sample for content identification that is invariant with respect to time stretching of the media sample comprises:

determining spectrogram peaks of the media sample, and each spectrogram peak has an associated time and frequency;

determining respective time differences between selected spectrogram peaks;

determining a reference frequency of the spectrogram peaks;

forming products of the reference frequency with the respective time differences; and

generating the fingerprint based on the products.

9. The method of claim 1, wherein generating the fingerprint of the media sample for content identification that is invariant with respect to time stretching of the media sample comprises:

determining spectrogram peaks of the media sample, and each spectrogram peak has an associated time and frequency;

determining respective time differences between selected spectrogram peaks;

forming products of selected frequencies of the spectrogram peaks with the respective time differences; and

generating the fingerprint based on the products.

10. The method of claim 1, wherein generating the fingerprint of the media sample for content identification that is invariant with respect to time stretching of the media sample comprises:

determining spectrogram peaks of the media sample, and each spectrogram peak has an associated frequency;

determining a reference frequency of a selected spectrogram peak;

forming quotients of the reference frequency with one or more other frequencies of other spectrogram peaks; and

generating the fingerprint based on the quotients.

11. The method of claim 1, wherein generating the fingerprint of the media sample for content identification that is invariant with respect to time stretching of the media sample comprises:

determining spectrogram peaks of the media sample, and each spectrogram peak has an associated time and frequency;

determining respective time differences between a first spectrogram peak and subsequent spectrogram peaks;

determining a reference frequency of the first spectrogram peak;

forming quotients of the reference frequency with one or more other frequencies of the subsequent spectrogram peaks;

forming products of the reference frequency with the respective time differences; and

combining the quotients and the products using a function to form a hashed fingerprint that is independent of time stretching.

12. The method of claim 11, further comprising:

wherein combining the quotients and the products comprises packing the quotients and the products into concatenated bitfields.

13. A system, comprising:

a computing device; and

a database index for storing fingerprints of known media based on media played at original recording speeds;

a memory, the memory configured to store instructions that when executed by the computing device cause the computing device to;

receiving a media sample of media subject to time stretching, wherein the time stretching of the media occurs based on the media being played at a speed different than an original recording speed;

generating a fingerprint of the media sample for content identification that is invariant with respect to time stretching of the media sample; and

performing a content identification of the media sample using the fingerprint by comparing the fingerprint to stored fingerprints in the database index.

14. The system of claim 13, wherein the memory further stores instructions that when executed by the computing device cause the computing device to:

determine spectrogram peaks of the media sample, and each spectrogram peak has an associated time;

determine respective time differences between selected spectrogram peaks; and

generate the fingerprint based on ratios of the respective time differences.

15. The system of claim 13, wherein the memory further stores instructions that when executed by the computing device cause the computing device to:

determine spectrogram peaks of the media sample, and each spectrogram peak has an associated time and frequency;

determine respective time differences between a first spectrogram peak and subsequent spectrogram peaks;

determine a reference frequency of the first spectrogram peak;

form quotients of the reference frequency with one or more other frequencies of the subsequent spectrogram peaks;

form products of the reference frequency with the respective time differences; and

combine the quotients and the products using a function to form a hashed fingerprint that is independent of time stretching.

16. The system of claim 13, wherein the media sample is received from a client device that has recorded the media sample by use of a microphone.

17. The system of claim 13, wherein the computing device generates the fingerprint based on ratios of frequencies of spectrogram peaks of the media sample.

18. The system of claim 13, wherein the memory further stores instructions that when executed by the computing device cause the computing device to:

determine spectrogram peaks of the media sample, and each spectrogram peak has an associated time and frequency;

generate intermediate values of the media sample based on the associated times and frequencies of the spectrogram peaks; and

combine the intermediate values using a function to form a hashed fingerprint of the media sample that is independent of time stretching of the media sample.

19. The system of claim 13, wherein the memory further stores instructions that when executed by the computing device cause the computing device to:

determine a time-frequency representation of the media sample;

determine salient points in the time-frequency representation of the media sample;

determine differences of times of the salient points; and

generate the fingerprint based at least on the differences of times of the salient points.

20. The system of claim 13, wherein the memory further stores instructions that when executed by the computing device cause the computing device to:

determine spectrogram peaks of the media sample, and each spectrogram peak has an associated time and frequency;

determine respective time differences between selected spectrogram peaks; and

generate the fingerprint of the media sample by hashing together the respective time differences and corresponding frequencies of the selected spectrogram peaks for at least a portion of the spectrogram peaks.