Systems and methods for recognizing sound and music signals in high noise and distortion

• US 10,497,378 B2
• Filed: 01/23/2018
• Issued: 12/03/2019
• Est. Priority Date: 07/31/2000
• Status: Expired due to Term

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Wang, Avery Li-Chun, Smith, III, Julius O.
• Primary Examiner(s): McFadden, Susan I
• Application Number:
  US15/877,955
• Publication Number:
  US 20180374491A1
• Time in Patent Office: 679 Days
• Field of Search: 704270
• CPC Class Codes: G06F 16/634 Query by example, e.g. query by humming
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
• CPC Class Codes: G10L 19/018 Audio watermarking, i.e. embedding inaudible data in the audio signal
• CPC Class Codes: G10L 25/54 for retrieval
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence

Abstract

A method for recognizing an audio sample locates an audio file that closely matches the audio sample from a database indexing a large set of original recordings. Each indexed audio file is represented in the database index by a set of landmark timepoints and associated fingerprints. Landmarks occur at reproducible locations within the file, while fingerprints represent features of the signal at or near the landmark timepoints. To perform recognition, landmarks and fingerprints are computed for the unknown sample and used to retrieve matching fingerprints from the database. For each file containing matching fingerprints, the landmarks are compared with landmarks of the sample at which the same fingerprints were computed. If a large number of corresponding landmarks are linearly related, i.e., if equivalent fingerprints of the sample and retrieved file have the same time evolution, then the file is identified with the sample.

Claims

1. A method performed by a computing device, the method comprising:

receiving a media sample;

converting, by a computing device, the media sample into frequency transform frames at periodic time intervals;

determining, by the computing device, an energy density representation of the frequency transform frames;

determining, by the computing device, time-frequency coordinates corresponding to local maxima of the energy density representation;

generating, by the computing device, fingerprints of the media sample based on determined time-frequency coordinates of the energy density representation; and

performing a content identification of the media sample using the fingerprint by comparing the fingerprint to stored fingerprints in memory of a database index.

2. The method of claim 1, wherein receiving the media sample

comprises:

receiving the media sample from a client device that has recorded the media sample by use of a microphone.

3. The method of claim 1, wherein converting the media sample into frequency transform frames at periodic time intervals comprises:

converting the media sample into frequency transform frames using a Fast Fourier Transform (FFT).

4. The method of claim 1, wherein converting the media sample into frequency transform frames at periodic time intervals comprises:

using a sampling rate of 8000 Hz and an FFT frame size of 1024 samples.

5. The method of claim 1, wherein determining the energy density representation of the frequency transform frames comprises:

determining a spectrogram plot depicting energy density at each time-frequency point for the frequency transform frames.

6. The method of claim 1, further comprising:

determining landmarks corresponding to the fingerprints, wherein the landmarks correspond to respective times of the time-frequency coordinates.

7. The method of claim 1, wherein generating the fingerprints of the media sample based on determined time-frequency coordinates of the energy density representation comprises:

generating fingerprints of the media sample for content identification that are invariant with respect to time stretching of the media sample.

8. The method of claim 7, wherein generating the fingerprint of the media sample for content identification that is invariant with respect to time stretching of the media sample comprises:

generating the fingerprint based on ratios of frequencies of spectrogram peaks of the media sample.

9. The method of claim 1, wherein determining the energy density representation of the frequency transform frames comprises determining windowed and overlapped frames of the media sample using a Fast Fourier Transform (FFT).

10. The method of claim 1, wherein determining the energy density representation of the frequency transform frames comprises determining a spectrogram of the media sample.

11. A system, comprising:

a computing device; and

a memory, the memory configured to store instructions that when executed by the computing device cause the system to perform functions comprising;

receiving a media sample;

converting, by the computing device, the media sample into frequency transform frames at periodic time intervals;

determining, by the computing device, an energy density representation of the frequency transform frames;

determining, by the computing device, time-frequency coordinates corresponding to local maxima of the energy density representation;

generating, by the computing device, fingerprints of the media sample based on determined time-frequency coordinates of the energy density representation; and

performing a content identification of the media sample using the fingerprint by comparing the fingerprint to stored fingerprints in memory of a database index.

12. The system of claim 11, wherein receiving the media sample

comprises:

receiving the media sample from a client device that has recorded the media sample by use of a microphone.

13. The system of claim 11, wherein determining the energy density representation of the frequency transform frames comprises:

determining a spectrogram plot depicting energy density at each time-frequency point for the frequency transform frames.

14. The system of claim 11, wherein the memory is further configured to store instructions that when executed by the computing device cause the system to perform functions comprising:

determining landmarks corresponding to the fingerprints, wherein the landmarks correspond to respective times of the time-frequency coordinates.

15. The system of claim 11, wherein generating the fingerprints of the media sample based on determined time-frequency coordinates of the energy density representation comprises:

generating fingerprints of the media sample for content identification that are invariant with respect to time stretching of the media sample.

16. A non-transitory computer readable medium having stored there instructions, that when executed by a computing device, cause the computing device to perform functions comprising:

receiving a media sample;

converting, by the computing device, the media sample into frequency transform frames at periodic time intervals;

determining, by the computing device, an energy density representation of the frequency transform frames;

determining, by the computing device, time-frequency coordinates corresponding to local maxima of the energy density representation; and

generating, by the computing device, fingerprints of the media sample based on determined time-frequency coordinates of the energy density representation.

17. The non-transitory computer readable medium of claim 16, further comprising instructions, that when executed by a computing device, cause the computing device to perform functions comprising:

determining landmarks corresponding to the fingerprints, wherein the landmarks correspond to respective times of the time-frequency coordinates.

18. The non-transitory computer readable medium of claim 16, wherein generating the fingerprints of the media sample based on determined time-frequency coordinates of the energy density representation comprises:

generating fingerprints of the media sample for content identification that are invariant with respect to time stretching of the media sample.