Systems and methods for recognizing sound and music signals in high noise and distortion

• US 8,386,258 B2
• Filed: 04/30/2012
• Issued: 02/26/2013
• Est. Priority Date: 07/31/2000
• Status: Expired due to Term

• Current Assignee: Apple Inc.
• Original Assignee: Shazam Ivestments Ltd. (Apple Inc.)
• Inventors: Wang, Avery Li-Chun, Smith, Julius O. III
• Primary Examiner(s): MCFADDEN, SUSAN IRIS
• Application Number:
  US13/460,079
• Publication Number:
  US 20120221131A1
• Time in Patent Office: 302 Days
• Field of Search: 704/270
• US Class Current: 704/270
• CPC Class Codes: G06F 16/634 Query by example, e.g. query by humming
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
• CPC Class Codes: G10L 19/018 Audio watermarking, i.e. embedding inaudible data in the audio signal
• CPC Class Codes: G10L 25/54 for retrieval
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence

Abstract

A method for recognizing an audio sample locates an audio file that closely matches the audio sample from a database indexing a large set of original recordings. Each indexed audio file is represented in the database index by a set of landmark timepoints and associated fingerprints. Landmarks occur at reproducible locations within the file, while fingerprints represent features of the signal at or near the landmark timepoints. To perform recognition, landmarks and fingerprints are computed for the unknown sample and used to retrieve matching fingerprints from the database. For each file containing matching fingerprints, the landmarks are compared with landmarks of the sample at which the same fingerprints were computed. If a large number of corresponding landmarks are linearly related, i.e., if equivalent fingerprints of the sample and retrieved file have the same time evolution, then the file is identified with the sample.

Claims

1. A method comprising:

receiving media samples into a buffer;

storing the received media samples as a sequentially growing segment of media; and

performing a content identification of the sequentially growing segment of media in the buffer to determine an identity of content of the received media samples.

2. The method of claim 1, further comprising receiving the media samples into the buffer on a continuous basis.

3. The method of claim 1, wherein the buffer has a size that grows as the media samples are received.

4. The method of claim 1, further comprising:

performing the content identification of the sequentially growing segment of media in the buffer at a first time to determine an identity of content of a first amount of received media samples;

performing a second content identification of the sequentially growing segment of media in the buffer at a second time to determine an identity of content of a second amount of received media samples that is more than the first amount of media samples.

5. The method of claim 4, further comprising:

after performing the content identification at the first time, determining whether the identity of content of the received media samples has been determined; and

based on the determination, performing the second content identification.

6. The method of claim 1, further comprising repeating said performing step for a sequentially growing size of the sequentially growing segment of media.

7. The method of claim 1, further comprising providing a report indicating a presence of identified content.

8. The method of claim 7, further comprising providing the report when the identity of content of the received media samples changes.

9. The method of claim 7, wherein the report includes information indicative of one or more of the identity of recognized content, a time of recognition, and a location of receipt of the media samples.

10. The method of claim 1, wherein the media samples include samples of music, and the method further comprises generating a list of music played in a received sound stream.

11. The method of claim 1, wherein receiving the media samples into the buffer comprises receiving the media samples via a microphone.

12. The method of claim 1, further comprising receiving the media samples from a sampling device.

13. The method of claim 1, wherein receiving the media samples into the buffer comprises receiving from a sampling device a set of sample fingerprints for each given media sample, each sample fingerprint characterizing a particular sample landmark within the media sample.

14. The method of claim 1, further comprising:

a sampling device receiving the media samples via a microphone; and

the sampling device performing the content identification.

15. The method of claim 1, wherein performing the content identification comprises:

determining a set of sample fingerprints for each given media sample, each sample fingerprint characterizing a particular sample landmark within the media sample;

obtaining a set of file fingerprints, each file fingerprint characterizing a particular file landmark within a known media entity;

generating correspondences between the sample landmarks and the file landmarks, wherein corresponding landmarks have substantially matching fingerprints; and

identifying the media entity if a plurality of the corresponding landmarks are substantially linearly related.

16. The method of claim 1, wherein performing the content identification comprises periodically performing the content identification of given media samples in the buffer.

17. The method of claim 1, wherein performing the content identification of the sequentially growing segment of media in the buffer comprises performing the content identification of contents in the buffer at predetermined time intervals to determine an identity of a segment of content, wherein the segment of content comprises media samples received during the predetermined time intervals.

18. The method of claim 1, further comprising performing the content identification on a rolling basis to determine an identity of a segment of the sequentially growing segment of media.

19. The method of claim 1, further comprising performing the content identification at a predetermined time interval to determine an identity of a segment of the sequentially growing segment of media, wherein the segment has a length corresponding to the predetermined time interval.

20. The method of claim 1, further comprising removing media samples from the buffer after a predetermined amount of time on a first in first out basis.

21. The method of claim 1, further comprising:

adding newly received media samples to a beginning of the buffer; and

removing media samples from an end of the buffer after a predetermined amount of time.

22. A non-transitory computer readable memory having stored thereon instructions executable by a computing device to cause the computing device to perform functions comprising:

receiving media samples into a buffer;

storing the received media samples as a sequentially growing segment of media; and

performing a content identification of the sequentially growing segment of media in the buffer to determine an identity of content of the received media samples.

23. The non-transitory computer readable memory of claim 22, further comprising instructions executable by the computing device to cause the computing device to perform functions comprising:

performing the content identification of the sequentially growing segment of media in the buffer at a first time to determine an identity of content of a first amount of received media samples;

determining whether the identity of content of the received media samples has been determined; and

based on the determination, performing a second content identification of the sequentially growing segment of media in the buffer at a second time to determine an identity of content of a second amount of received media samples that is more than the first amount of media samples.

24. The non-transitory computer readable memory of claim 22, further comprising repeating said performing for a sequentially growing size of the sequentially growing segment of media.

25. The non-transitory computer readable memory of claim 22, wherein performing the content identification comprises periodically performing the content identification of given media samples in the buffer.

26. The non-transitory computer readable memory of claim 22, wherein performing the content identification of the sequentially growing segment of media in the buffer comprises performing the content identification of contents in the buffer at predetermined time intervals to determine an identity of a segment of content, wherein the segment of content comprises media samples received during the predetermined time intervals.

27. The non-transitory computer readable memory of claim 22, further comprising performing the content identification on a rolling basis to determine an identity of a segment of the sequentially growing segment of media.

28. The non-transitory computer readable memory of claim 22, further comprising performing the content identification at a predetermined time interval to determine an identity of a segment of the sequentially growing segment of media, wherein the segment has a length corresponding to the predetermined time interval.

29. The non-transitory computer readable memory of claim 22, further comprising removing media samples from the buffer after a predetermined amount of time on a first in first out basis.

30. The non-transitory computer readable memory of claim 22, further comprising:

adding newly received media samples to a beginning of the buffer; and

removing media samples from an end of the buffer after a predetermined amount of time.

31. A device comprising:

a buffer configured to receive media samples;

a processor;

instructions stored in memory and executable by the processor to cause the processor to perform functions comprising;

receiving media samples into the buffer;

storing the received media samples as a sequentially growing segment of media; and

performing a content identification of the sequentially growing segment of media in the buffer to determine an identity of content of the received media samples.

32. The device of claim 31, further comprising instructions executable by the processor to cause the processor to perform functions comprising:

performing the content identification of the sequentially growing segment of media in the buffer at a first time to determine an identity of content of a first amount of received media samples;

determining whether the identity of content of the received media samples has been determined; and

based on the determination, performing a second content identification of the sequentially growing segment of media in the buffer at a second time to determine an identity of content of a second amount of received media samples that is more than the first amount of media samples.

33. The device of claim 31, further comprising repeating said performing for a sequentially growing size of the sequentially growing segment of media.

34. The device of claim 31, wherein performing the content identification of the sequentially growing segment of media in the buffer comprises performing the content identification of contents in the buffer at predetermined time intervals to determine an identity of a segment of content, wherein the segment of content comprises media samples received during the predetermined time intervals.

35. The device of claim 31, further comprising performing the content identification at a predetermined time interval to determine an identity of a segment of the sequentially growing segment of media, wherein the segment has a length corresponding to the predetermined time interval.