Automatic identification of repeated material in audio signals

• US 8,090,579 B2
• Filed: 02/08/2006
• Issued: 01/03/2012
• Est. Priority Date: 02/08/2005
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Landmark Digital Services LLC
• Inventors: DeBusk, David L., Briggs, Darren P., Karliner, Michael, Li-Chun Wang, Avery, Cheong Tang, Richard Wing
• Primary Examiner(s): SAINT CYR, LEONARD
• Application Number:
  US11/350,254
• Publication Number:
  US 20060277047A1
• Time in Patent Office: 2,155 Days
• Field of Search: 704/235, 704/239, 704/243, 704/251
• US Class Current: 704/235
• CPC Class Codes: G06F 16/685 using automatically derived transcript of audio data, e.g. lyrics speech recognition G10L15/00
• CPC Class Codes: G06F 2218/08 Feature extraction
• CPC Class Codes: G10L 19/018 Audio watermarking, i.e. embedding inaudible data in the audio signal
• CPC Class Codes: G10L 25/48 specially adapted for particular use
• CPC Class Codes: G10L 25/54 for retrieval
• CPC Class Codes: H04H 60/37 for identifying segments of broadcast information, e.g. scenes or extracting programme ID
• CPC Class Codes: H04H 60/58 of audio determination or detection of speech characteristics in general G10L25/00; speech recognition in general G10L15/00

Abstract

A system and method are described for recognizing repeated audio material within at least one media stream without prior knowledge of the nature of the repeated material. The system and method are able to create a screening database from the media stream or streams. An unknown sample audio fragment is taken from the media stream and compared against the screening database to find if there are matching fragments within the media streams by determining if the unknown sample matches any samples in the screening database.

Claims

1. A method of recognizing repeated audio material within at least one media stream without prior knowledge of the nature of the repeated material, comprising:

receiving a sample audio fragment from the at least one media stream;

determining, via a processor, whether the sample audio fragment matches an entry in a database of known media samples, wherein the sample audio fragment is an unknown sample when the sample audio fragment does not match an entry in the database;

determining whether the unknown sample matches an unknown audio fragment indexed in a screening database;

recording a match between the unknown sample and the unknown audio fragment when, based on the determination, the unknown sample matches the unknown audio fragment;

determining whether the unknown sample is ready to be published for identification and inclusion into the database of known media samples based on repeated matches between the unknown sample and the unknown audio fragment; and

publishing the unknown sample for identification and inclusion into the database of known media samples when the repeated matches exceed a threshold.

2. The method of claim 1 wherein determining whether the unknown sample is ready to be published comprises:

mapping matching fragments found within the media stream into a candidate group; and

evaluating the candidate group to find a best exemplar suitable for publication whereby a best exemplar matching parallel chain may constitute recognized repeated material.

3. The method of claim 2 in which evaluating the candidate group is based upon best mutual match to all the other samples in the candidate group.

4. The method of claim 2 wherein mapping the matching fragments further comprises collating parallel chains of adjacent sample fragments, based on continuity of matching of adjacent sample fragments within each chain.

5. The method of claim 4 wherein collating parallel chains includes:

selecting a first chain of unknown probe sample fragments adjacent in time, each probe sample fragment having a time offset within a corresponding media stream;

for each probe sample fragment in the first chain, finding a set of matching fragments from the screening database, each matching sample having a time offset within a corresponding media stream; and

grouping the matching sample fragments in parallel to the probe sample fragments into chains according time adjacency of the matching sample fragments within corresponding media streams, thereby forming parallel chains.

6. The method of claim 5 wherein grouping the matching sample fragments further includes:

for each probe sample and each matching sample, determining a relative time offset between the probe sample and the matching sample;

generating a histogram of relative time offsets; and

for each peak in the histogram of relative time offsets, forming a chain of adjacent matching samples from the matching samples associated with each point in the histogram peak, whereby each chain is a member of the candidate group.

7. The method of claim 6 wherein the relative time offset is determined as the difference between the time offset of the probe sample and the time offset of the matching sample.

8. The method of claim 6 wherein for each probe sample and each matching sample a speed correction factor between the probe sample and the matching sample is obtained, and the relative time offset is determined as the difference between the time offset of the probe sample and the speed-corrected time offset of the matching sample.

9. The method of claim 4 wherein each parallel chain is extended as far as possible in time to form a maximal matching parallel chain.

10. The method of claim 1 in which the unknown sample is delimited by time.

11. The method of claim 1 in which the unknown sample is delimited by segments of already recognized material.

12. The method of claim 1 further comprising:

periodically pruning the screening database by identifying reference material that no longer passes criteria for inclusion.

13. The method of claim 1, wherein publishing the unknown sample to the database of known media samples when the unknown sample is ready for publication comprises adding the unknown sample and a fingerprint of the unknown content to the database of known media samples.

14. The method of claim 1, further comprising sending the unknown sample for positive identification via a manual process.

15. A system for recognizing repeated segments of non-recognized media content in at least one source of non-recognized media content, the system comprising:

a database storing known media samples;

a screening database storing non-recognized media segments received from the at least one source;

a candidate manager receiving the non-recognized media after the non-recognized media has failed to match an entry in the database of known media samples, and associating an identifier with samples of the non-recognized media;

a fingerprint generator operable to create fingerprints for non-recognized media segments; and

a media search engine connected to the candidate manager and the fingerprint generator, the media search engine configured to;

compare fingerprints of non-recognized media against the previously stored non-recognized media fingerprints in the screening database,

record repeated matches of the non-recognized media within the screening database,

determine if the non-recognized media is ready for publication for identification and inclusion into the database of known media samples based on whether the repeated matches in the screening database, and publish the unknown sample for identification and inclusion into the database of known media samples when the repeated matches exceed a threshold value.

16. The system of claim 15 in which the non-recognized media is delimited by time.

17. The system of claim 15 in which the non-recognized media is delimited by segments of already recognized material.

18. The system of claim 15 wherein the non-recognized media is mapped into fragments which are matched against fragments in the screening database associated with the fingerprint generator, and the matching fragments are grouped into a candidate group.

19. The system of claim 18 wherein the candidate group is evaluated to find a best exemplar suitable for publication whereby the best exemplar may constitute recognized repeated material.

20. The system of claim 18 in which evaluating the candidate group is based upon mutual match to the other samples in the candidate group.

21. The system of claim 15 wherein the screening database is periodically pruned by identifying a non-recognized media segment that no longer passes criteria for inclusion.

22. A method of recognizing repeated audio material within at least one media stream without prior knowledge of the nature of the repeated material, comprising:

receiving a sample audio fragment from the at least one media stream;

determining, via a processor, whether the sample audio fragment is a non-recognized segment of audio (NRA) based on a comparison with a database of known media samples;

creating a screening database from the at least one media stream, said screening database including previously-determined NRA'"'"'s;

determining if the NRA matches one or more of the previously-determined NRA'"'"'s in the screening database;

publishing the NRA for identification and inclusion into the database of known media samples when the matches between the NRA and the one or more of the previously-determined NRA'"'"'s exceeds a threshold value.