Automatic identification of repeated material in audio signals

• US 9,092,518 B2
• Filed: 10/28/2013
• Issued: 07/28/2015
• Est. Priority Date: 02/08/2005
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Shazam Ivestments Ltd. (Apple Inc.)
• Inventors: DeBusk, David L., Briggs, Darren P., Karliner, Michael, Tang, Richard W. Cheong, Wang, Avery Li-Chun
• Primary Examiner(s): Saint Cyr, Leonard
• Application Number:
  US14/064,552
• Publication Number:
  US 20140052284A1
• Time in Patent Office: 638 Days
• Field of Search: 704/233, 704/235, 704/239, 704/243, 704/251
• US Class Current: 1/1
• CPC Class Codes: G06F 16/685 using automatically derived transcript of audio data, e.g. lyrics speech recognition G10L15/00
• CPC Class Codes: G06F 2218/08 Feature extraction
• CPC Class Codes: G10L 19/018 Audio watermarking, i.e. embedding inaudible data in the audio signal
• CPC Class Codes: G10L 25/48 specially adapted for particular use
• CPC Class Codes: G10L 25/54 for retrieval
• CPC Class Codes: H04H 60/37 for identifying segments of broadcast information, e.g. scenes or extracting programme ID
• CPC Class Codes: H04H 60/58 of audio determination or detection of speech characteristics in general G10L25/00; speech recognition in general G10L15/00

Abstract

A system and method are described for recognizing repeated audio material within at least one media stream without prior knowledge of the nature of the repeated material. The system and method are able to create a screening database from the media stream or streams. An unknown sample audio fragment is taken from the media stream and compared against the screening database to find if there are matching fragments within the media streams by determining if the unknown sample matches any samples in the screening database.

Claims

1. A method comprising:

receiving, at a computing device, a segment of non-recognized data from a media stream, wherein the non-recognized data comprises unknown content;

determining whether the segment of non-recognized data matches previous non-recognized data from one or more media streams so as to identify repeated portions of non-recognized content, wherein the one or more media streams includes the media stream;

grouping matching results of previous non-recognized data into a matching fragment set;

determining a first boundary of data in the previous non-recognized data that matches the segment of non-recognized data based on a combination of adjacent data fragments in the matching fragment set, wherein given adjacent data fragments are fragments adjacent in time;

determining a second boundary of data in the previous non-recognized data that matches the segment of non-recognized data based on a number of elements in the matching fragment set; and

determining, by the computing device, a hierarchical nesting of repeated portions of non-recognized content based on the first boundary and the second boundary.

2. The method of claim 1, wherein the first boundary indicates a starting and ending point of first media content, and the second boundary indicates a starting and ending point of second media content, wherein the second media content includes the first media content.

3. The method of claim 1, further comprising providing, by the computing device, data bounded by the first boundary for content identification.

4. The method of claim 1, further comprising providing, by the computing device, data bounded by the second boundary for content identification.

5. The method of claim 1, further comprising providing data bounded by the first boundary or by the second boundary for content identification based on a number of the repeated portions exceeding a threshold.

6. The method of claim 1, further comprising processing the segment of non-recognized data to extract fingerprint features from the segment of non-recognized data.

7. The method of claim 6, further comprising indexing the fingerprint features into a database, wherein the database includes unknown content from media streams.

8. The method of claim 7, wherein determining whether the segment of non-recognized data matches previous non-recognized data from one or more media streams so as to identify repeated portions of non-recognized content comprises comparing the fingerprint features with other fingerprint features stored in the database.

9. The method of claim 1, wherein determining the first boundary of data in the previous non-recognized data that matches the segment of non-recognized data based on the combination of adjacent data fragments in the matching fragment set comprises determining a count of the adjacent data fragments.

10. The method of claim 1, further comprising:

forming a matching chain of data based on the adjacent data fragments; and

providing the matching chain of data for content identification.

11. The method of claim 1, further comprising:

applying a weight to the matching results based on the number of elements in the matching fragment set; and

determining the second boundary of data in the previous non-recognized data that matches the segment of non-recognized data based on the weight.

12. A non-transitory computer readable medium having stored thereon instructions, that when executed by a computing device, cause the computing device to perform functions comprising:

receiving a segment of non-recognized data from a media stream, wherein the non-recognized data comprises unknown content;

determining whether the segment of non-recognized data matches previous non-recognized data from one or more media streams so as to identify repeated portions of non-recognized content, wherein the one or more media streams includes the media stream;

grouping matching results of previous non-recognized data into a matching fragment set;

determining a first boundary of data in the previous non-recognized data that matches the segment of non-recognized data based on a combination of adjacent data fragments in the matching fragment set, wherein given adjacent data fragments are fragments adjacent in time;

determining a second boundary of data in the previous non-recognized data that matches the segment of non-recognized data based on a number of elements in the matching fragment set; and

determining a hierarchical nesting of repeated portions of non-recognized content based on the first boundary and the second boundary.

13. The non-transitory computer readable medium of claim 12, wherein the first boundary indicates a starting and ending point of first media content, and the second boundary indicates a starting and ending point of second media content, wherein the second media content includes the first media content.

14. The non-transitory computer readable medium of claim 12, wherein the functions further comprise providing data bounded by the first boundary or by the second boundary for content identification based on a number of the repeated portions exceeding a threshold.

15. The non-transitory computer readable medium of claim 12, wherein determining the first boundary of data in the previous non-recognized data that matches the segment of non-recognized data based on the combination of adjacent data fragments in the matching fragment set comprises determining a count of the adjacent data fragments.

16. The non-transitory computer readable medium of claim 12, wherein the functions further comprise:

forming a matching chain of data based on the adjacent data fragments; and

providing the matching chain of data for content identification.

17. A system comprising:

one or more processors;

non-transitory computer readable medium having stored thereon instructions, that when executed by the one or more processors, cause the one or more processors to perform functions comprising;

receiving a segment of non-recognized data from a media stream, wherein the non-recognized data comprises unknown content;

determining whether the segment of non-recognized data matches previous non-recognized data from one or more media streams so as to identify repeated portions of non-recognized content, wherein the one or more media streams includes the media stream;

grouping matching results of previous non-recognized data into a matching fragment set;

determining a first boundary of data in the previous non-recognized data that matches the segment of non-recognized data based on a combination of adjacent data fragments in the matching fragment set, wherein given adjacent data fragments are fragments adjacent in time;

determining a second boundary of data in the previous non-recognized data that matches the segment of non-recognized data based on a number of elements in the matching fragment set; and

determining a hierarchical nesting of repeated portions of non-recognized content based on the first boundary and the second boundary.

18. The system of claim 17, wherein the first boundary indicates a starting and ending point of first media content, and the second boundary indicates a starting and ending point of second media content, wherein the second media content includes the first media content.

19. The system of claim 17, wherein the functions further comprise providing data bounded by the first boundary or by the second boundary for content identification based on a number of the repeated portions exceeding a threshold.

20. The system of claim 17, wherein the functions further comprise:

processing the segment of non-recognized data to extract fingerprint features from the segment of non-recognized data; and

indexing the fingerprint features into a database, wherein the database includes unknown content from media streams.