Commercial detection based on audio fingerprinting

• US 9,258,604 B1
• Filed: 11/24/2014
• Issued: 02/09/2016
• Est. Priority Date: 11/24/2014
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Bilobrov, Sergiy, Poutivski, Iouri
• Primary Examiner(s): IDOWU, OLUGBENGA O
• Application Number:
  US14/552,039
• Time in Patent Office: 442 Days
• Field of Search: 725 18- 19
• US Class Current: 1/1
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4586 Content update operation triggered locally, e.g. by comparing the version of software modules in a DVB carousel to the version stored locally deployment, distribution, installation, update of software G06F8/65; error detection or correction of the data by redundancy during software upgrading G06F11/1433; arrangements for updating broadcast information or broadcast-related information H04H60/25
• CPC Class Codes: H04N 21/466 Learning process for intelligent management, e.g. learning user preferences for recommending movies services using the results of monitoring in broadcast systems H04H60/61
• CPC Class Codes: H04N 21/4665 involving classification methods, e.g. Decision trees
• CPC Class Codes: H04N 21/482 End-user interface for program selection broadcast systems using EPGs H04H60/72
• CPC Class Codes: H04N 21/4821 using a grid, e.g. sorted out by channel and broadcast time
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02
• CPC Class Codes: H04N 21/8358 involving watermark protecting executable software by watermarking G06F21/16; image watermarking in general G06T1/0021; watermarks inserted in still images for transmission purposes H04N1/32144; inserting watermarks during video coding H04N19/467

Abstract

A commercial detection system generates a probe audio fingerprint of an audio signal associated with a media stream on a broadcast channel and determines whether the media stream has commercial content based on analysis of the probe audio fingerprint. The commercial detection system determines whether a same match between the probe audio fingerprint and a reference audio fingerprint is observed across multiple broadcast channels. Responsive to the number of same matches exceeding a predetermined threshold, the commercial detection system determines that the media stream has commercial content. The commercial detection system may also apply a trained feature analysis model to extracted acoustic features of the audio signal. The commercial detection system determines whether the media stream has commercial content based on a confidence score assigned to the probe audio fingerprint. The commercial detection system reduces false positive detection using program guide information of the media stream.

Claims

1. A computer-implemented method comprising:

receiving an audio signal including a plurality of frames, each frame representing a portion of the audio signal, the audio signal associated with a media stream on a broadcast channel;

generating a probe audio fingerprint based on one or more of the plurality of frames;

comparing the probe audio fingerprint with one or more reference audio fingerprints from a plurality of reference audio fingerprints, a reference audio fingerprint of the plurality of reference audio fingerprints representing a set of acoustic features of an audio signal associated with commercial content;

determining whether the probe audio fingerprint matches a reference audio fingerprint of the plurality of reference audio fingerprints based on comparison of the probe audio fingerprint and the reference audio fingerprint; and

responsive to a match between the probe audio fingerprint and the reference audio fingerprint of the plurality of reference audio fingerprints;

determining whether an audio fingerprint associated with a media stream broadcast over a channel of a plurality of other broadcast channels matches the reference audio fingerprint; and

determining that the media stream associated with the probe audio fingerprint has commercial content in response to the number of matches between the audio fingerprints associated with the plurality of other broadcast channels and the reference audio fingerprint exceeding a predetermined threshold.

2. The computer-implemented method of claim 1, further comprising:

responsive to a match between the probe audio fingerprint and the reference audio fingerprint of the plurality of reference audio fingerprints;

determining whether an audio fingerprint associated with a media stream of a plurality of other media streams matches the reference audio fingerprint;

determining that the media stream associated with the probe audio fingerprint has commercial content in response to the number of matches between the audio fingerprints associated with the plurality of other media streams and the reference audio fingerprint exceeding a predetermined threshold.

3. The computer-implemented method of claim 1, further comprising:

receiving program guide information associated with the media stream on the broadcast channel, the program guide information describing scheduling information of known commercials broadcast along with the media stream on the broadcast channel; and

verifying the commercial content determination based on the program guide information.

4. The computer-implemented method of claim 1, further comprising:

receiving program guide information associated with other media streams on the plurality of other broadcast channels, the program guide information describing scheduling information of known commercials broadcast along with the other media streams on the other broadcast channels.

5. The computer-implemented method of claim 1, further comprising:

extracting a plurality of acoustic features from the audio signal associated with the media stream;

applying a trained analysis model to the extracted acoustic features of the audio signal;

assigning a confidence score to the probe audio signal based on the analysis of the extracted acoustic features of the media stream;

detecting commercial content in the media stream based on the confidence score assigned to the probe audio signal.

6. The computer-implemented method of claim 5, wherein the plurality of acoustic features of the audio signal comprise a plurality of acoustic features of the audio signal in frequency domain comprising:

magnitude spectrum of the audio signal;

Mel-frequency cepstral coefficients of the audios signal;

spectral bandwidth of the audio signal;

spectral flatness measure of the audio signal;

spectral fluctuation of the audio signal;

number of extreme value frequencies; and

number of silent frequencies.

7. The computer-implemented method of claim 5, wherein the plurality of acoustic features of the audio signal comprise a plurality of acoustic features of the audio signal in time domain comprising mean, standard deviation and covariance matrix of features vectors of the audio signal, the feature vectors of the audio signal being observed over a predefined time window.

8. The computer-implemented method of claim 5, wherein the confidence score assigned to the probe audio fingerprint indicates a likelihood that the media stream associated with the probe audio fingerprint has commercial content.

9. The computer-implemented method of claim 5, wherein assigning a confidence score to the probe audio signal comprises:

determining a degree of similarities between the acoustic features associated with the probe audio fingerprint and the acoustic features of reference audio fingerprints of known commercial content.

10. A non-transitory computer-readable storage medium storing computer program instructions, executed by a computer processor, for detecting commercial in a media stream, the computer program instructions comprising instructions for:

receiving an audio signal including a plurality of frames, each frame representing a portion of the audio signal, the audio signal associated with a media stream on a broadcast channel;

generating a probe audio fingerprint based on one or more of the plurality of frames;

comparing the probe audio fingerprint with one or more reference audio fingerprints from a plurality of reference audio fingerprints, a reference audio fingerprint of the plurality of reference audio fingerprints representing a set of acoustic features of an audio signal associated with commercial content;

determining whether the probe audio fingerprint matches a reference audio fingerprint of the plurality of reference audio fingerprints based on comparison of the probe audio fingerprint and the reference audio fingerprint; and

responsive to a match between the probe audio fingerprint and the reference audio fingerprint of the plurality of reference audio fingerprints;

determining whether an audio fingerprint associated with a media stream broadcast over a channel of a plurality of other broadcast channels matches the reference audio fingerprint; and

determining that the media stream associated with the probe audio fingerprint has commercial content in response to the number of matches between the audio fingerprints associated with the plurality of other broadcast channels and the reference audio fingerprint exceeding a predetermined threshold.

11. The computer-readable storage medium of claim 10, further comprising computer program instructions for:

responsive to a match between the probe audio fingerprint and the reference audio fingerprint of the plurality of reference audio fingerprints;

determining whether an audio fingerprint associated with a media stream of a plurality of other media streams matches the reference audio fingerprint;

determining that the media stream associated with the probe audio fingerprint has commercial content in response to the number of matches between the audio fingerprints associated with the plurality of other media streams and the reference audio fingerprint exceeding a predetermined threshold.

12. The computer-readable storage medium of claim 10, further comprising computer program instructions for:

receiving program guide information associated with the media stream on the broadcast channel, the program guide information describing scheduling information of known commercials broadcast along with the media stream on the broadcast channel; and

verifying the commercial content determination based on the program guide information.

13. The computer-readable storage medium of claim 10, further comprising computer program instructions for:

receiving program guide information associated with other media streams on the plurality of other broadcast channels, the program guide information describing scheduling information of known commercials broadcast along with the other media streams on the other broadcast channels.

14. The computer-readable storage medium of claim 10, further comprising computer program instructions for:

extracting a plurality of acoustic features from the audio signal associated with the media stream;

applying a trained analysis model to the extracted acoustic features of the audio signal;

assigning a confidence score to the probe audio signal based on the analysis of the extracted acoustic features of the media stream;

detecting commercial content in the media stream based on the confidence score assigned to the probe audio signal.

15. The computer-readable storage medium of claim 14, wherein the plurality of acoustic features of the audio signal comprise a plurality of acoustic features of the audio signal in frequency domain comprising:

magnitude spectrum of the audio signal;

Mel-frequency cepstral coefficients of the audios signal;

spectral bandwidth of the audio signal;

spectral flatness measure of the audio signal;

spectral fluctuation of the audio signal;

number of extreme value frequencies; and

number of silent frequencies.

16. The computer-readable storage medium of claim 14, wherein the plurality of acoustic features of the audio signal comprise a plurality of acoustic features of the audio signal in time domain comprising mean, standard deviation and covariance matrix of features vectors of the audio signal, the feature vectors of the audio signal being observed over a predefined time window.

17. The computer-readable storage medium of claim 14, wherein the confidence score assigned to the probe audio fingerprint indicates a likelihood that the media stream associated with the probe audio fingerprint has commercial content.

18. The computer-readable storage medium of claim 14, wherein the computer program instructions for assigning a confidence score to the probe audio signal comprise instructions for:

determining a degree of similarities between the acoustic features associated with the probe audio fingerprint and the acoustic features of reference audio fingerprints of known commercial content.