Learning-based automatic commercial content detection

• US 20040161154A1
• Filed: 02/18/2003
• Published: 08/19/2004
• Est. Priority Date: 02/18/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Li, Mingjing, Hua, Xian-Sheng, Zhang, Hong-Jiang, Lu, Lie
• Granted Patent: US 7,164,798 B2
• Time in Patent Office: Days
• US Class Current: 382/229
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44

Abstract

Systems and methods for learning-based automatic commercial content detection are described. In one aspect, program data is divided into multiple segments. The segments are analyzed to determine visual, audio, and context-based feature sets that differentiate commercial content from non-commercial content. The context-based features are a function of single-side left and/or right neighborhoods of segments of the multiple segments.

Claims

1. A method for learning-based automatic commercial content detection, the method comprising:

dividing program data into multiple segments;

analyzing the segment to determine visual, audio, and context-based feature sets that differentiate commercial content from non-commercial content; and

wherein the context-based features are a function of one or more single-side left and/or right neighborhoods of segments of the multiple segments.

2. A method as recited in claim 1, wherein the visual feature set comprises segment-based features and time-based features.

3. A method as recited in claim 1, wherein the visual feature set is a function of average of edge change ratio, variance of edge change ratio, average of frame difference, variance of frame difference, shot frequency, and/or black frame rate.

4. A method as recited in claim 1, wherein the audio feature set is determined as a function of one or more of speech, music, silence, background sound and audio break frequency.

5. A method as recited in claim 1, wherein the single-side left and/or right neighborhoods comprise a current segment, n neighborhoods left of the current segment, and n neighborhoods right of the current segment.

6. A method as recited in claim 1, wherein the single-side left and/or right neighborhoods are (2n+1) neighborhoods of a current segment C_i, and wherein the method further comprises:

calculating each of the (2n+1) neighborhoods as follows;

$N^k=\left[N_s^k,N_e^k\right]=\{\begin{array}{c}\left[\min <br><br>\left(e_j+\mathrm{\alpha <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>k,0\right),e_i\right]\\ \left[s_i,e_i\right]\\ \left[s_i,\min <br><br>\left(s_i+\mathrm{\alpha <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>k,L\right)\right]\end{array}<br><br>\begin{array}{c}k<<br><br>0\\ k=0\\ k><br><br>0,\end{array}$

 

n representing a number of neighborhoods left and/or right of C_i, [s_i, e_i] denoting start and end frame numbers for C_i and start and end times for C_i, L indicating a length of the program data, k∈

Z, |k|≦

n, and a comprising a time step.

7. A method as recited in claim 1, wherein analyzing further comprises:

calculating context-based feature sets from segment-based visual features as an average value of visual features of S^k, S^k representing a set of all segments of the multiple segments that are partially or totally included in the single-side left and/or right neighborhoods such that S^k={C_j^k;

0≦

j<

M^k}={C_i;

C_i∩

N^k≠

Φ

}, M^k being a number of segments in S^k.

8. A method as recited in claim 1, wherein the method further comprises classifying each of the multiple segments as being commercial or non-commercial content as a function of a trained commercial content classification model, and the visual, audio, and context-based feature sets.

9. A method as recited in claim 8, wherein classifying further comprises:

training a Support Vector machine (SVM) classification model with commercial content; and

wherein the trained commercial content classification model comprises the SVM classification model.

10. A method as recited in claim 1, wherein the method further comprises:

classifying each of the multiple segments as being commercial or non-commercial content as a function of a trained commercial content classification model, and the visual, audio, and context-based feature sets; and

post-processing classified commercial and/or non-commercial segments to generate commercial and/or non-commercial blocks, each block comprising one or more segments of the multiple segments.

11. A method as recited in claim 10, wherein post-processing further comprises:

grouping segments into scenes; and

merging scenes into the commercial and/or non-commercial blocks.

12. A method as recited in claim 11, wherein post-processing further comprises evaluating features of the commercial and/or non-commercial blocks to remove short scenes, remove embedded non-commercial content from long commercial scenes, detect long commercial portions in a non-commercial scene, and/or refine the boundaries of commercial and non-commercial scenes and/or segments.

13. A method as recited in claim 11, wherein post-processing further comprises:

responsive to applying the one or more heuristic criteria to remove short scenes, remove embedded non-commercial content from long commercial scenes, detect long commercial portions in a non-commercial scene, and/or refine the boundaries of commercial and non-commercial scenes, re-grouping the segments into scenes, and re-merging scenes into the commercial and/or non-commercial blocks.

14. A computer-readable medium for learning-based automatic commercial content detection, the computer-readable medium comprising computer-program executable instructions executable by a processor for:

dividing program data into multiple segments;

analyzing the segments to determine visual, audio, and context-based feature sets that differentiate commercial content from non-commercial content; and

wherein the context-based features are a function of one or more single-side left and/or right neighborhoods of segments of the multiple segments.

15. A computer-readable medium as recited in claim 14, wherein the visual feature set comprises segment-based features and time-based features.

16. A computer-readable medium as recited in claim 14, wherein the visual feature set is a function of average of edge change ratio, variance of edge change ratio, average of frame difference, variance of frame difference, shot frequency, and/or black frame rate.

17. A computer-readable medium as recited in claim 14, wherein the audio feature set is determined as a function of one or more of speech, music, silence, background sound and audio break frequency.

18. A computer-readable medium as recited in claim 14, wherein the single-side left and/or right neighborhoods comprise a current segment, n neighborhoods left of the current segment, and n neighborhoods right of the current segment.

19. A computer-readable medium as recited in claim 14, wherein the single-side left and/or right neighborhoods are (2n+1) neighborhoods of a current segment C_i, and wherein the computer-program instructions further comprise instructions for:

calculating each of the (2n+1) neighborhoods as follows;

$N^k=\left[N_s^k,N_e^k\right]=\{\begin{array}{c}\left[\min <br><br>\left(e_j+\mathrm{\alpha <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>k,0\right),e_i\right]\\ \left[s_i,e_i\right]\\ \left[s_i,\min <br><br>\left(s_i+\mathrm{\alpha <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>k,L\right)\right]\end{array}<br><br>\begin{array}{c}k<<br><br>0\\ k=0\\ k><br><br>0,\end{array}$

n representing a number of neighborhoods left and/or right of C_i, [s_i, e_i]denoting start and end frame numbers for C_i and start and end times for C_i, L indicating a length of the program data, k∈

Z, |k|≦

n, and a comprising a time step.

20. A computer-readable medium as recited in claim 14, wherein the computer-program instructions for analyzing further comprise instructions for:

calculating context-based feature sets from segment-based visual features as an average value of visual features of S^k, S_k representing a set of all segments of the multiple segments that are partially or totally included in the single-side left and/or right neighborhoods such that S^k={C_j^k;

0≦

j<

M^k}={C_i;

C_i∩

N^k≠

Φ

}, M^k being a number of segments in S^k.

21. A computer-readable medium as recited in claim 14, wherein the computer-program instructions further comprise instructions for classifying each of the multiple segments as being commercial or non-commercial content as a function of a trained commercial content classification model, and the visual, audio, and context-based feature sets.

22. A computer-readable medium as recited in claim 21, wherein the computer-program instructions for classifying further comprise instructions for:

training a Support Vector machine (SVM) classification model with commercial content; and

wherein the trained commercial content classification model comprises the SVM classification model.

23. A computer-readable medium as recited in claim 14, wherein the computer-program instructions further comprise instructions for:

classifying each of the multiple segments as being commercial or non-commercial content as a function of a trained commercial content classification model, and the visual, audio, and context-based feature sets; and

post-processing classified commercial and/or non-commercial segments to generate commercial and/or non-commercial blocks, each block comprising one or more segments of the multiple segments.

24. A computer-readable medium as recited in claim 23, wherein the computer-program instructions for post-processing further comprise instructions for:

grouping segments into scenes; and

merging scenes into the commercial and/or non-commercial blocks.

25. A computer-readable medium as recited in claim 24, wherein post-processing further comprises:

determining whether to apply one or more heuristic criteria to remove short scenes, double check long commercial scenes, detect long commercial portions of a non-commercial scene, and/or refine the boundaries of commercial and non-commercial scenes; and

responsive to applying the one or more heuristic criteria, re-grouping the segments into scenes, and re-merging scenes into the commercial and/or non-commercial blocks.

26. A computing device for learning-based automatic commercial content detection, the computing device comprising:

a processor; and

a memory coupled to the processor, the memory comprising computer-program executable instructions executable by the processor for;

dividing program data into multiple segments;

analyzing the segments to determine visual, audio, and context-based feature sets that differentiate commercial content from non-commercial content; and

wherein the context-based features are a function of one or more single-side left and/or right neighborhoods of segments of the multiple segments.

27. A computing device as recited in claim 26, wherein the visual feature set comprises segment-based features and time-based features.

28. A computing device as recited in claim 26, wherein the visual feature set is a function of average of edge change ratio, variance of edge change ratio, average of frame difference, variance of frame difference, shot frequency, and/or black frame rate.

29. A computing device as recited in claim 26, wherein the audio feature set is determined as a function of one or more of speech, music, silence, background sound, and audio break frequency.

30. A computing device as recited in claim 26, wherein the single-side left and/or right neighborhoods comprise a current segment, n neighborhoods left of the current segment, and n neighborhoods right of the current segment.

31. A computing device as recited in claim 26, wherein the single-side left and/or right neighborhoods are (2n+1) neighborhoods of a current segment C_i, and wherein the computer-program instructions further comprise instructions for:

calculating each of the (2n+1) neighborhoods as follows;

$N^k=\left[N_s^k,N_e^k\right]=\{\begin{array}{c}\left[\min <br><br>\left(e_j+\mathrm{\alpha <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>k,0\right),e_i\right]\\ \left[s_i,e_i\right]\\ \left[s_i,\min <br><br>\left(s_i+\mathrm{\alpha <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>k,L\right)\right]\end{array}<br><br>\begin{array}{c}k<<br><br>0\\ k=0\\ k><br><br>0,\end{array}$

 

n representing a number of neighborhoods left and/or right of C_i, [s_i, e_i]denoting start and end frame numbers for C_i and start and end times for C_i, L indicating a length of the program data, k∈

Z,|k|≦

n, and α

comprising a time step.

32. A computing device as recited in claim 26, wherein the computer-program instructions for analyzing further comprise instructions for:

calculating context-based feature sets from segment-based visual features as an average value of visual features of S^k, S^k representing a set of all segments of the multiple segments that are partially or totally included in the single-side left and/or right neighborhoods such that S^k{C_j^k;

0≦

j<

M^k}={C_i;

C<sub>i∩

N</sub>^k≠

Φ

}, M^k being a number of segments in S^k.

33. A computing device as recited in claim 26, wherein the computer-program instructions further comprise instructions for classifying each of the multiple segments as being commercial or non-commercial content as a function of a trained commercial content classification model, and the visual, audio, and context-based feature sets.

34. A computing device as recited in claim 33, wherein the computer-program instructions for classifying further comprise instructions for:

training a Support Vector machine (SVM) classification model with commercial content; and

wherein the trained commercial content classification model comprises the SVM classification model.

35. A computing device as recited in claim 26, wherein the computer-program instructions further comprise instructions for:

classifying each of the multiple segments as being commercial or non-commercial content as a function of a trained commercial content classification model, and the visual, audio, and context-based feature sets; and

post-processing classified commercial and/or non-commercial segments to generate commercial and/or non-commercial blocks, each block comprising one or more segments of the multiple segments.

36. A computing device as recited in claim 35, wherein the computer-program instructions for post-processing further comprise instructions for:

grouping segments into scenes; and

merging scenes into the commercial and/or non-commercial blocks.

37. A computing device as recited in claim 36, wherein post-processing further comprises:

determining whether to apply one or more heuristic criteria to remove short scenes, double check long commercial scenes, detect long commercial portions of a non-commercial scene, and/or refine the boundaries of commercial and non-commercial scenes; and

responsive to applying the one or more heuristic criteria, re-grouping the segments into scenes, and re-merging scenes into the commercial and/or non-commercial blocks.

38. A computing device for learning-based automatic commercial content detection, the computing device comprising:

means for dividing program data into multiple segments;

means for analyzing the segments to determine visual, audio, and context-based feature sets that differentiate commercial content from non-commercial content; and

wherein the context-based features are a function of one or more single-side left and/or right neighborhoods of segments of the multiple segments.

39. A computing device as recited in claim 38, wherein the computing device is a set-top box, a digital video recorder, or a general purpose computing device.

40. A computing device as recited in claim 38, wherein the visual feature set comprises segment-based features and time-based features.

41. A computing device as recited in claim 38, wherein the visual feature set is a function of average of edge change ratio, variance of edge change ratio, average of frame difference, variance of frame difference, shot frequency, and/or black frame rate.

42. A computing device as recited in claim 38, wherein the audio feature set is determined as a function of one or more of speech, music, silence, background sound and audio break frequncy.

43. A computing device as recited in claim 38, wherein the single-side left and/or right neighborhoods comprise a current segment, n neighborhoods left of the current segment, and n neighborhoods right of the current segment.

44. A computing device as recited in claim 38, wherein the single-side left and/or right neighborhoods are (2n+1) neighborhoods of a current segment C_i, and wherein the computing device further comprises:

means for calculating each of the (2n+1) neighborhoods as follows;

$N^k=\left[N_s^k,N_e^k\right]=\{\begin{array}{c}\left[\min <br><br>\left(e_j+\mathrm{\alpha <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>k,0\right),e_i\right]\\ \left[s_i,e_i\right]\\ \left[s_i,\min <br><br>\left(s_i+\mathrm{\alpha <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>k,L\right)\right]\end{array}<br><br>\begin{array}{c}k<<br><br>0\\ k=0\\ k><br><br>0,\end{array}$

 

n representing a number of neighborhoods left and/or right of C_i, [s_i,e_i] denoting start and end frame numbers for C_i and start and end times for C_i, L indicating a length of the program data, k∈

Z, |k|≦

n, and α

comprising a time step.

45. A computing device as recited in claim 38, wherein the computing device further comprises means for calculating context-based feature sets from segment-based visual features as an average value of visual features of S^k, S^k representing a set of all segments of the multiple segments that are partially or totally included in the single-side left and/or right neighborhoods such that S^k={C_j^k=0≦

j<

M^k}={C_i;

C_i∩

N^k≠

Φ

}, M^k being a number of segments in S^k.

46. A computing device as recited in claim 38, wherein the computing device further comprises means for classifying each of the multiple segments as being commercial or non-commercial content as a function of a trained commercial content classification model, and the visual, audio, and context-based feature sets.

47. A computing device as recited in claim 45, wherein the means for classifying further comprise:

means for training a Support Vector machine (SVM) classification model with commercial content; and

wherein the trained commercial content classification model comprises the SVM classification model.

48. A computing device as recited in claim 38, wherein the computing device further comprises:

means for classifying each of the multiple segments as being commercial or non-commercial content as a function of a trained commercial content classification model, and the visual, audio, and context-based feature sets; and

means for post-processing classified commercial and/or non-commercial segments to generate commercial and/or non-commercial blocks, each block comprising one or more segments of the multiple segments.

49. A computing device as recited in claim 48, wherein the means for post-processing further comprise:

means for grouping segments into scenes; and

means for merging scenes into the commercial and/or non-commercial blocks.

50. A computing device as recited in claim 50, wherein the means for post-processing further comprises:

means for determining whether to apply one or more heuristic criteria to remove short scenes, double check long commercial scenes, detect long commercial portions of a non-commercial scene, and/or refine the boundaries of commercial and non-commercial scenes; and

responsive to applying the one or more heuristic criteria, means for re-grouping the segments into scenes, and re-merging scenes into the commercial and/or non-commercial blocks.