Learning-Based Automatic Commercial Content Detection

• US 20070112583A1
• Filed: 01/15/2007
• Published: 05/17/2007
• Est. Priority Date: 02/18/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Li, Mingjing, Hua, Xian-Sheng, Zhang, Hong-Jiang, Lu, Lie
• Granted Patent: US 7,565,016 B2
• Time in Patent Office: Days
• US Class Current: 382/229
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44

Abstract

Systems and methods for learning-based automatic commercial content detection are described. In one aspect, the systems and methods include a training component and an analyzing component. The training component trains a commercial content classification model using a kernel support vector machine. The analyzing component analyzes program data such as video and audio data using the commercial content classification model and one or more of single-side left neighborhood(s) and right neighborhood(s) of program data segments. Based on this analysis, each of the program data segments are classified as being commercial or non-commercial segments.

Claims

1. A computing device comprising:

a processor; and

a memory coupled to the processor, the memory comprising components executable by the processor, the components comprising;

a training component to train a commercial content classification model using a kernel support vector machine;

an analyzing component to analyze program data using the commercial content classification model and one or more of single-side left neighborhood(s) and right neighborhood(s) of program data segments, the analyzing component classifying respective ones of the program data segments as commercial or non-commercial; and

wherein the program data comprises one or more of video and audio data.

2. The computing device of claim 1, wherein the components further comprise a post-processing component to merge multiple classified ones of the program data segments into one or more commercial content blocks and non-commercial content blocks;

and

wherein the analyzing component is further configured to analyze the one or more commercial content blocks and non-commercial content blocks in view of multiple threshold criteria to determine whether segments within respective ones of commercial and non-commercial blocks should be reclassified and merged with a different commercial or non-commercial block.

4. The method of claim 1, wherein the visual feature set comprises segment-based features and time-based features.

5. The method of claim 1, wherein the visual feature set is a function of average of edge change ratio, variance of edge change ratio, average of frame difference, variance of frame difference, shot frequency, and/or black frame rate.

6. The method of claim 1, wherein the audio feature set is determined as a function of one or more of speech, music, silence, background sound and audio break frequency.

7. The method of claim 1, wherein the single-side left and/or right neighborhoods comprise a current segment, the current segment having n neighborhoods left of the current segment and/or n neighborhoods right of the current segment.

8. The method of claim 1, wherein analyzing further comprises:

calculating context-based feature sets from segment-based visual features as an average value of visual features of S^k, S^k representing a set of all segments of the multiple segments that are partially or totally included in the single-side left and/or right neighborhoods such that S^k={C_j^k;

0≦

j<

M^k}={C_i;

C_i∩

N^k≠

Φ

}, M^k being a number of segments in S^k, and

wherein N^k represents 2n+1 neighborhoods, n represents a number of neighborhoods left and/or right of a current segment C_i, S^k is a set of segments that are partially or totally included in N^k, C^k_j represents is a j-th element of S^k, M^k represents a total number of elements in S^k, and Φ

represents an empty set.

9. The method of claim 1, wherein the method further comprises classifying each of the multiple segments as being commercial or non-commercial content as a function of a trained commercial content classification model, and the visual, audio, and context-based feature sets.

10. The method of claim 9, wherein classifying further comprises:

training a Support Vector machine (SVM) classification model with commercial content; and

wherein the trained commercial content classification model comprises the SVM classification model.

11. The method of claim 1, wherein the method further comprises:

classifying each of the multiple segments as being commercial or non-commercial content as a function of a trained commercial content classification model, and the visual, audio, and context-based feature sets; and

post-processing classified commercial and/or non-commercial segments to generate commercial and/or non-commercial blocks, each block comprising one or more segments of the multiple segments.

12. The method of claim 11, wherein post-processing further comprises:

grouping segments into scenes; and

merging scenes into the commercial and/or non-commercial blocks.

13. The method of claim 12, wherein post-processing further comprises evaluating features of the commercial and/or non-commercial blocks to remove short scenes, remove embedded non-commercial content from long commercial scenes, detect long commercial portions in a non-commercial scene, and/or refine the boundaries of commercial and non-commercial scenes and/or segments.

14. The method of claim 12, wherein post-processing further comprises:

responsive to applying the one or more heuristic criteria to remove short scenes, remove embedded non-commercial content from long commercial scenes, detect long commercial portions in a non-commercial scene, and/or refine the boundaries of commercial and non-commercial scenes, re-grouping the segments into scenes, and re-merging scenes into the commercial and/or non-commercial blocks.

3. A computer-implemented method for learning-based automatic commercial content detection, the method comprising:

dividing program data into multiple segments;

analyzing the segments to determine visual, audio, and context-based feature sets that differentiate commercial content from non-commercial content; and

wherein the context-based features are a function of one or more single-side left and/or right neighborhoods of segments of the multiple segments.

15. A tangible computer-readable data storage medium for learning-based automatic commercial content detection, the computer-readable medium comprising computer-program executable instructions executable by a processor for:

dividing program data into multiple segments;

analyzing the segments to determine visual, audio, and context-based feature sets that differentiate commercial content from non-commercial content; and

wherein the context-based features are a function of one or more single-side left and/or right neighborhoods of segments of the multiple segments.

16. The tangible computer-readable data storage medium of claim 15, wherein the visual feature set comprises segment-based features and time-based features.

17. The tangible computer-readable data storage medium of claim 15, wherein the single-side left and/or right neighborhoods comprise a current segment, the current segment having n neighborhoods left of the current segment and/or n neighborhoods right of the current segment.

18. The tangible computer-readable data storage medium of claim 15, wherein the computer-program instructions for analyzing further comprise instructions for:

calculating context-based feature sets from segment-based visual features as an average value of visual features of S^k, S^k representing a set of all segments of the multiple segments that are partially or totally included in the single-side left and/or right neighborhoods such that S^k={C_j^k;

0≦

j<

M^k}={C_i;

C_i∩

N^k≠

Φ

)}, M^k being a number of segments in S^k, and

wherein N^k represents 2n+1 neighborhoods, n represents a number of neighborhoods left and/or right of a current segment C_i, S^k is a set of segments that are partially or totally included in N^k, C^k represents is a j-th element of S^k, M^k represents a total number of elements in S^k, and Φ

represents an empty set.

19. The tangible computer-readable data storage medium of claim 15, wherein the computer-program instructions further comprise instructions for classifying each of the multiple segments as being commercial or non-commercial content as a function of a trained commercial content classification model, and the visual, audio, and context-based feature sets.

20. The tangible computer-readable data storage medium of claim 15, wherein the computer-program instructions further comprise instructions for:

classifying each of the multiple segments as being commercial or non-commercial content as a function of a trained commercial content classification model, and the visual, audio, and context-based feature sets; and

post-processing classified commercial and/or non-commercial segments to generate commercial and/or non-commercial blocks, each block comprising one or more segments of the multiple segments.