DETECTING SENTINEL FRAMES IN VIDEO DELIVERY USING A PATTERN ANALYSIS

• US 20170330041A1
• Filed: 05/12/2016
• Published: 11/16/2017
• Est. Priority Date: 05/12/2016
• Status: Active Grant

• Current Assignee: ARRIS Enterprises LLC (CommScope Holding Company, Inc.)
• Original Assignee: ARRIS Enterprises LLC (CommScope Holding Company, Inc.)
• Inventors: Emeott, Stephen P., Li, Renxiang
• Granted Patent: US 11,256,923 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/48 Matching video sequences
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes

Abstract

Particular embodiments extract features for frames from a video. Then, a sequence of frames is identified based on a pattern analysis of frames based on the features. This pattern may be used to select sentinel features from the sequence of frames. For example, a pattern may include a first sentinel sequence followed by a transitional frame and then a second sentinel sequence. The transitional frame may be a black frame that is used to identify back-to-back sentinel sequences. Then, a sentinel frame that demarks a transition from a first content type to a second content type is identified. For example, the frames on either side of the black frame may be very similar and be identified as sentinel frames that mark a transition from program content to advertisement content. The above process may allow automated detection of sentinel frames that can run without user supervision.

Claims

1. A method comprising:

extracting, by a computing device, frame features for a plurality of frames from a video;

identifying, by the computing device, a pattern from a sequence of frames of the plurality of frames, the pattern being identified based on a pattern analysis using the frame features for frames in the sequence of frames;

selecting, by the computing device, sentinel features from the frame features for frames in the sequence of frames based on the pattern; and

outputting, by the computing device, a sentinel frame that identifies a transition in the video from a first content type to a second content type, wherein the sentinel frame includes the sentinel features.

2. The method of claim 1, wherein identifying the pattern comprises:

selecting a transitional frame in the sequence of frames, wherein the transitional frame is different from the sentinel frame; and

using at least one frame in the sequence of frames that is before or after the transitional frame to select the sentinel features.

3. The method of claim 2, wherein selecting the sentinel features comprises using a frame in the sequence of frames on both sides of the transitional frame in the sequence of frames to select the sentinel features.

4. The method of claim 2, wherein identifying the pattern comprises:

comparing a first frame on a first side of the transitional frame to a second frame on a second side of the transitional frame to determine if the first frame and the second frame are similar within a threshold; and

when the first frame and the second frame are similar within the threshold, identifying the sequence of frames as including the pattern.

5. The method of claim 2, further comprising:

grouping sequential transitional frames in the sequence of frames into a group, wherein the at least one frame in the sequence of frames is near the group within a threshold.

6. The method of claim 2, wherein the transitional frame is a solid color frame.

7. The method of claim 1, wherein the pattern comprises a first sequence of frames and a transitional frame.

8. The method of claim 7, further comprising:

locating a second sequence of frames in addition to the first sequence of frames and the transitional frame, wherein;

a first frame in the first sequence of frames matches a second frame in the second sequence of frames within a threshold, and

the sentinel features are selected from the first frame or the second frame.

9. The method of claim 1, wherein the sentinel features comprise features determined from decoding the frames of the video.

10. The method of claim 1, further comprising:

receiving an input identifying the pattern.

11. The method of claim 1, wherein the set of sentinel frames are not previously known before selecting the sentinel features.

12. A method comprising:

extracting, by a computing device, frame features for a plurality of frames from a video;

identifying, by the computing device, locations of first frames in the video using the frame features, the first frames including a first set of frame features;

using, by the computing device, frame features for second frames that are within a threshold distance to the first frames to generate a set of candidate frames;

clustering, by the computing device, the set of candidate frames into one or more groups based on the frame features for the set of candidate frames;

selecting, by the computing device, sentinel features for each of the one or more groups; and

outputting, by the computing device, a set of sentinel frames for each of the one or more groups using the sentinel features, the set of sentinel frames identifying a transition in the video from a first content type to a second content type.

13. The method of claim 12, wherein using the frame features for the second frames comprises:

for each first frame, comparing a second frame on at least one side of the first frame to identify a pattern of the second frame and the first frame.

14. The method of claim 12, wherein using the frame features for the second frames comprises:

for each first frame, comparing a second frame on each side of the first frame to generate a difference between the second frame on each side of the first frames; and

adding the second frame on each side of the first frame to the set of candidate frames if the second frame on each side of the first frame are similar within a threshold.

15. The method of claim 12, wherein clustering comprises using a sequential clustering process.

16. The method of claim 15, wherein the sequential clustering process comprises:

selecting a first candidate frame in the set of candidate frames;

creating a first group of candidate frames that include frame features that are similar to the first candidate frame within a similarity threshold;

generating an average sentinel frame based on an average of the frame features of the first group of candidate frames;

selecting a candidate frame in the first group of candidate frames that is closest to the average sentinel frame; and

including the candidate frame in the set of sentinel frames.

17. The method of claim 12, wherein clustering comprises using a non-sequential clustering process.

18. The method of claim 17, wherein the non-sequential clustering process comprises:

clustering candidate frames in the set of candidate frames into a first set of clusters;

merging one or more pluralities of clusters, from the first set of clusters, that have frame features within a similarity threshold, to generate a second set of clusters; and

selecting a sentinel frame from each of the second set of clusters using a frame feature within each of the second set of clusters.

19. The method of claim 12, wherein the set of sentinel frames are not previously known before selecting the sentinel features.

20. An apparatus comprising:

one or more computer processors; and

a non-transitory computer-readable storage medium comprising instructions that, when executed, control the one or more computer processors to be configured for;

extracting frame features for a plurality of frames from a video;

identifying a pattern from a sequence of frames of the plurality of frames, the pattern being identified based on a pattern analysis using the frame features for frames in the sequence of frames;

selecting sentinel features from the frame features for frames in the sequence of frames based on the pattern; and

outputting a sentinel frame that identifies a transition in the video from a first content type to a second content type, wherein the sentinel frame includes the sentinel features.