Selecting and Presenting Representative Frames for Video Previews

• US 20160070962A1
• Filed: 09/08/2015
• Published: 03/10/2016
• Est. Priority Date: 09/08/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Shetty, Sanketh, Izo, Tomas, Vijayanarasimhan, Sudheendra, Toderici, George Dan, Muscettola, Nicola, Tsai, Min-Hsuan, Natsev, Apostol, Abu-El-Haija, Sami, Ricco, Susanna, Varadarajan, Balakrishnan, Gu, WeiHsin, Yang, Weilong, Khandelwal, Nitin, Le, Phuong
• Granted Patent: US 9,953,222 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata

Abstract

A computer-implemented method for selecting representative frames for videos is provided. The method includes receiving a video and identifying a set of features for each of the frames of the video. The features including frame-based features and semantic features. The semantic features identifying likelihoods of semantic concepts being present as content in the frames of the video. A set of video segments for the video is subsequently generated. Each video segment includes a chronological subset of frames from the video and each frame is associated with at least one of the semantic features. The method generates a score for each frame of the subset of frames for each video segment based at least on the semantic features, and selecting a representative frame for each video segment based on the scores of the frames in the video segment. The representative frame represents and summarizes the video segment.

Claims

1. A computer-implemented method for selecting representative frames for videos, comprising:

receiving a video including a set of frames;

identifying a set of features for each of the frames of the video, the features including frame-based features and semantic features, the semantic features identifying likelihoods of semantic concepts being present as content in the frames of the video;

generating a set of video segments for the video, each video segment including a chronological subset of frames from the video, and each frame associated with at least one of the semantic features;

generating a score for each frame of the subset of frames for each video segment based at least on the semantic features; and

selecting a representative frame for each video segment based on the scores of the frames in the video segment, the representative frame representing and summarizing the video segment.

2. The method of claim 1, further comprising generating a segment table for the video, the segment table storing the representative frames for the video segments of the video and the set of semantic concepts associated with each of the representative frames.

3. The computer-implemented method of claim 1, wherein the frame-based features comprise at least one of:

a visual feature comprising at least one from the group consisting of a color histogram, a histogram of oriented gradients, color-differencing of a frame with adjacent frames, motion features, or feature tracking, and;

an audio feature comprising at least one from the group consisting of a volume, an audio spectrogram, a speech-no-speech indicator, or a stabilized auditory image.

4. The computer-implemented method of claim 1, wherein the step of identifying the set of features comprises:

applying a plurality of semantic classifiers to determine the semantic features,

wherein a semantic classifier receives a designation of a frame associated with frame-based features and outputs a likelihood the frame is relevant to or depicts a semantic concept, and

wherein the semantic concept is a label assigned to the content of the video and corresponds to an entity or free text.

5. The computer-implemented method of claim 1, wherein the step of generating the set of video segments comprises:

analyzing the frame-based features to determine a set of shot boundaries within the video;

wherein a shot includes a set of sequential frames and a shot boundary indicates a frame between neighboring shots and.

6. The computer-implemented method of claim 5, wherein the step of determining the set of shot boundaries comprises:

applying a classifier to frames associated with frame-based features to determine whether a frame is a shot boundary;

wherein the classifier is trained using labeled shot boundaries as a positive feature set and frames near the shot boundaries as a hard-negative training set, and

wherein the frame-based features comprise color differences with adjacent frames, motion features, audio volume, and audio speech detection.

7. The computer-implemented method of claim 5, wherein the step of determining the set of shot boundaries comprises:

analyzing a coherence of the frame-based features;

wherein the coherence measures similarity of frame-based features in a pre-determined temporal segment, and

wherein the similarity provides a distance measure for segmenting the video.

8. The computer-implemented method of claim 5, wherein the step of determining the set of shot boundaries comprises:

tracking the frame-based features across the series of frames of the video; and

wherein a frame is determined as a shot boundary when a change of frame-based features between the frame and neighboring frames is greater than a threshold.

9. The computer-implemented method of claim 1, wherein the score comprises a semantic score and the step of generating the semantic score for the frame comprises:

identifying a set of semantic concepts for a video segment containing the frame by comparing each semantic feature generated for the chronological subset of frames included in the video segment to a threshold, each semantic concept of the set having the corresponding semantic feature greater than the threshold; and

for each semantic concept of the set, determining a frame-level score for each frame of the chronological subset of frames in the video segment by determining an amount the semantic concept being present in the frame compared to a reference value; and

determining the semantic score for the frame by aggregating the frame-level scores of the frames in the segment.

10. The computer-implemented method of claim 1, wherein the step of generating the score for the each frame comprises combining semantic concepts and corresponding likelihood in the frame.

11. The computer-implemented method of claim 1, wherein the score combines a semantic score and an aesthetic score and the step of generating the score for the each frame comprises:

calculating the semantic score based on the determined semantic features;

calculating the aesthetic score using a set of quality measures comprising at least one from the group consisting of sharpness, an amount of motion, distance from a segment boundary, and photo quality; and

combining the semantic score and the aesthetic score.

12. The computer-implemented method of claim 1, further comprising presenting the representative frame of a video segment responsive to receiving a request for the video segment.

13. A non-transitory computer-readable storage medium comprising computer program instructions executable by a processor, the computer program instructions comprising:

receiving a video including a set of frames;

identifying a set of features for each of the frames of the video, the features including frame-based features and semantic features, the semantic features identifying likelihoods of semantic concepts being present as content in the frames of the video;

generating a set of video segments for the video, each video segment including a chronological subset of frames from the video, and each frame associated with at least one of the semantic features;

generating a score for each frame of the subset of frames for each video segment based at least on the semantic features; and

selecting a representative frame for each video segment based on the scores of the frames in the video segment, the representative frame representing and summarizing the video segment.

14. The non-transitory computer-readable storage medium of claim 13, wherein the computer program instructions further comprise generating a segment table for the video, the segment table storing the representative frames for the video segments of the video and the set of semantic concepts associated with each of the representative frames.

15. The computer program product of claim 13, wherein the step of identifying the set of features comprises:

applying a plurality of semantic classifiers to determine the semantic features,

wherein a semantic classifier receives a designation of a frame associated with frame-based features and outputs a likelihood the frame is relevant to or depicts a semantic concept, and

wherein the semantic concept is a label assigned to the content of the video and corresponds to an entity or free text.

16. The computer program product of claim 13, wherein the step of generating the set of video segments comprises:

analyzing the frame-based features to determine a set of shot boundaries within the video;

wherein a shot includes a set of sequential frames and a shot boundary indicates a frame between neighboring shots and.

17. The computer program product of claim 16, wherein the step of determining the set of shot boundaries comprises:

applying a classifier to frames associated with frame-based features to determine whether a frame is a shot boundary;

wherein the classifier is trained using labeled shot boundaries as a positive feature set and frames near the shot boundaries as a hard-negative training set, and

wherein the frame-based features comprise color differences with adjacent frames, motion features, audio volume, and audio speech detection.

18. The computer program product of claim 16, wherein the step of determining shot boundaries comprises:

analyzing a coherence of the frame-based features;

wherein the coherence measures similarity of frame-based features in a pre-determined temporal segment, and

wherein the similarity provides a distance measure for segmenting the video.

19. The computer program product of claim 13, wherein the score comprises a semantic score and the step of generating the semantic score for the frame comprises:

identifying a set of semantic concepts for a video segment containing the frame by comparing each semantic feature generated for the chronological subset of frames included in the video segment to a threshold, each semantic concept of the set having the corresponding semantic feature greater than the threshold; and

for each semantic concept of the set, determining a frame-level score for each frame of the chronological subset of frames in the video segment by determining an amount the semantic concept being present in the frame compared to a reference value; and

determining the semantic score for the frame by aggregating the frame-level scores of the frames in the segment.

20. A system comprising:

a processor for executing computer program instructions; and

a non-transitory computer-readable storage medium comprising computer program instructions executable by the processor, the computer program instructions comprising;

receiving a video including a set of frames;

identifying a set of features for each of the frames of the video, the features including frame-based features and semantic features, the semantic features identifying likelihoods of semantic concepts being present as content in the frames of the video;

generating a set of video segments for the video, each video segment including a chronological subset of frames from the video, and each frame associated with at least one of the semantic features;

generating a score for each frame of the subset of frames for each video segment based at least on the semantic features; and

selecting a representative frame for each video segment based on the scores of the frames in the video segment, the representative frame representing and summarizing the video segment.