Systems and Methods for Semantically Classifying and Extracting Shots in Video

• US 20130259375A1
• Filed: 04/03/2012
• Published: 10/03/2013
• Est. Priority Date: 02/15/2008
• Status: Active Grant

• Current Assignee: Heather Dunlop, Matthew Berry
• Original Assignee: TiVo Inc. (Adeia Inc.)
• Inventors: Dunlop, Heather, Berry, Matthew
• Granted Patent: US 9,020,263 B2
• Time in Patent Office: Days
• US Class Current: 382/173
• CPC Class Codes: G06V 10/464 using a plurality of salient features, e.g. bag-of-words [BoW] representations
• CPC Class Codes: G06V 10/993 Evaluation of the quality of the acquired pattern
• CPC Class Codes: G06V 20/35 Categorising the entire scene, e.g. birthday party or wedding scene
• CPC Class Codes: G06V 20/38 Outdoor scenes
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames

Abstract

The present disclosure relates to systems and methods for classifying videos based on video content. For a given video file including a plurality of frames, a subset of frames is extracted for processing. Frames that are too dark, blurry, or otherwise poor classification candidates are discarded from the subset. Generally, material classification scores that describe type of material content likely included in each frame are calculated for the remaining frames in the subset. The material classification scores are used to generate material arrangement vectors that represent the spatial arrangement of material content in each frame. The material arrangement vectors are subsequently classified to generate a scene classification score vector for each frame. The scene classification results are averaged (or otherwise processed) across all frames in the subset to associate the video file with one or more predefined scene categories related to overall types of scene content of the video file.

Claims

1. A method for classifying videos based on video content, comprising the steps of:

receiving a video file, the video file including a plurality of frames, where each frame includes a plurality of pixels;

extracting a set of frames from the video file;

for each frame in the extracted set of frames, determining whether the frame comprises a poor classification frame, removing the one or more poor classification frames from the extracted set of frames;

dividing each frame in the extracted set of frames into one or more segments, where each segment includes relatively uniform image content;

extracting image features from each segment to form a feature vector associated with each segment;

generating a material classification score vector for each segment via one or more material classifiers based on the feature vector associated with each segment, where each material classification score vector includes one or more material classification scores associated with one or more predefined material content categories; and

assigning each material classification score vector associated with its respective segment to each pixel in each respective segment for each respective frame in the set of frames.

2. The method of claim 1, further comprising the step of storing the material classification score vectors assigned to each pixel in a database for subsequent use in video file classifications.

3. The method of claim 1, further comprising the step of combining adjacent segments based on similar image content features extracted from the segments.

4. The method of claim 1, where poor classification frames are determined via one or more classifiers.

5. The method of claim 1, where a poor classification frame comprises a frame associated with at least one of the following frame types:

a frame shot in low light,

a frame shot at night,

a blurry frame, and

an undetermined frame.

6. The method of claim 1, where image features comprise one or more features selected from the group comprising:

color features, edge features, texture features, and shape features.

7. The method of claim 1, where image features comprise data associated with image content.

8. The method of claim 1, where one or more material classifiers are hierarchically related.

9. The method of claim 1, where the one or more predefined material content categories are selected from the group comprising:

building, grass, person, road/sidewalk, rock, sand/gravel/soil, sky/clouds, now/ice, trees/plants, vehicle, water, and miscellaneous.

10. The method of claim 1, where each of the one or more material classification scores represents the probability that a frame includes content associated with each of the respective predefined material content categories.

11. The method of claim 1, where the video file comprises a shot of video.