Systems and Methods for Semantically Classifying and Normalizing Shots in Video

• US 20130259390A1
• Filed: 04/03/2012
• Published: 10/03/2013
• Est. Priority Date: 02/15/2008
• Status: Active Grant

• Current Assignee: TiVo Solutions Inc. (Adeia Inc.)
• Original Assignee: TiVo Inc. (Adeia Inc.)
• Inventors: Dunlop, Heather, Berry, Matthew
• Granted Patent: US 9,111,146 B2
• Time in Patent Office: Days
• US Class Current: 382/224
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content

Abstract

The present disclosure relates to systems and methods for classifying videos based on video content. For a given video file including a plurality of frames, a subset of frames is extracted for processing. Frames that are too dark, blurry, or otherwise poor classification candidates are discarded from the subset. Generally, material classification scores that describe type of material content likely included in each frame are calculated for the remaining frames in the subset. The material classification scores are used to generate material arrangement vectors that represent the spatial arrangement of material content in each frame. The material arrangement vectors are subsequently classified to generate a scene classification score vector for each frame. The scene classification results are averaged (or otherwise processed) across all frames in the subset to associate the video file with one or more predefined scene categories related to overall types of scene content of the video file.

Claims

1. A method for classifying a video file according to one or more scene classes, the video file including a plurality of frames, where each frame includes a plurality of pixels, and where each pixel is associated with a vector of material classification scores describing the material content in its respective frame, comprising the steps of:

(a) dividing each frame into a plurality of grid cells;

(b) for each frame, retrieving the vector of material classification scores for each pixel in each grid cell;

(c) for each grid cell, averaging the material classification scores across each pixel in the grid cell to form a material occurrence vector for the grid cell;

(d) concatenating the material occurrence vectors for the plurality of grid cells in each frame to generate a material arrangement vector for each frame;

(e) generating a scene classification score associated with each of the one or more scene classes for each frame in the video file via one or more scene classifiers based on the material arrangement vectors generated for each frame;

(f) generating a representative scene classification score for the video file for each of the one or more scene classes based on the scene classification scores generated for each frame; and

(g) if one of more of the representative scene classification scores is above a predetermined threshold value, labeling the video file according to the respective scene classes associated with the one or more scene classification scores that are above the predetermined threshold value.

2. The method of claim 1, further comprising the step of repeating steps (a)-(e) one or more times using varying numbers of grid cells.

3. The method of claim 2, where the one or more scene classifiers comprise a spatial pyramid of classifiers, and where varying weights are associated with the scene classification scores.

4. The method of claim 1, where the material content is selected from the group comprising:

building, grass, person, road/sidewalk, rock, sand/gravel/soil, sky/clouds, snow/ice, trees/plants, vehicle, water, and miscellaneous.

5. The method of claim 1, where the material occurrence vector represents the proportion of each type of material content included in the cell.

6. The method of claim 1, where the material occurrence vectors are concatenated in an ordered manner.

7. The method of claim 1, where each of the material classification scores represents the probability that a frame includes content associated with each of the respective types of material content.

8. The method of claim 1, where each of the one or more scene classification scores represents the probability that a frame includes content associated with each of the respective one or more scene classes.

9. The method of claim 1, where the material arrangement vector generated for each frame represents the spatial arrangement of material content in the frame.

10. The method of claim 1, where the one or more scene classes are selected from the group comprising:

coast/beach, desert, forest, grassland, highway, indoor, lake/river, mountainous, open water, outdoor, sky, snow, and urban.

11. The method of claim 1, where the representative scene classification score for each of the one or more scene classes comprises a statistical property of the generated scene classification scores.

12. The method of claim 1, where the statistical property is selected from the group comprising:

average, median, maximum, and minimum.

13. The method of claim 1, where the video file comprises a shot of video.

14. The method of claim 1, further comprising the step of generating a report based on the labeled video file.

15. The method of claim 1, where the labeled one or more scene classes are used for indexing and retrieval of the video file.