SYSTEMS AND METHODS FOR SEMANTICALLY CLASSIFYING AND NORMALIZING SHOTS IN VIDEO

• US 20150356354A1
• Filed: 08/18/2015
• Published: 12/10/2015
• Est. Priority Date: 02/15/2008
• Status: Active Grant

• Current Assignee: TiVo Solutions Inc. (Adeia Inc.)
• Original Assignee: TiVo Inc. (Adeia Inc.)
• Inventors: Dunlop, Heather, Berry, Matthew
• Granted Patent: US 9,405,976 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content

Abstract

The present disclosure relates to systems and methods for classifying videos based on video content. For a given video file including a plurality of frames, a subset of frames is extracted for processing. Frames that are too dark, blurry, or otherwise poor classification candidates are discarded from the subset. Generally, material classification scores that describe type of material content likely included in each frame are calculated for the remaining frames in the subset. The material classification scores are used to generate material arrangement vectors that represent the spatial arrangement of material content in each frame. The material arrangement vectors are subsequently classified to generate a scene classification score vector for each frame. The scene classification results are averaged (or otherwise processed) across all frames in the subset to associate the video file with one or more predefined scene categories related to overall types of scene content of the video file.

Claims

1. A method, comprising:

receiving a video file comprising a plurality of scenes, each scene of the plurality of scenes comprising a plurality of frames;

for each scene of the plurality of scenes;

determining, based on one or more classifiers, that one or more frames from the scene includes content associated with one or more predefined scene categories;

assigning, based on the one or more predefined scene categories associated with the one or more frames, one or more scene categories to the scene;

storing, in one or more indexes, data indicating (a) the one or more scene categories assigned to each scene of the plurality of scenes, and (b) a start time and end time for each scene of the plurality of scenes.

2. The method of claim 1, further comprising:

receiving a search query specifying one or more particular scene categories;

responding to the search query with one or more scene results, each scene result of the one or more scene results associated with at least one of the one or more particular scene categories.

3. The method of claim 1, further comprising:

extracting a set of frames from the scene;

in response to determining that one or more particular frames from the set of frames comprises a poor classification frame, removing the one or more particular frames from the set of frames.

4. The method of claim 1, further comprising:

extracting a set of frames from the scene;

in response to determining that one or more particular frames from the set of frames comprises a poor classification frame, removing the one or more particular frames from the set of frames;

wherein one or more particular frames are determined to comprise a poor classification frame based on one or more classifiers.

5. The method of claim 1, further comprising:

wherein determining that one or more frames from the scene includes content associated with one or more predefined scene categories includes generating one or more scene classification scores for each frame of the one or more frames; and

wherein the one or more classifiers comprise a spatial pyramid of classifiers, and wherein varying weights are associated with one or more scene classification scores.

6. The method of claim 1, further comprising:

dividing each frame of the one or more frames into one or more segments, each segment including relatively uniform image content;

extracting image features from each segment to form a feature vector associated with each segment;

generating a material classification score vector for each segment via one or more material classifiers based on the feature vector associated with each segment, wherein each material classification score vector includes one or more material classification scores associated with one or more predefined material content categories.

7. The method of claim 1, further comprising:

dividing each frame of the one or more frames into one or more segments, each segment including relatively uniform image content;

extracting image features from each segment to form a feature vector associated with each segment;

generating a material classification score vector for each segment via one or more material classifiers based on the feature vector associated with each segment, wherein each material classification score vector includes one or more material classification scores associated with one or more predefined material content categories;

wherein the one or more predefined material content categories include one or more of;

building, grass, person, road/sidewalk, rock, sand/gravel/soil, sky/clouds, snow/ice, trees/plants, vehicle, water, and miscellaneous.

8. The method of claim 1, further comprising:

dividing each frame of the one or more frames into one or more segments, each segment including relatively uniform image content;

extracting image features from each segment to form a feature vector associated with each segment;

generating a material classification score vector for each segment via one or more material classifiers based on the feature vector associated with each segment, wherein each material classification score vector includes one or more material classification scores associated with one or more predefined material content categories;

wherein each of the one or more material classification scores represents a probability that a frame includes content associated with each of the respective predefined material content categories.

9. The method of claim 1, wherein the one or more predefined scene categories includes one or more of:

coast/beach, desert, forest, grassland, highway, indoor, lake/river, mountainous, open water, outdoor, sky, snow, and urban.

10. The method of claim 1, wherein the data further indicates one or more scene boundaries.

11. One or more non-transitory storage media storing instructions which, when executed by one or more computing devices, cause:

receiving a video file comprising a plurality of scenes, each scene of the plurality of scenes comprising a plurality of frames;

for each scene of the plurality of scenes;

determining, based on one or more classifiers, that one or more frames from the scene includes content associated with one or more predefined scene categories;

assigning, based on the one or more predefined scene categories associated with the one or more frames, one or more scene categories to the scene;

storing, in one or more indexes, data indicating (a) the one or more scene categories assigned to each scene of the plurality of scenes, and (b) a start time and end time for each scene of the plurality of scenes.

12. The one or more non-transitory storage media of claim 11, wherein the instructions, when executed by the one or more computing devices, further cause:

receiving a search query specifying one or more particular scene categories;

responding to the search query with one or more scene results, each scene result of the one or more scene results associated with at least one of the one or more particular scene categories

13. The one or more non-transitory storage media of claim 11, wherein the instructions, when executed by the one or more computing devices, further cause:

extracting a set of frames from the scene;

in response to determining that one or more particular frames from the set of frames comprises a poor classification frame, removing the one or more particular frames from the set of frames.

14. The one or more non-transitory storage media of claim 11, wherein the instructions, when executed by the one or more computing devices, further cause:

extracting a set of frames from the scene;

in response to determining that one or more particular frames from the set of frames comprises a poor classification frame, removing the one or more particular frames from the set of frames;

wherein one or more particular frames are determined to comprise a poor classification frame based on one or more classifiers.

15. The one or more non-transitory storage media of claim 11, wherein the instructions, when executed by the one or more computing devices, further cause:

wherein determining that one or more frames from the scene includes content associated with one or more predefined scene categories includes generating one or more scene classification scores for each frame of the one or more frames; and

wherein the one or more classifiers comprise a spatial pyramid of classifiers, and wherein varying weights are associated with one or more scene classification scores.

16. The one or more non-transitory storage media of claim 11, wherein the instructions, when executed by the one or more computing devices, further cause:

dividing each frame of the one or more frames into one or more segments, each segment including relatively uniform image content;

extracting image features from each segment to form a feature vector associated with each segment;

generating a material classification score vector for each segment via one or more material classifiers based on the feature vector associated with each segment, wherein each material classification score vector includes one or more material classification scores associated with one or more predefined material content categories.

17. The one or more non-transitory storage media of claim 11, wherein the instructions, when executed by the one or more computing devices, further cause:

dividing each frame of the one or more frames into one or more segments, each segment including relatively uniform image content;

extracting image features from each segment to form a feature vector associated with each segment;

generating a material classification score vector for each segment via one or more material classifiers based on the feature vector associated with each segment, wherein each material classification score vector includes one or more material classification scores associated with one or more predefined material content categories;

wherein the one or more predefined material content categories include one or more of;

building, grass, person, road/sidewalk, rock, sand/gravel/soil, sky/clouds, snow/ice, trees/plants, vehicle, water, and miscellaneous.

18. The one or more non-transitory storage media of claim 11, wherein the instructions, when executed by the one or more computing devices, further cause:

dividing each frame of the one or more frames into one or more segments, each segment including relatively uniform image content;

extracting image features from each segment to form a feature vector associated with each segment;

generating a material classification score vector for each segment via one or more material classifiers based on the feature vector associated with each segment, wherein each material classification score vector includes one or more material classification scores associated with one or more predefined material content categories;

wherein each of the one or more material classification scores represents a probability that a frame includes content associated with each of the respective predefined material content categories.

19. The one or more non-transitory storage media of claim 11, wherein the one or more predefined scene categories includes one or more of:

coast/beach, desert, forest, grassland, highway, indoor, lake/river, mountainous, open water, outdoor, sky, snow, and urban.

20. The one or more non-transitory storage media of claim 11, wherein the data further indicates one or more scene boundaries.

21. An apparatus comprising:

a subsystem, implemented at least partially in hardware, that receives a video file comprising a plurality of scenes, each scene of the plurality of scenes comprising a plurality of frames;

a subsystem, implemented at least partially in hardware, that for each scene of the plurality of scenes;

determines, based on one or more classifiers, that one or more frames from the scene includes content associated with one or more predefined scene categories;

assigns, based on the one or more predefined scene categories associated with the one or more frames, one or more scene categories to the scene;

a subsystem, implemented at least partially in hardware, that stores, in one or more indexes, data indicating (a) the one or more scene categories assigned to each scene of the plurality of scenes, and (b) a start time and end time for each scene of the plurality of scenes

22. The apparatus of claim 21, further comprising:

a subsystem, implemented at least partially in hardware, that receives a search query specifying one or more particular scene categories;

a subsystem, implemented at least partially in hardware, that responds to the search query with one or more scene results, each scene result of the one or more scene results associated with at least one of the one or more particular scene categories

23. The apparatus of claim 21, further comprising:

a subsystem, implemented at least partially in hardware, that extracts a set of frames from the scene;

a subsystem, implemented at least partially in hardware, that in response to determining that one or more particular frames from the set of frames comprises a poor classification frame, removes the one or more particular frames from the set of frames.

24. The apparatus of claim 21, further comprising:

a subsystem, implemented at least partially in hardware, that extracts a set of frames from the scene;

a subsystem, implemented at least partially in hardware, that in response to determining that one or more particular frames from the set of frames comprises a poor classification frame, removes the one or more particular frames from the set of frames;

wherein one or more particular frames are determined to comprise a poor classification frame based on one or more classifiers.

25. The apparatus of claim 21, further comprising:

wherein determining that one or more frames from the scene includes content associated with one or more predefined scene categories includes generating one or more scene classification scores for each frame of the one or more frames; and

wherein the one or more classifiers comprise a spatial pyramid of classifiers, and wherein varying weights are associated with one or more scene classification scores.

26. The apparatus of claim 21, further comprising:

a subsystem, implemented at least partially in hardware, that divides each frame of the one or more frames into one or more segments, each segment including relatively uniform image content;

a subsystem, implemented at least partially in hardware, that extracts image features from each segment to form a feature vector associated with each segment;

a subsystem, implemented at least partially in hardware, that generates a material classification score vector for each segment via one or more material classifiers based on the feature vector associated with each segment, wherein each material classification score vector includes one or more material classification scores associated with one or more predefined material content categories.

27. The apparatus of claim 21, further comprising:

a subsystem, implemented at least partially in hardware, that divides each frame of the one or more frames into one or more segments, each segment including relatively uniform image content;

a subsystem, implemented at least partially in hardware, that extracts image features from each segment to form a feature vector associated with each segment;

a subsystem, implemented at least partially in hardware, that generates a material classification score vector for each segment via one or more material classifiers based on the feature vector associated with each segment, wherein each material classification score vector includes one or more material classification scores associated with one or more predefined material content categories;

wherein the one or more predefined material content categories include one or more of;

building, grass, person, road/sidewalk, rock, sand/gravel/soil, sky/clouds, snow/ice, trees/plants, vehicle, water, and miscellaneous.

28. The apparatus of claim 21, further comprising:

a subsystem, implemented at least partially in hardware, that divides each frame of the one or more frames into one or more segments, each segment including relatively uniform image content;

a subsystem, implemented at least partially in hardware, that extracts image features from each segment to form a feature vector associated with each segment;

a subsystem, implemented at least partially in hardware, that generates a material classification score vector for each segment via one or more material classifiers based on the feature vector associated with each segment, wherein each material classification score vector includes one or more material classification scores associated with one or more predefined material content categories;

wherein each of the one or more material classification scores represents a probability that a frame includes content associated with each of the respective predefined material content categories.

29. The apparatus of claim 21, wherein the one or more predefined scene categories includes one or more of:

coast/beach, desert, forest, grassland, highway, indoor, lake/river, mountainous, open water, outdoor, sky, snow, and urban.

30. The apparatus of claim 21, wherein the data further indicates one or more scene boundaries.