Systems and methods for semantically classifying and normalizing shots in video

• US 9,405,976 B2
• Filed: 08/18/2015
• Issued: 08/02/2016
• Est. Priority Date: 02/15/2008
• Status: Active Grant

• Current Assignee: TiVo Solutions Inc. (Adeia Inc.)
• Original Assignee: TiVo Inc. (Adeia Inc.)
• Inventors: Dunlop, Heather, Berry, Matthew
• Primary Examiner(s): Park, Soo Jin
• Application Number:
  US14/829,593
• Publication Number:
  US 20150356354A1
• Time in Patent Office: 350 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content

Abstract

The present disclosure relates to systems and methods for classifying videos based on video content. For a given video file including a plurality of frames, a subset of frames is extracted for processing. Frames that are too dark, blurry, or otherwise poor classification candidates are discarded from the subset. Generally, material classification scores that describe type of material content likely included in each frame are calculated for the remaining frames in the subset. The material classification scores are used to generate material arrangement vectors that represent the spatial arrangement of material content in each frame. The material arrangement vectors are subsequently classified to generate a scene classification score vector for each frame. The scene classification results are averaged (or otherwise processed) across all frames in the subset to associate the video file with one or more predefined scene categories related to overall types of scene content of the video file.

Claims

1. A method, comprising:

receiving a video file comprising a plurality of frames, each frame of the plurality of frames associated with a plurality of scene classification scores, each scene classification score of the plurality of scene classification scores representing a probability that a respective frame includes content associated with a scene category of a plurality of scene categories;

determining, based on analyzing the plurality of scene classification scores associated with one or more frames of the plurality of frames, a start time and an end time of each scene of a plurality of scenes;

for at least one scene of the plurality of scenes;

determining that one or more frames from the at least one scene includes content associated with one or more scene categories;

assigning, based on the one or more scene categories associated with the one or more frames, one or more scene categories from the plurality of scene categories to the at least one scene;

storing, in one or more indexes, data indicating (a) the one or more scene categories assigned to the at least one scene of the plurality of scenes, and (b) a start time and an end time for the at least one scene of the plurality of scenes.

2. The method of claim 1, further comprising:

receiving a search query specifying one or more particular scene categories;

responding to the search query with one or more scene results, each scene result of the one or more scene results associated with at least one of the one or more particular scene categories.

3. The method of claim 1, further comprising:

extracting a set of frames from the at least one scene;

in response to determining that one or more particular frames from the set of frames comprises a poor classification frame, removing the one or more particular frames from the set of frames.

4. The method of claim 1, further comprising:

extracting a set of frames from the at least one scene;

in response to determining that one or more particular frames from the set of frames comprises a poor classification frame, removing the one or more particular frames from the set of frames;

wherein one or more particular frames are determined to comprise a poor classification frame based on one or more classifiers.

5. The method of claim 1, further comprising:

generating the plurality of scene classification scores for each frame of the plurality of frames;

wherein determining that one or more frames from the at least one scene includes content associated with one or more scene categories is based on one or more classifiers;

wherein the one or more classifiers comprise a spatial pyramid of classifiers, and wherein varying weights are associated with the plurality of scene classification scores.

6. The method of claim 1, further comprising:

dividing each frame of the plurality of frames into one or more segments, each segment including relatively uniform image content;

extracting image features from each segment to form a feature vector associated with each segment;

generating a material classification score vector for each segment via one or more material classifiers based on the feature vector associated with each segment, wherein each material classification score vector includes one or more material classification scores associated with one or more material content categories.

7. The method of claim 1, further comprising:

dividing each frame of the plurality of frames into one or more segments, each segment including relatively uniform image content;

extracting image features from each segment to form a feature vector associated with each segment;

generating a material classification score vector for each segment via one or more material classifiers based on the feature vector associated with each segment, wherein each material classification score vector includes one or more material classification scores associated with one or more material content categories;

wherein the one or more material content categories include one or more of;

building, grass, person, road/sidewalk, rock, sand/gravel/soil, sky/clouds, snow/ice, trees/plants, vehicle, water, and miscellaneous.

8. The method of claim 1, further comprising:

dividing each frame of the plurality of frames into one or more segments, each segment including relatively uniform image content;

extracting image features from each segment to form a feature vector associated with each segment;

generating a material classification score vector for each segment via one or more material classifiers based on the feature vector associated with each segment, wherein each material classification score vector includes one or more material classification scores associated with one or more material content categories;

wherein each of the one or more material classification scores represents a probability that a frame includes content associated with each of the respective material content categories.

9. The method of claim 1, wherein the one or more scene categories includes one or more of:

coast/beach, desert, forest, grassland, highway, indoor, lake/river, mountainous, open water, outdoor, sky, snow, and urban.

10. One or more non-transitory storage media storing instructions which, when executed by one or more computing devices, cause:

receiving a video file comprising a plurality of frames, each frame of the plurality of frames associated with a plurality of scene classification scores, each scene classification score of the plurality of scene classification scores representing a probability that a respective frame includes content associated with a scene category of a plurality of scene categories;

determining, based on analyzing the plurality of scene classification scores associated with one or more frames of the plurality of frames, a start time and an end time of each scene of a plurality of scenes;

for at least one scene of the plurality of scenes;

determining that one or more frames from the at least one scene includes content associated with one or more scene categories;

assigning, based on the one or more scene categories associated with the one or more frames, one or more scene categories from the plurality of scene categories to the at least one scene;

storing, in one or more indexes, data indicating (a) the one or more scene categories assigned to the at least one scene of the plurality of scenes, and (b) a start time and an end time for the at least one scene of the plurality of scenes.

11. The one or more non-transitory storage media of claim 10, wherein the instructions, when executed by the one or more computing devices, further cause:

receiving a search query specifying one or more particular scene categories;

responding to the search query with one or more scene results, each scene result of the one or more scene results associated with at least one of the one or more particular scene categories.

12. The one or more non-transitory storage media of claim 10, wherein the instructions, when executed by the one or more computing devices, further cause:

extracting a set of frames from the at least one scene;

in response to determining that one or more particular frames from the set of frames comprises a poor classification frame, removing the one or more particular frames from the set of frames.

13. The one or more non-transitory storage media of claim 10, wherein the instructions, when executed by the one or more computing devices, further cause:

extracting a set of frames from the at least one scene;

in response to determining that one or more particular frames from the set of frames comprises a poor classification frame, removing the one or more particular frames from the set of frames;

wherein one or more particular frames are determined to comprise a poor classification frame based on one or more classifiers.

14. The one or more non-transitory storage media of claim 10, wherein the instructions, when executed by the one or more computing devices, further cause:

generating the plurality of scene classification scores for each frame of the plurality of frames;

wherein determining that one or more frames from the at least one scene includes content associated with one or more scene categories is based on one or more classifiers;

wherein the one or more classifiers comprise a spatial pyramid of classifiers, and wherein varying weights are associated with the plurality of scene classification scores.

15. The one or more non-transitory storage media of claim 10, wherein the instructions, when executed by the one or more computing devices, further cause:

dividing each frame of the plurality of frames into one or more segments, each segment including relatively uniform image content;

extracting image features from each segment to form a feature vector associated with each segment;

generating a material classification score vector for each segment via one or more material classifiers based on the feature vector associated with each segment, wherein each material classification score vector includes one or more material classification scores associated with one or more predefined material content categories.

16. The one or more non-transitory storage media of claim 10, wherein the instructions, when executed by the one or more computing devices, further cause:

dividing each frame of the plurality of frames into one or more segments, each segment including relatively uniform image content;

extracting image features from each segment to form a feature vector associated with each segment;

generating a material classification score vector for each segment via one or more material classifiers based on the feature vector associated with each segment, wherein each material classification score vector includes one or more material classification scores associated with one or more material content categories;

wherein the one or more material content categories include one or more of;

building, grass, person, road/sidewalk, rock, sand/gravel/soil, sky/clouds, snow/ice, trees/plants, vehicle, water, and miscellaneous.

17. The one or more non-transitory storage media of claim 10, wherein the instructions, when executed by the one or more computing devices, further cause:

dividing each frame of the plurality of frames into one or more segments, each segment including relatively uniform image content;

extracting image features from each segment to form a feature vector associated with each segment;

generating a material classification score vector for each segment via one or more material classifiers based on the feature vector associated with each segment, wherein each material classification score vector includes one or more material classification scores associated with one or more material content categories;

wherein each of the one or more material classification scores represents a probability that a frame includes content associated with each of the respective material content categories.

18. The one or more non-transitory storage media of claim 10, wherein the one or more scene categories includes one or more of:

coast/beach, desert, forest, grassland, highway, indoor, lake/river, mountainous, open water, outdoor, sky, snow, and urban.

19. An apparatus comprising:

a subsystem, implemented at least partially in hardware, that receives a video file comprising a plurality of frames, each frame of the plurality of frames associated with a plurality of scene classification scores, each scene classification score of the plurality of scene classification scores representing a probability that a respective frame includes content associated with a scene category of a plurality of scene categories;

a subsystem, implemented at least partially in hardware, that determines, based on analyzing the plurality of scene classification scores associated with one or more frames of the plurality of frames, a start time and an end time of each scene of a plurality of scenes;

a subsystem, implemented at least partially in hardware, that for at least one scene of the plurality of scenes;

determines, that one or more frames from the at least one scene includes content associated with one or more

assigns, based on the one or more scene categories associated with the one or more frames, one or more scene categories from the plurality of scene categories to the at least one scene;

a subsystem, implemented at least partially in hardware, that stores, in one or more indexes, data indicating (a) the one or more scene categories assigned to the at least one scene of the plurality of scenes, and (b) a start time and an end time for the at least one scene of the plurality of scenes.

20. The apparatus of claim 19, further comprising:

a subsystem, implemented at least partially in hardware, that receives a search query specifying one or more particular scene categories;

a subsystem, implemented at least partially in hardware, that responds to the search query with one or more scene results, each scene result of the one or more scene results associated with at least one of the one or more particular scene categories.

21. The apparatus of claim 19, further comprising:

a subsystem, implemented at least partially in hardware, that extracts a set of frames from the at least one scene;

a subsystem, implemented at least partially in hardware, that in response to determining that one or more particular frames from the set of frames comprises a poor classification frame, removes the one or more particular frames from the set of frames.

22. The apparatus of claim 19, further comprising:

a subsystem, implemented at least partially in hardware, that extracts a set of frames from the at least one scene;

a subsystem, implemented at least partially in hardware, that in response to determining that one or more particular frames from the set of frames comprises a poor classification frame, removes the one or more particular frames from the set of frames;

wherein one or more particular frames are determined to comprise a poor classification frame based on one or more classifiers.

23. The apparatus of claim 19, further comprising:

a subsystem, implemented at least partially in hardware, that generates the plurality of scene classification scores for each frame of the plurality of frames;

wherein determining that one or more frames from the at least one scene includes content associated with one or more scene categories is based on one or more classifiers;

wherein the one or more classifiers comprise a spatial pyramid of classifiers, and wherein varying weights are associated with the plurality of scene classification scores.

24. The apparatus of claim 19, further comprising:

a subsystem, implemented at least partially in hardware, that divides each frame of the plurality of frames into one or more segments, each segment including relatively uniform image content;

a subsystem, implemented at least partially in hardware, that extracts image features from each segment to form a feature vector associated with each segment;

a subsystem, implemented at least partially in hardware, that generates a material classification score vector for each segment via one or more material classifiers based on the feature vector associated with each segment, wherein each material classification score vector includes one or more material classification scores associated with one or more material content categories.

25. The apparatus of claim 19, further comprising:

a subsystem, implemented at least partially in hardware, that divides each frame of the plurality of frames into one or more segments, each segment including relatively uniform image content;

a subsystem, implemented at least partially in hardware, that extracts image features from each segment to form a feature vector associated with each segment;

a subsystem, implemented at least partially in hardware, that generates a material classification score vector for each segment via one or more material classifiers based on the feature vector associated with each segment, wherein each material classification score vector includes one or more material classification scores associated with one or more material content categories;

wherein the one or more material content categories include one or more of;

building, grass, person, road/sidewalk, rock, sand/gravel/soil, sky/clouds, snow/ice, trees/plants, vehicle, water, and miscellaneous.

26. The apparatus of claim 19, further comprising:

a subsystem, implemented at least partially in hardware, that divides each frame of the plurality of frames into one or more segments, each segment including relatively uniform image content;

a subsystem, implemented at least partially in hardware, that extracts image features from each segment to form a feature vector associated with each segment;

a subsystem, implemented at least partially in hardware, that generates a material classification score vector for each segment via one or more material classifiers based on the feature vector associated with each segment, wherein each material classification score vector includes one or more material classification scores associated with one or more predefined material content categories;

wherein each of the one or more material classification scores represents a probability that a frame includes content associated with each of the respective material content categories.

27. The apparatus of claim 19, wherein the one or more scene categories includes one or more of:

coast/beach, desert, forest, grassland, highway, indoor, lake/river, mountainous, open water, outdoor, sky, snow, and urban.