Systems and methods for semantically classifying shots in video

• US 8,311,344 B2
• Filed: 02/17/2009
• Issued: 11/13/2012
• Est. Priority Date: 02/15/2008
• Status: Active Grant

• Current Assignee: TiVo Solutions Inc. (Adeia Inc.)
• Original Assignee: Digitalsmiths Corporation (Adeia Inc.)
• Inventors: Dunlop, Heather, Berry, Matthew G.
• Primary Examiner(s): Bhatnagar, Anand
• Assistant Examiner(s): Park, Soo
• Application Number:
  US12/372,561
• Publication Number:
  US 20090208106A1
• Time in Patent Office: 1,365 Days
• Field of Search: 382224-228
• US Class Current: 382/224
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44

Abstract

The present disclosure relates to systems and methods for classifying videos based on video content. For a given video file including a plurality of frames, a subset of frames is extracted for processing. Frames that are too dark, blurry, or otherwise poor classification candidates are discarded from the subset. Generally, material classification scores that describe type of material content likely included in each frame are calculated for the remaining frames in the subset. The material classification scores are used to generate material arrangement vectors that represent the spatial arrangement of material content in each frame. The material arrangement vectors are subsequently classified to generate a scene classification score vector for each frame. The scene classification results are averaged (or otherwise processed) across all frames in the subset to associate the video file with one or more predefined scene categories related to overall types of scene content of the video file.

Claims

1. A system for classifying videos based on video content, comprising:

a processor;

one or more software modules including one or more classifiers;

the one or more software modules operable on the processor to perform the steps of;

receiving a video file, the video file including a plurality of frames;

extracting a subset of frames from the video file;

if one or more frames in the extracted subset of frames comprises a dark frame, discarding the one or more dark frames from the subset;

determining whether each frame in the extracted subset includes content associated with a general content category;

for each frame in the extracted subset that includes content associated with the general content category;

dividing the frame into one or more segments based on image content in each segment;

generating a material classification score vector for each segment, each material classification score vector including one or more material values associated with one or more predefined material content types, wherein each material value represents a probability that the respective segment includes that type of material content;

assigning the material classification score vector for each segment to each respective pixel in the segment;

generating a material arrangement vector for the frame based on the material classification score vectors assigned to each pixel;

generating a scene classification score vector for the frame via one or more scene classifiers, the scene classification score vector including one or more scene classification scores associated with one or more predefined scene categories within the general content category;

determining a representative scene classification score vector for the video file based on the generated scene classification score vectors for each extracted frame in the subset that includes content associated with the general content category; and

associating the video file with the one or more predefined scene categories based on the representative scene classification score vector.

2. The system of claim 1, wherein the one or more software modules are selected from the group comprising:

an intensity classification module, an indoor/outdoor classification module, an outdoor classification module, a segmentation module, a material arrangement module, and a video file classification module.

7. The system of claim 1, wherein a dark frame comprises a frame shot in low or no light.

8. The system of claim 1, further comprising the step of determining whether any of the frames in the extracted subset of frames comprises a dark frame.

9. The system of claim 8, wherein the step of determining whether any of the frames in the extracted subset of frames comprises a dark frame further comprises the steps of:

for each frame, dividing the frame into a plurality of grid cells;

calculating an intensity value for each pixel in each cell;

calculating an average intensity value across all pixels in each cell;

concatenating the average intensity values for each cell in the frame to form an intensity feature vector for the frame; and

classifying the intensity feature vector via an intensity classifier to determine if the frame comprises a dark frame.

10. The system of claim 1, wherein the step of determining whether each frame in the extracted subset includes content associated with the general content category further comprises the steps of:

extracting a plurality of features from each frame;

generating a feature vector for each frame, wherein each feature vector includes the extracted features for the respective frame; and

classifying the feature vector for each frame via a general category classifier to determine whether each frame includes content associated with the general content category.

11. The system of claim 1, wherein the general content category comprises outdoor content.

12. The system of claim 1, wherein the one or more predefined scene categories are selected from the group comprising:

coast/beach, desert, forest, grassland, highway, indoor, lake/river, mountainous, open water, outdoor, sky, snow, and urban.

13. The system of claim 1, wherein each of the one or more scene classification scores represents the probability that a frame includes content associated with each of the respective predefined scene categories.

14. The system of claim 1, wherein each of the one or more scene classification scores comprises a value greater than or equal to zero and less than or equal to one.

15. The system of claim 1, wherein the representative scene classification score vector comprises a statistical property of the generated scene classification score vectors.

16. The system of claim 15, wherein the statistical property is selected from the group comprising:

average, median, maximum, and minimum.

18. The system of claim 1, wherein the video file comprises a shot of video.

19. The system of claim 1, further comprising the step of generating a report including the one or more predefined scene categories associated with the video file.

20. The system of claim 1, wherein the one or more scene categories associated with the video file are used for indexing and retrieval of the video file.

3. A system for classifying videos based on video content, comprising:

a processor;

one or more software modules including one or more classifiers;

a computer program product that includes a computer-readable medium that is usable by the processor, the medium having stored thereon a sequence of instructions associated with the one or more software modules that when executed by the processor causes the execution of the steps of;

receiving a video file, the video file including a plurality of frames;

extracting a subset of frames from the video file;

if one or more frames in the extracted subset of frames comprises a dark frame, discarding the one or more dark frames from the subset;

determining whether each frame in the extracted subset includes content associated with a general content category;

for each frame in the extracted subset that includes content associated with the general content category,

dividing the frame into one or more segments based on image content in each segment;

generating a material classification score vector for each segment, each material classification score vector including one or more material values associated with one or more predefined material content types, wherein each material value represents a probability that the respective segment includes that type of material content;

assigning the material classification score vector for each segment to each respective pixel in the segment;

generating a material arrangement vector for the frame based on the material classification score vectors assigned to each pixel;

classifying the material arrangement vector via the one or more scene classifiers to generate the scene classification score vector for the frame;

determining a representative scene classification score vector for the video file based on the generated scene classification score vectors for each extracted frame in the subset that includes content associated with the general content category; and

associating the video file with the one or more predefined scene categories based on the representative scene classification score vector.

4. The system of claim 3, further comprising the step of combining adjacent segments based on similar content properties of the segments.

5. The system of claim 3, wherein the one or more predefined material content types are selected from the group comprising:

building, grass, person, road/sidewalk, rock, sand/gravel/soil, sky/clouds, snow/ice, trees/plants, vehicle, water, and miscellaneous.

6. The system of claim 3, wherein the material arrangement vector represents the spatial arrangement of material content in the frame.

17. A system for classifying videos based on video content, comprising:

a processor;

one or more software modules including one or more classifiers;

the one or more software modules operable on the processor to perform the steps of;

receiving a video file, the video file including a plurality of frames;

extracting a subset of frames from the video file;

if one or more frames in the extracted subset of frames comprises a dark frame, discarding the one or more dark frames from the subset;

determining whether each frame in the extracted subset includes content associated with a general content category;

for each frame in the extracted subset that includes content associated with the general content category;

generating a scene classification score vector for the frame via one or more scene classifiers, the scene classification score vector including one or more scene classification scores associated with one or more predefined scene categories within the general content category;

determining a representative scene classification score vector for the video file based on the generated scene classification score vectors for each extracted frame in the subset that includes content associated with the general content category;

associating the video file with the one or more predefined scene categories based on the representative scene classification score vector;

identifying representative scene classification scores in the representative scene classification score vector that exceed a predetermined threshold value; and

for each representative scene classification score that exceeds the threshold value, associating the video file with the one or more predefined scene categories associated with the classification scores that exceeded the threshold value.

21. A method for classifying videos based on video content, comprising the steps of:

receiving a video file, the video file including a plurality of frames;

extracting a subset of frames from the video file;

if one or more frames in the extracted subset of frames comprises a dark frame, discarding the one or more dark frames from the subset;

determining whether each frame in the extracted subset includes content associated with a general content category;

for each frame in the extracted subset that includes content associated with the general content category;

dividing the frame into one or more segments based on image content in each segment;

generating a material classification score vector for each segment, each material classification score vector including one or more material values associated with one or more predefined material content types, wherein each material value represents a probability that the respective segment includes that type of material content;

assigning the material classification score vector for each segment to each respective pixel in the segment;

generating a material arrangement vector for the frame based on the material classification score vectors assigned to each pixel;

classifying the material arrangement vector via the one or more scene classifiers to generate the scene classification score vector for the frame;

generating a scene classification score vector for the frame via one or more scene classifiers, the scene classification score vector including one or more scene classification scores associated with one or more predefined scene categories within the general content category;

determining a representative scene classification score vector for the video file based on the generated scene classification score vectors for each extracted frame in the subset that includes content associated with the general content category; and

labeling the video file according to the one or more predefined scene categories based on the representative scene classification score vector.

22. The method of claim 21, further comprising the step of combining adjacent segments based on similar content properties of the segments.

23. The method of claim 21, wherein the one or more predefined material content types are selected from the group comprising:

building, grass, person, road/sidewalk, rock, sand/gravel/soil, sky/clouds, snow/ice, trees/plants, vehicle, water, and miscellaneous.

24. The method of claim 21, wherein the material arrangement vector represents the spatial arrangement of material content in the frame.

25. The method of claim 21, wherein a dark frame comprises a frame shot in low or no light.

26. The method of claim 21, further comprising the step of determining whether any of the frames in the extracted subset of frames comprises a dark frame.

27. The method of claim 26, wherein the step of determining whether any of the frames in the extracted subset of frames comprises a dark frame further comprises the steps of:

for each frame, dividing the frame into a plurality of grid cells;

calculating an intensity value for each pixel in each cell;

calculating an average intensity value across all pixels in each cell;

concatenating the average intensity values for each cell in the frame to form an intensity feature vector for the frame; and

classifying the intensity feature vector via an intensity classifier to determine if the frame comprises a dark frame.

28. The method of claim 21, wherein the step of determining whether each frame in the extracted subset includes content associated with the general content category further comprises the steps of:

extracting a plurality of features from each frame;

generating a feature vector for each frame, wherein each feature vector includes the extracted features for the respective frame; and

classifying the feature vector for each frame via a general category classifier to determine whether each frame includes content associated with the general content category.

29. The method of claim 21, wherein the general content category comprises outdoor content.

30. The method of claim 21, wherein the one or more predefined scene categories are selected from the group comprising:

coast/beach, desert, forest, grassland, highway, indoor, lake/river, mountainous, open water, outdoor, sky, snow, and urban.

31. The method of claim 21, wherein each of the one or more scene classification scores represents the probability that a frame includes content associated with each of the respective predefined scene categories.

32. The method of claim 21, wherein each of the one or more scene classification scores comprises a value greater than or equal to zero and less than or equal to one.

33. The method of claim 21, wherein the representative scene classification score vector comprises a statistical property of the generated scene classification score vectors.

34. The method of claim 33, wherein the statistical property is selected from the group comprising:

average, median, maximum, and minimum.

35. A method for classifying videos based on video content, comprising:

receiving a video file, the video file including a plurality of frames;

extracting a subset of frames from the video file;

if one or more frames in the extracted subset of frames comprises a dark frame, discarding the one or more dark frames from the subset;

determining whether each frame in the extracted subset includes content associated with a general content category;

for each frame in the extracted subset that includes content associated with the general content category;

generating a scene classification score vector for the frame via one or more scene classifiers, the scene classification score vector including one or more scene classification scores associated with one or more predefined scene categories within the general content category;

determining a representative scene classification score vector for the video file based on the generated scene classification score vectors for each extracted frame in the subset that includes content associated with the general content category;

labeling the video file according to the one or more predefined scene categories based on the representative scene classification score vector;

identifying representative scene classification scores in the representative scene classification score vector that exceed a predetermined threshold value; and

for each representative scene classification score that exceeds the threshold value, associating the video file with the one or more predefined scene categories associated with the classification scores that exceeded the threshold value.

36. The method of claim 35, wherein the video file comprises a shot of video.

37. The method of claim 35, further comprising the step of generating a report based on the labeled video file.

38. The method of claim 35, wherein the labeled one or more scene categories are used for indexing and retrieval of the video file.

39. A method for labeling videos based on video content, comprising the steps of:

receiving a video file, wherein the video file includes a plurality of frames;

extracting a set of frames from the plurality of frames in the video file;

for each frame in the extracted set of frames, calculating a probability that the frame includes content associated with a predefined scene category;

dividing the frame into one or more segments based on image content in each segment;

generating a material classification score vector for each segment, each material classification score vector including one or more material values associated with one or more predefined material content types, wherein each material value represents a probability that the respective segment includes that type of material content;

assigning the material classification score vector for each segment to each respective pixel in the segment;

generating a material arrangement vector for the frame based on the material classification score vectors assigned to each pixel;

determining a representative probability for the set of frames based on the calculated probabilities for each frame; and

if the representative probability exceeds a predetermined threshold, associating the scene category with the video file.

40. The method of claim 39, wherein the representative probability comprises a statistical property of the calculated probabilities for each frame.

41. The method of claim 40, wherein the statistical property is selected from the group comprising:

average, median, maximum, and minimum.

42. The method of claim 39, further comprising the step of indexing the video file according to the associated scene category for search and retrieval purposes.

43. The method of claim 39, wherein the scene category is selected from the list comprising:

coast/beach, desert, forest, grassland, highway, indoor, lake/river, mountainous, open water, outdoor, sky, snow, and urban.

44. The method of claim 39, wherein the calculated probability for each frame is calculated via a classifier.