SYSTEMS AND METHODS FOR SEMANTICALLY CLASSIFYING SHOTS IN VIDEO

• US 20090208106A1
• Filed: 02/17/2009
• Published: 08/20/2009
• Est. Priority Date: 02/15/2008
• Status: Active Grant

• Current Assignee: TiVo Solutions Inc. (Adeia Inc.)
• Original Assignee: Digitalsmiths Corporation (Adeia Inc.)
• Inventors: Dunlop, Heather, Berry, Matthew G.
• Granted Patent: US 8,311,344 B2
• Time in Patent Office: Days
• US Class Current: 382/173
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44

Abstract

The present disclosure relates to systems and methods for classifying videos based on video content. For a given video file including a plurality of frames, a subset of frames is extracted for processing. Frames that are too dark, blurry, or otherwise poor classification candidates are discarded from the subset. Generally, material classification scores that describe type of material content likely included in each frame are calculated for the remaining frames in the subset. The material classification scores are used to generate material arrangement vectors that represent the spatial arrangement of material content in each frame. The material arrangement vectors are subsequently classified to generate a scene classification score vector for each frame. The scene classification results are averaged (or otherwise processed) across all frames in the subset to associate the video file with one or more predefined scene categories related to overall types of scene content of the video file.

Claims

1. A system for classifying videos based on video content, comprising:

a processor;

one or more software modules including one or more classifiers;

a computer program product that includes a computer-readable medium that is usable by the processor, the medium having stored thereon a sequence of instructions associated with the one or more software modules that when executed by the processor causes the execution of the steps of;

receiving a video file, the video file including a plurality of frames;

extracting a subset of frames from the video file;

if one or more frames in the extracted subset of frames comprises a dark frame, discarding the one or more dark frames from the subset;

determining whether each frame in the extracted subset includes content associated with a general content category;

for each frame in the extracted subset that includes content associated with the general content category, generating a scene classification score vector for the frame via one or more scene classifiers, the scene classification score vector including one or more scene classification scores associated with one or more predefined scene categories within the general content category;

determining a representative scene classification score vector for the video file based on the generated scene classification score vectors for each extracted frame in the subset that includes content associated with the general content category; and

associating the video file with the one or more predefined scene categories based on the representative scene classification score vector.

2. The system of claim 1, wherein the one or more software modules are selected from the group comprising:

an intensity classification module, an indoor/outdoor classification module, an outdoor classification module, a segmentation module, a material arrangement module, and a video file classification module.

3. The system of claim 1, wherein the step of generating the scene classification score vector for each frame in the extracted subset of frames that includes content associated with the general content category further comprises the steps of:

dividing the frame into one or more segments based on image content in each segment;

generating a material classification score vector for each segment, each material classification score vector including one or more material values associated with one or more predefined material content types, wherein each material value represents a probability that the respective segment includes that type of material content;

assigning the material classification score vector for each segment to each respective pixel in the segment;

generating a material arrangement vector for the frame based on the material classification score vectors assigned to each pixel; and

classifying the material arrangement vector via the one or more scene classifiers to generate the scene classification score vector for the frame.

4. The system of claim 3, further comprising the step of combining adjacent segments based on similar content properties of the segments.

5. The system of claim 3, wherein the one or more predefined material content types are selected from the group comprising:

building, grass, person, road/sidewalk, rock, sand/gravel/soil, sky/clouds, snow/ice, trees/plants, vehicle, water, and miscellaneous.

6. The system of claim 3, wherein the material arrangement vector represents the spatial arrangement of material content in the frame.

7. The system of claim 1, wherein a dark frame comprises a frame shot in low or no light.

8. The system of claim 1, further comprising the step of determining whether any of the frames in the extracted subset of frames comprises a dark frame.

9. The system of claim 8, wherein the step of determining whether any of the frames in the extracted subset of frames comprises a dark frame further comprises the steps of:

for each frame, dividing the frame into a plurality of grid cells;

calculating an intensity value for each pixel in each cell;

calculating an average intensity value across all pixels in each cell;

concatenating the average intensity values for each cell in the frame to form an intensity feature vector for the frame; and

classifying the intensity feature vector via an intensity classifier to determine if the frame comprises a dark frame.

10. The system of claim 1, wherein the step of determining whether each frame in the extracted subset includes content associated with the general content category further comprises the steps of:

extracting a plurality of features from each frame;

generating a feature vector for each frame, wherein each feature vector includes the extracted features for the respective frame; and

classifying the feature vector for each frame via a general category classifier to determine whether each frame includes content associated with the general content category.

11. The system of claim 1, wherein the general content category comprises outdoor content.

12. The system of claim 1, wherein the one or more predefined scene categories are selected from the group comprising:

coast/beach, desert, forest, grassland, highway, indoor, lake/river, mountainous, open water, outdoor, sky, snow, and urban.

13. The system of claim 1, wherein each of the one or more scene classification scores represents the probability that a frame includes content associated with each of the respective predefined scene categories.

14. The system of claim 1, wherein each of the one or more scene classification scores comprises a value greater than or equal to zero and less than or equal to one.

15. The system of claim 1, wherein the representative scene classification score vector comprises a statistical property of the generated scene classification score vectors.

16. The system of claim 15, wherein the statistical property is selected from the group comprising:

average, median, maximum, and minimum.

17. The system of claim 1, wherein the step of associating the video file with the one or more scene categories further comprises the steps of:

identifying representative scene classification scores in the representative scene classification score vector that exceed a predetermined threshold value; and

for each representative scene classification score that exceeds the threshold value, associating the video file with the one or more predefined scene categories associated with the classification scores that exceeded the threshold value.

18. The system of claim 1, wherein the video file comprises a shot of video.

19. The system of claim 1, further comprising the step of generating a report including the one or more predefined scene categories associated with the video file.

20. The system of claim 1, wherein the one or more scene categories associated with the video file are used for indexing and retrieval of the video file.

21. A method for classifying videos based on video content, comprising the steps of:

receiving a video file, the video file including a plurality of frames;

extracting a subset of frames from the video file;

if one or more frames in the extracted subset of frames comprises a dark frame, discarding the one or more dark frames from the subset;

determining whether each frame in the extracted subset includes content associated with a general content category;

for each frame in the extracted subset that includes content associated with the general content category, generating a scene classification score vector for the frame via one or more scene classifiers, the scene classification score vector including one or more scene classification scores associated with one or more predefined scene categories within the general content category;

determining a representative scene classification score vector for the video file based on the generated scene classification score vectors for each extracted frame in the subset that includes content associated with the general content category; and

labeling the video file according to the one or more predefined scene categories based on the representative scene classification score vector.

22. The method of claim 21, wherein the step of generating the scene classification score vector for each frame in the extracted subset of frames that includes content associated with the general content category further comprises the steps of:

dividing the frame into one or more segments based on image content in each segment;

generating a material classification score vector for each segment, each material classification score vector including one or more material values associated with one or more predefined material content types, wherein each material value represents a probability that the respective segment includes that type of material content;

assigning the material classification score vector for each segment to each respective pixel in the segment;

generating a material arrangement vector for the frame based on the material classification score vectors assigned to each pixel; and

classifying the material arrangement vector via the one or more scene classifiers to generate the scene classification score vector for the frame.

23. The method of claim 22, further comprising the step of combining adjacent segments based on similar content properties of the segments.

24. The method of claim 22, wherein the one or more predefined material content types are selected from the group comprising:

building, grass, person, road/sidewalk, rock, sand/gravel/soil, sky/clouds, snow/ice, trees/plants, vehicle, water, and miscellaneous.

25. The method of claim 22, wherein the material arrangement vector represents the spatial arrangement of material content in the frame.

26. The method of claim 21, wherein a dark frame comprises a frame shot in low or no light.

27. The method of claim 21, further comprising the step of determining whether any of the frames in the extracted subset of frames comprises a dark frame.

28. The method of claim 27, wherein the step of determining whether any of the frames in the extracted subset of frames comprises a dark frame further comprises the steps of:

for each frame, dividing the frame into a plurality of grid cells;

calculating an intensity value for each pixel in each cell;

calculating an average intensity value across all pixels in each cell;

concatenating the average intensity values for each cell in the frame to form an intensity feature vector for the frame; and

classifying the intensity feature vector via an intensity classifier to determine if the frame comprises a dark frame.

29. The method of claim 21, wherein the step of determining whether each frame in the extracted subset includes content associated with the general content category further comprises the steps of:

extracting a plurality of features from each frame;

generating a feature vector for each frame, wherein each feature vector includes the extracted features for the respective frame; and

classifying the feature vector for each frame via a general category classifier to determine whether each frame includes content associated with the general content category.

30. The method of claim 21, wherein the general content category comprises outdoor content.

31. The method of claim 21, wherein the one or more predefined scene categories are selected from the group comprising:

coast/beach, desert, forest, grassland, highway, indoor, lake/river, mountainous, open water, outdoor, sky, snow, and urban.

32. The method of claim 21, wherein each of the one or more scene classification scores represents the probability that a frame includes content associated with each of the respective predefined scene categories.

33. The method of claim 21, wherein each of the one or more scene classification scores comprises a value greater than or equal to zero and less than or equal to one.

34. The method of claim 21, wherein the representative scene classification score vector comprises a statistical property of the generated scene classification score vectors.

35. The method of claim 34, wherein the statistical property is selected from the group comprising:

average, median, maximum, and minimum.

36. The method of claim 21, wherein the step of labeling the video file further comprises the steps of:

identifying representative scene classification scores in the representative scene classification score vector that exceed a predetermined threshold value; and

for each representative scene classification score that exceeds the threshold value, associating the video file with the one or more predefined scene categories associated with the classification scores that exceeded the threshold value.

37. The method of claim 21, wherein the video file comprises a shot of video.

38. The method of claim 21, further comprising the step of generating a report based on the labeled video file.

39. The method of claim 21, wherein the labeled one or more scene categories are used for indexing and retrieval of the video file.

40. A method for classifying videos based on video content, comprising the steps of:

receiving a video file, the video file including a plurality of frames, wherein each frame includes a plurality of pixels;

extracting a set of frames from the video file;

for each frame in the extracted set of frames, determining whether the frame comprises a poor classification frame;

if one or more frames in the extracted set of frames comprises a poor classification frame, removing the one or more poor classification frames from the extracted set of frames;

dividing each frame in the extracted set of frames into one or more segments, wherein each segment includes relatively uniform image content;

extracting image features from each segment to form a feature vector associated with each segment;

generating a material classification score vector for each segment via one or more material classifiers based on the feature vector associated with each segment, wherein each material classification score vector includes one or more material classification scores associated with one or more predefined material content categories; and

assigning each material classification score vector associated with its respective segment to each pixel in each respective segment for each respective frame in the set of frames.

41. The method of claim 40, further comprising the step of storing the material classification score vectors assigned to each pixel in a database for subsequent use in video file classification.

42. The method of claim 40, further comprising the step of combining adjacent segments based on similar image content features extracted from the segments.

43. The method of claim 40, wherein poor classification frames are determined via one or more classifiers.

44. The method of claim 40, wherein a poor classification frame comprises a frame associated with at least one of the following frame types:

a frame shot in low light, a frame shot at night, a blurry frame, and an undetermined frame.

45. The method of claim 40, wherein image features comprise one or more features selected from the group comprising:

color features, edge features, line features, texture features, and shape features.

46. The method of claim 40, wherein image features comprise data associated with image content.

47. The method of claim 40, wherein the one or more material classifiers are hierarchically related.

48. The method of claim 40, wherein the one or more predefined material content categories are selected from the group comprising:

building, grass, person, road/sidewalk, rock, sand/gravel/soil, sky/clouds, snow/ice, trees/plants, vehicle, water, and miscellaneous.

49. The method of claim 40, wherein each of the one or more material classification scores represents the probability that a frame includes content associated with each of the respective predefined material content categories.

50. The method of claim 40, wherein the video file comprises a shot of video.

51. A method for classifying a video file according to one or more scene classes, the video file including a plurality of frames, wherein each frame includes a plurality of pixels, and wherein each pixel is associated with a vector of material classification scores describing the material content in its respective frame, comprising the steps of:

(a) dividing each frame into a plurality of grid cells;

(b) for each frame, retrieving the vector of material classification scores for each pixel in each grid cell;

(c) for each grid cell, averaging the material classification scores across each pixel in the grid cell to form a material occurrence vector for the grid cell;

(d) concatenating the material occurrence vectors for the plurality of grid cells in each frame to generate a material arrangement vector for each frame;

(e) generating a scene classification score associated with each of the one or more scene classes for each frame in the video file via one or more scene classifiers based on the material arrangement vectors generated for each frame;

(f) generating a representative scene classification score for the video file for each of the one or more scene classes based on the scene classification scores generated for each frame; and

(g) if one or more of the representative scene classification scores is above a predetermined threshold value, labeling the video file according to the respective scene classes associated with the one or more scene classification scores that are above the predetermined threshold value.

52. The method of claim 51, further comprising the step of repeating steps (a)-(e) one or more times using varying numbers of grid cells.

53. The method of claim 52, wherein the one or more scene classifiers comprise a spatial pyramid of classifiers, and wherein varying weights are associated with the scene classification scores.

54. The method of claim 51, wherein the material content is selected from the group comprising:

building, grass, person, road/sidewalk, rock, sand/gravel/soil, sky/clouds, snow/ice, trees/plants, vehicle, water, and miscellaneous.

55. The method of claim 51, wherein the material occurrence vector represents the proportion of each type of material content included in the cell.

56. The method of claim 51, wherein the material occurrence vectors are concatenated in an ordered manner.

57. The method of claim 51, wherein each of the material classification scores represents the probability that a frame includes content associated with each of the respective types of material content.

58. The method of claim 51, wherein each of the one or more scene classification scores represents the probability that a frame includes content associated with each of the respective one or more scene classes.

59. The method of claim 51, wherein the material arrangement vector generated for each frame represents the spatial arrangement of material content in the frame.

60. The method of claim 51, wherein the one or more scene classes are selected from the group comprising:

coast/beach, desert, forest, grassland, highway, indoor, lake/river, mountainous, open water, outdoor, sky, snow, and urban.

61. The method of claim 51, wherein the representative scene classification score for each of the one or more scene classes comprises a statistical property of the generated scene classification scores.

62. The method of claim 61, wherein the statistical property is selected from the group comprising:

average, median, maximum, and minimum.

63. The method of claim 51, wherein the video file comprises a shot of video.

64. The method of claim 51, further comprising the step of generating a report based on the labeled video file.

65. The method of claim 51, wherein the labeled one or more scene classes are used for indexing and retrieval of the video file.

66. A method for labeling videos based on video content, comprising the steps of:

receiving a video file, wherein the video file includes a plurality of frames;

extracting a set of frames from the plurality of frames in the video file;

for each frame in the extracted set of frames, calculating a probability that the frame includes content associated with a predefined scene category;

determining a representative probability for the set of frames based on the calculated probabilities for each frame; and

if the representative probability exceeds a predetermined threshold, associating the scene category with the video file.

67. The method of claim 66, wherein the representative probability comprises a statistical property of the calculated probabilities for each frame.

68. The method of claim 67, wherein the statistical property is selected from the group comprising:

average, median, maximum, and minimum.

69. The method of claim 66, further comprising the step of indexing the video file according to the associated scene category for search and retrieval purposes.

70. The method of claim 66, wherein the scene category is selected from the list comprising:

coast/beach, desert, forest, grassland, highway, indoor, lake/river, mountainous, open water, outdoor, sky, snow, and urban.

71. The method of claim 66, wherein the calculated probability for each frame is calculated via a classifier.