IDENTIFYING PRESENTATION STYLES OF EDUCATIONAL VIDEOS

• US 20160026872A1
• Filed: 07/23/2014
• Published: 01/28/2016
• Est. Priority Date: 07/23/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Kannan, Anitha, Baker, Simon John
• Granted Patent: US 9,652,675 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/735 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: G06F 16/738 Presentation of query results
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/784 the detected or recognised objects being people
• CPC Class Codes: G06F 16/7847 using low-level visual features of the video content
• CPC Class Codes: G06F 16/7867 using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G09B 5/02 with visual presentation of the material to be studied, e.g. using film strip

Abstract

The presentation style of a video is identified. A set of features that represents the video is computed. A pre-learned video presentation style classifier is then used to weight each of the features in the set of features and determine a presentation style that is predominately employed in the video based on the weighting of the features.

Claims

1. A computer-implemented process for identifying the presentation style of a video, comprising the actions of:

using one or more computing devices that are in communication with each other via a computer network to perform the following process actions;

receiving the video;

computing a set of features that represents the video; and

using a pre-learned video presentation style classifier to weight each of the features in said set and determine a presentation style that is predominately employed in the video, said presentation style determination being based on the weighting of said features.

2. The process of claim 1, wherein the determined presentation style comprises a one of the presentation styles in a set of possible presentation styles comprising:

a rendered video class of presentation styles; and

a real-world video class of presentation styles.

3. The process of claim 2, wherein the video comprises an educational video, and the rendered video class of presentation styles comprises one or more of:

a rendered slide show presentation style;

or

a rendered slide show comprising a video of a presenter presentation style;

or

a rendered animation presentation style;

or

a rendered photographs presentation style;

or

a rendered hand-drawn slides presentation style.

4. The process of claim 2, wherein the video comprises an educational video, and the real-world video class of presentation styles comprises one or more of:

a natural video presentation style;

or

a video of an interview presentation style;

or

a video of handwriting on paper presentation style;

or

a video of projected slides presentation style;

or

a video of a whiteboard presentation style;

or

a video of a blackboard presentation style.

5. The process of claim 1, wherein the set of features that represents the video comprises image features that are computed for each frame of the video independently, the image features comprising one or more of:

a low-contrast feature measuring pixel intensity statistics for the video according to a prescribed low-contrast threshold;

or

a high-contrast feature measuring pixel intensity statistics for the video according to a prescribed high-contrast threshold;

or

a zero-gradients feature measuring the amount of zero gradients that exist in the video;

or

a low-gradients feature measuring the amount of weak but non-zero gradients that exist in the video;

or

a high-gradients feature measuring the amount of strong gradients that exist in the video;

or

a noise feature measuring the amount of pixel intensity noise that exists in the video.

6. The process of claim 1, wherein the set of features that represents the video comprises face features that are based on the detection of one or more faces in the video, the face features comprising one or more of:

a face detection feature measuring the percentage of frames in the video where just one face is detected;

or

one or more moving face features each measuring, for each frame of the video where just one face is detected, whether or not said detected face is moving;

or

a face not present feature measuring the length of the longest sequence of frames in the video where no face is detected;

or

a face present feature measuring the length of the longest sequence of frames in the video where just one face is detected;

or

a face size feature measuring, across the frames in the video where just one face is detected, the average size of said detected face.

7. The process of claim 1, wherein the set of features that represents the video comprises motion features that are based on how the video changes from frame to frame, the motion features comprising one or more of:

frequency of motion features measuring how often motion occurs in the video;

or

amount of motion features measuring how much motion takes place in the video;

or

type of motion features specifying the type of motion that takes place in the video.

8. The process of claim 7, wherein the frequency of motion features comprise one or more of:

a motion frequency feature measuring the percentage of frames in the video where the magnitude of motion is greater than or equal to a prescribed motion frequency threshold;

or

another motion frequency feature measuring the percentage of frames in the video where the magnitude of motion is greater than or equal to another prescribed motion frequency threshold;

or

a motion present feature measuring the length of the longest sequence of frames in the video where there is motion;

or

a motion not present feature measuring the length of the longest sequence of frames in the video where there is no motion.

9. The process of claim 7, wherein the amount of motion features comprise one or more of:

a motion amount feature measuring the number of pixels in the video whose intensity changes between successive frames of the video according to a prescribed motion pixel threshold;

or

another motion amount feature measuring the number of pixels in the video whose intensity changes between successive frames of the video according to another prescribed motion pixel threshold.

10. The process of claim 7, wherein the type of motion features comprise one or more of:

a motion type feature (feat_mott1) given by the equation feat_mott1=Percentile_f (NRFlow(f), T_mott1), NRFlow(f) denoting the magnitude of non-rigid motion between a given frame (f) of the video and the immediately succeeding frame (f+1) of the video, T_mott1 denoting a prescribed motion type threshold, and Percentile_f sorting the values of NRFlow(f) across all of the frames of the video and then selecting the value of NRFlow(f) at the T_mott1 percentile;

or

another motion type feature (feat_mott2) given by the equation feat_mott2=Percentile_f (NRFlow(f)/OFlow(f), T_mott2), OFlow(f) denoting the magnitude of optical flow across frame f, T_mott2 denoting another prescribed motion type threshold, NRFlow(f)/OFlow(f) denoting the fraction of OFlow(f) that is non-rigid, and Percentile_f sorting the values of NRFlow(f)/OFlow(f) across all of the frames of the video and then selecting the value of NRFlow(f)/OFlow(f) at the T_mott2 percentile;

or

yet another motion type feature (feat_mott3) given by the equation feat_mott3=Percentile_f (OFRes(f), T_mott3), OFRes(f) denoting an optical flow residual indicating the degree to which changes between frame f and frame f+1 are due to the motion of scene elements in the video, T_mott3 denoting yet another prescribed motion type threshold, and Percentile_f sorting the values of OFRes(f) across all of the frames of the video and then selecting the value of OFRes(f) at the T_mott3 percentile.

11. A system for learning a video presentation style classifier, comprising:

one or more computing devices, said computing devices being in communication with each other via a computer network whenever there is a plurality of computing devices; and

a computer program having program modules executable by the one or more computing devices, the one or more computing devices being directed by the program modules of the computer program to,

receive a training dataset comprising a plurality of videos each of which is labeled with a presentation style identifier specifying the presentation style that is predominately employed in the video, said specified presentation style being a one of the presentation styles in a set of possible presentation styles;

use the training dataset to independently learn a different classifier for each possible unordered pair of presentation styles in the set of possible presentation styles; and

combine the different classifiers using probabilistic fusion, said combination producing the video presentation style classifier.

12. The process of claim 11, wherein the plurality of videos in the training dataset comprise educational videos, and the set of possible presentation styles comprises a rendered video class of presentation styles comprising one or more of:

a rendered slide show presentation style;

or

a rendered slide show comprising a video of a presenter presentation style;

or

a rendered animation presentation style;

or

a rendered photographs presentation style;

or

a rendered hand-drawn slides presentation style.

13. The process of claim 11, wherein the plurality of videos in the training dataset comprise educational videos, and the set of possible presentation styles comprises a real-world video class of presentation styles comprising one or more of:

a natural video presentation style;

or

a video of an interview presentation style;

or

a video of handwriting on paper presentation style;

or

a video of projected slides presentation style;

or

a video of a whiteboard presentation style;

or

a video of a blackboard presentation style.

14. The process of claim 11, wherein the different classifier that is independently learned for each possible unordered pair of presentation styles in the set of possible presentation styles comprises a three-way classifier that is learned between a given unordered pair of presentation styles in said set and an additional background category comprising all of the different presentation styles in said set except for those that are in said given unordered pair.

15. The process of claim 11, wherein the different classifier that is independently learned for each possible unordered pair of presentation styles in the set of possible presentation styles and the video presentation style classifier are trained using a Decision Forests method.

16. The process of claim 11, further comprising an action of computing a set of features that represents each of the videos in the training dataset, said features comprising:

one or more image features that are computed for each frame of the video independently;

one or more face features that are based on the detection of one or more faces in the video; and

one or more motion features that are based on how the video changes from frame to frame.

17. A computer-implemented process for allowing a user to search for videos, comprising the actions of:

using one or more computing devices that are in communication with each other via a computer network to perform the following process actions;

implicitly learning the presentation style preferences of the user, said implicit learning comprising;

for each video that the user views,

computing a set of features that represents the video, and

using a pre-learned video presentation style classifier to weight each of the features in said set and determine a presentation style that is predominately employed in the video, said presentation style determination being based on the weighting of said features;

receiving a user video query;

submitting the user video query to a search engine;

receiving search results for the user video query from the search engine; and

whenever the user video query does not explicitly specify one or more particular presentation styles the user is interested in,

using the learned presentation style preferences of the user to refine the search results, and

providing the refined search results to the user.

18. The process of claim 17, wherein the process action of using the learned presentation style preferences of the user to refine the search results comprises an action of either:

using said preferences to filter the search results such that the refined search results are restricted to videos that match said preferences;

or

using said preferences to rank order the search results such that videos matching said preferences appear at the top of the refined search results.

19. The process of claim 17, further comprising the actions of, whenever the user video query explicitly specifies one or more particular presentation styles the user is interested in,

using said specified presentation styles to refine the search results, and

providing the refined search results to the user.

20. The process of claim 19, wherein the process action of using said specified presentation styles to refine the search results comprises an action of either:

using said specified presentation styles to filter the search results such that the refined search results are restricted to videos that match said specified presentation styles;

or

using said specified presentation styles to rank order the search results such that videos matching said specified presentation styles appear at the top of the refined search results.