Rule-based video importance analysis

• US 10,664,687 B2
• Filed: 06/12/2014
• Issued: 05/26/2020
• Est. Priority Date: 06/12/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Suri, Nitin, Wang, Tzong-Jhy, Mehendale, Omkar, Ivory, Andrew S., Sproule, William D.
• Primary Examiner(s): Tran, Thai Q
• Assistant Examiner(s): Hasan, Syed Y
• Application Number:
  US14/303,466
• Publication Number:
  US 20150363635A1
• Time in Patent Office: 2,175 Days
• Field of Search: 386241, 382103, 382107, 707769, 715720, 715723, 348580, 348581, 600595
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence

Abstract

The importance of video sections of a video file may be determined from features of the video file. The video file may be decoded to obtain video frames and audio data associated with the video frames. Feature scores for each video frame may be obtained by analyzing features of the video frame or the audio data associated with the video frame based on a local rule, a global rule, or both. The feature scores are further combined to derive a frame importance score for the video frame. Based on the feature scores of the video frames in the video file, the video file may be segmented into video sections of different section importance values.

Claims

1. One or more computer storage media storing computer-executable instructions that are executable to cause one or more processors to perform acts comprising:

decoding a video file to obtain at least one of a video frame or audio data associated with the video frame;

analyzing one or more high-level features of the at least one of the video frame or the audio data associated with the video frame based on at least one of a local rule or a global rule to obtain one or more feature scores for the video frame, the one or more high-level features including at least one feature that is related to a movement or a position of a human face in the video frame;

combining the one or more feature scores of the video frame to derive a frame importance score for the video frame; and

segmenting the video file into video sections of different section importance values based at least on the frame importance scores of the video frame and other frame importance scores of additional video frames of the video file.

2. The one or more computer storage media of claim 1, further comprising ranking the video sections according to the section importance values.

3. The one or more computer storage media of claim 1, further comprising:

calculating a video importance value for the video file based at least on the frame importance scores of all video frames in the video file; and

ranking the video file relative to additional video files based at least on the video importance value of the video file and other video importance values of the additional video files.

4. The one or more computer storage media of claim 3, further comprising storing metadata regarding one or more video frames in the video file, the metadata including one or more feature scores of a video frame.

5. The one or more computer storage media of claim 1, wherein the local rule affects an assignment of feature importance for the video frame and the global rule affects the assignment of feature importance for multiple video frames in the video file that includes the video frame.

6. The one or more computer storage media of claim 1, wherein the one or more high-level features includes at least one of a position of a human face in the video frame, an occurrence frequency of the human face over multiple video frames, frame saliency of the video frame, an audio power of the audio data associated with the video frame, or an audio type of the audio data associated with the video frame.

7. The one or more computer storage media of claim 6, wherein the one or more low-level features includes at least one of a frame exposure quality of the video frame, a frame saturation quality of the video frame, a frame hue variety of the video frame, a frame shakiness of the video frame, an average brightness of the video frame, a color entropy of the video frame, or a histogram difference between the video frame and an adjacent video frame.

8. The one or more computer storage media of claim 6, wherein analysis of frame saliency of the video frame includes:

calculating a running average of motion data of the video frames;

determining a shakiness movement of the video frames;

including the shakiness movement in the running average;

determining an intentional movement of the video frames; and

subtracting the intentional movement from the running average.

9. The one or more computer storage media of claim 1, wherein the analyzing further includes analyzing one or more low-level features of the video frame, and wherein the combining further includes combining the feature scores for the high-level features and the low-level features of the video frame to derive the frame importance score for the video frame.

10. The one or more computer storage media of claim 1, wherein the analyzing includes:

executing face detection on the video frame to detect one or more human faces;

executing face tracking to track the one or more human faces;

assigning a negative or a positive score bias to the video frame based at least on a face size, a face position, or one or more facial feature of at least one human face in the video frame, the one or more facial features including an absence or presence of at least one of a smile or open eyes in the at least one human face;

executing at least one of face recognition or face grouping on at least one human face of the one or more human faces;

identifying one or more main characters based at least on at least one of face grouping data or face recognition data;

assigning an additional positive score bias to the video frame for a presence of a main character in the video frame; and

computing a face importance score for the video frame that accounts for the negative score bias and the positive score bias.

11. A computer-implemented method, comprising:

decoding a video file to obtain at least one of a video frame or audio data associated with the video frame;

analyzing one or more features of at least one of the video frame or the audio data associated with the video frame based on at least one of a local rule or a global rule to obtain one or more feature scores for the video frame;

combining the one or more feature scores of the video frame to derive a frame importance score for the video frame;

calculating a video importance value for the video file based at least on the frame importance scores of the video frame and other frame importance scores of additional video frames of the video file;

segmenting the video file into video sections of different section importance based at least on the frame importance scores of video frames in the video file data associated with a plurality of video frames in the video file; and

ranking the video file relative to additional video files based at least on video importance values of the video file and other video importance values of the additional video files.

12. The computer-implemented method of claim 11, wherein the local rule affects an assignment of feature importance for the video frame and the global rule affects the assignment of feature importance for multiple video frames in the video file that includes the video frame.

13. The computer-implemented method of claim 11, wherein the segmenting the video file based at least on the motion data associated with a plurality of video frames in the video file includes:

detecting feature points in the video frame of the video file by detecting common points between multiple down sampled images of the video frame;

matching the feature points in the video frame to an additional set of feature points in a preceding video frame of the video file;

calculating a homography transform matrix that describes movement of the feature points between the video frames;

computing motion data for the video frame from the homography transform matrix, the motion data including magnitude and direction of zooming and vertical translation components of the movement of the feature points;

combining the motion data for the video frame with motion data for one or more remaining frames in the video file;

calculating running averages for the motion data of the video frames in the video file;

ascertaining zero crossing zones to find local peak and valley information for the motion data based at least on the running averages; and

determining video sections of the video file based at least on the local peak and valley information of the motion data.

14. The computer-implemented method of claim 11, further comprising:

calculating section importance values of the video sections based at least on frame importance scores of corresponding video frames in the video sections; and

ranking the video sections according to the section importance values.

15. The computer-implemented method of claim 11, wherein the one or more features includes frame saliency of the video frame, an audio power of the audio data associated with the video frame, an audio type of the audio data associated with the video frame, a frame exposure quality of the video frame, a frame saturation quality of the video frame, a frame hue variety of the video frame, a frame shakiness of the video frame, an average brightness of the video frame, a color entropy of the video frame, or a histogram difference between the video frame and an adjacent video frame.

16. The computer-implemented method of claim 11, wherein the analyzing includes:

executing face detection on a video frame to detect one or more human faces;

executing face tracking to track the one or more human faces;

assigning a negative score bias or positive score bias to the video frame based at least on a face size, a face position, or one or more facial features of at least one human face in the video frame, the one or more facial features including an absence or presence of at least one of a smile or open eyes in the at least one human face;

executing at least one of face recognition or face grouping on at least one human face of the one or more human faces;

identifying one or more main characters based at least on at least one of face grouping data or face recognition data;

assigning a positive score bias to the video frame for a presence of a main character in the video frame; and

computing a face importance score for the video frame that accounts for the negative score bias and the positive score bias.

17. A system, comprising:

one or more processors;

a memory that includes a plurality of computer-executable components that are executable by the one or more processors to perform a plurality of actions, the plurality of actions comprising;

decoding a video file to obtain at least one of a video frame or audio data associated with the video frame;

analyzing one or more features of the at least one of the video frame or the audio data associated with the video frame based on at least one of a local rule or a global rule to obtain one or more feature scores for the video frame, the local rule affects an assignment of feature importance for the video frame and the global rule affects the assignment of feature importance for multiple video frames in the video file that includes the video frame;

combining the one or more feature scores of the video frame to derive a frame importance score for the video frame;

segmenting the video file into video sections of different section importance based at least on the frame importance scores of video frames in the video file associated with a plurality of video frames in the video file; and

ranking the video sections according section importance values of the video sections that are calculated based at least on the frame importance scores of the video frame and the additional video frames in the video file.

18. The system of claim 17, further comprising:

calculating a video importance value for the video file based at least on the frame importance scores of the video frames in the video file; and

ranking the video file relative to additional video files based at least on the video importance value of the video file and other video importance values of additional video files.

19. The system of claim 17, wherein the segmenting the video file includes:

detecting feature points in the video frame of the video file by detecting common points between multiple down sampled images of the video frame;

matching the feature points in the video frame to an additional set of feature points in a preceding video frame of the video file;

calculating a homography transform matrix that describes movement of the feature points between the video frames;

computing motion data for the video frame from the homography transform matrix, the motion data including magnitude and direction of zooming and vertical translation components of the movement of the feature points;

combining the motion data for the video frame with motion data for one or more remaining frames in the video file;

calculating running averages for the motion data of the video frames in the video file;

ascertaining zero crossing zones to find local peak and valley information for the motion data based at least on the running averages; and

determining video sections of the video file based at least on the local peak and valley information of the motion data.

20. The system of claim 17, further comprising combining multiple video sections into a highlight video file of the video file.