Method and system for video clip compression

• US 20060228029A1
• Filed: 03/29/2005
• Published: 10/12/2006
• Est. Priority Date: 03/29/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Zhengyou, Liu, Zicheng
• Granted Patent: US 7,612,832 B2
• Time in Patent Office: Days
• US Class Current: 382/232
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence

Abstract

In a method for compressing a video clip containing audio content and image content, an image and/or an audio portion of individual video frames of the video clip are analyzed. Next frame scores are calculated for the video frames. Each frame score is based on at least one image attribute of the image of the video frame, and/or an audio attribute of the audio portion of the video frame. Next, key frames are identified that have a frame score that exceeds a threshold frame score. Finally, a compressed video clip is formed in which the images of non-key frames are removed. A system for implementing the method is also disclosed.

Claims

1. A method for compressing a video clip comprising steps of:

analyzing at least one of an image and an audio portion of individual video frames of the video clip;

calculating frame scores for the individual video frames each based on at least one of an image attribute of the image of the video frame and an audio attribute of the audio portion of the video frame;

identifying key frames of the video frames having a frame score that exceeds a threshold frame score; and

forming a compressed video clip in which the images of non-key frames are removed.

2. The method of claim 1, wherein the forming step includes:

extracting the identified key frames; and

forming a compressed video clip including the extracted key frames.

3. The method of claim 2, wherein the forming step includes synchronizing the images of the key frames with the audio portions of the video frames.

4. The method of claim 1, wherein:

each video frame includes a time code period; and

the forming step includes extending the time code period for the images of the key frames.

5. The method of claim 1, wherein the at least one image attribute is selected from a group consisting of a contrast of the image, a histogram of the image, an optical flow measurement of the image, a difference between the image and the image of a preceding video frame, a number of faces in the image, a size of a face in the image, an expression of a face in the image, a state of a mouth of a face in the image, and a state of eyes of a face in the image.

6. The method of claim 5, wherein the at least one audio attribute is selected from a group consisting of an energy level of the audio portion, a pitch of the audio portion, a change in energy level of the audio portion from the audio portion of a preceding video frame, and a change in pitch of the audio portion from the audio portion of a preceding video frame.

7. The method of claim 6, wherein the calculating step includes calculating the frame score for each video frame based on at least one attribute score for the video frame, each attribute score corresponding to a measurement of one of the image attributes and the audio attributes of the video frame.

8. The method of claim 7, wherein the calculating step includes calculating the frame score for each video frame based on at least two of the attribute scores for the video frame.

9. The method of claim 7, wherein each attribute score corresponds to a raw attribute score for the corresponding attribute multiplied by a predetermined weight value.

10. The method of claim 1 including modifying at least one of the images of the key frames.

11. A system for compressing a video clip comprising:

a video analyzer having an output of frame scores for video frames of the video clip, each frame score based on at least one of an image attribute of an image of the video frame and an audio attribute of an audio portion of the video frame;

a key frame selector having an output identifying key frames of the video frames having a frame score that exceeds a threshold frame score; and

a video compiler configured to form a compressed video clip in which the images of non-key frames of the video clip are removed.

12. The system of claim 11 including:

a key frame extractor having an output of the identified key frames; and

a video compiler receiving the key frames and having an output of a compressed video clip including the key frames.

13. The system of claim 11, wherein the at least one image attribute is selected from a group consisting of a contrast of the image, a histogram of the image, an optical flow measurement of the image, a difference between the image and the image of a preceding video frame, a number of faces in the image, a size of a face in the image, an expression of a face in the image, a state of a mouth of a face in the image, and a state of eyes of a face in the image.

14. The system of claim 13, wherein the at least one audio attribute is selected from a group consisting of an energy level of the audio portion, a pitch of the audio portion, a change in energy level of the audio portion from the audio portion of a preceding video frame, and a change in pitch of the audio portion from the audio portion of a preceding video frame.

15. The system of claim 14, wherein the video analyzer calculates the frame score for each video frame based on at least one attribute score for the video frame, each attribute score corresponding to a measurement of one of the image attributes and the audio attributes of the video frame.

16. The system of claim 15, wherein the video analyzer calculates the frame score for each video frame based on at least two attribute scores for the video frame.

17. The system of claim 15, wherein the attribute scores each correspond to a raw attribute score multiplied by a predetermined weight value.

18. The system of claim 11 including an image processor configured to modify the images of the key frames and output enhanced key frames containing the modified images.

19. A computer readable medium including instructions executable by a microprocessor for compressing a video clip, the instructions comprising:

instructions for analyzing at least one of an image and an audio portion of individual video frames of the video clip;

instructions for calculating frame scores for the individual video frames each based on at least one of an image attribute of the image of the video frame and an audio attribute of the audio portion of the video frame;

instructions for identifying key frames having a frame score that exceeds a threshold frame score; and

instructions for forming a compressed video clip in which the images of the non-key frames are removed.

20. The instructions of claim 19, wherein the at least one image attribute is selected from a group consisting of a contrast of the image, a histogram of the image, an optical flow measurement of the image, a difference between the image and the image of a preceding video frame, a number of faces in the image, a size of a face in the image, an expression of a face in the image, a state of a mouth of a face in the image, and a state of eyes of a face in the image.

21. The instructions of claim 20, wherein the at least one audio attribute is selected from a group consisting of an energy level of the audio portion, a pitch of the audio portion, a change in energy level of the audio portion from the audio portion of a preceding video frame, and a change in pitch of the audio portion from the audio portion of a preceding video frame.

22. The instructions of claim 21, wherein the instructions for calculating include instructions for calculating the frame score for each video frame based on at least one attribute score for the video frame, each attribute score corresponding to a measurement of one of the image attributes and the audio attributes of the video frame.