Method and system for video clip compression

• US 7,612,832 B2
• Filed: 03/29/2005
• Issued: 11/03/2009
• Est. Priority Date: 03/29/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Zhengyou, Liu, Zicheng
• Primary Examiner(s): Tran; Thai
• Assistant Examiner(s): KHAN, ASHER R
• Application Number:
  US11/092,389
• Publication Number:
  US 20060228029A1
• Time in Patent Office: 1,680 Days
• Field of Search: 348/700, 375/240.11, 375/240.25
• US Class Current: 348/700
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence

Abstract

In a method for compressing a video clip containing audio content and image content, an image and/or an audio portion of individual video frames of the video clip are analyzed. Next frame scores are calculated for the video frames. Each frame score is based on at least one image attribute of the image of the video frame, and/or an audio attribute of the audio portion of the video frame. Next, key frames are identified that have a frame score that exceeds a threshold frame score. Finally, a compressed video clip is formed in which the images of non-key frames are removed. A system for implementing the method is also disclosed.

Claims

1. A method for compressing a video clip, implemented by a computer comprising a tangible storage medium, the method comprising:

analyzing at least one of an image and an audio portion of individual video frames of the video clip;

calculating frame scores for the individual video frames, each frame score based on at least one of an image attribute of the image of the video frame and an audio attribute of the audio portion of the video frame;

identifying key frames of the video frames having a frame score that exceeds a threshold frame score;

forming a compressed video clip in which the images of non-key video frames are removed, the compressed video clip comprising the images of the key frames, the audio portions of the key frames and the audio portions of the non-key video frames.

2. The method of claim 1, wherein forming a compressed video clip in which the images of non-key video frames are removed includes synchronizing the images of the identified key frames with the audio portions of the key frames and the non-key video frames.

3. The method of claim 1, wherein:

each video frame includes a time code period; and

the method further comprises filling in image gaps of the non-key video frames of the compressed video clip with the images of the key frames including extending the time code period for the images of the key frames.

4. The method of claim 1, wherein:

each frame score is based on a contrast measurement of the image of the video frame, which indicates a relative intensity difference between neighboring pixels of the image of the video frame; and

calculating frame scores for the individual video frames comprises assigning higher frame scores to the video frames whose images have a higher relative intensity difference between neighboring pixels than the video frames whose images have a lower relative intensity difference between neighboring pixels.

5. The method of claim 4, wherein each frame score is based on a change in pitch of the audio portion from the audio portion of a preceding video frame.

6. The method of claim 1 including modifying at least one of the images of the key frames.

13. The method of claim 1, further comprising filling in image gaps of the non-key video frames of the compressed video clip with the images of the key frames.

14. The method of claim 1, wherein:

each frame score is based on a histogram of the image of the video frame, from which a dynamic range for the image is obtained; and

calculating frame scores for the individual video frames comprises assigning higher frame scores to the video frames whose images have a higher dynamic range than the video frames whose images have a lower dynamic range.

15. The method of claim 1, wherein:

each frame score is based on a measurement of a magnitude of optical flow of the image of the video frame; and

calculating frame scores for the individual video frames comprises assigning higher frame scores to the video frames whose images have a lower magnitude of optical flow than the video frames whose images have a higher magnitude of optical flow.

7. A system for compressing a video clip comprising:

a video analyzer having an output of frame scores for video frames of the video clip, each frame score based on at least one of an image attribute of an image of the video frame and an audio attribute of an audio portion of the video frame;

a key frame selector having an output identifying key frames of the video frames having a frame score that exceeds a threshold frame score; and

a video compiler configured to form a compressed video clip in which the images of non-key frames of the video clip are removed, the compressed video clip comprising the images of the key frames, the audio portions of the key frames and the audio portions of the non-key video frames.

8. The system of claim 7, wherein:

each frame score is based on contrast measurement of the image of the video frame, which indicates a relative intensity difference between neighboring pixels of the image of the video frame; and

the video analyzer assigns higher frame scores for the individual video frames whose images have a higher relative intensity difference between neighboring pixels than the video frames whose images have a lower relative intensity difference between neighboring pixels.

9. The system of claim 8, wherein each frame score is based on a change in pitch of the audio portion from the audio portion of a preceding video frame.

16. The system of claim 7, wherein the video compiler is further configured to fill in image gaps of the non-key video frames of the compressed video clip with the images of the key frames.

17. The system of claim 7, wherein the video compiler is further configured to extend a time code period for the images of the key frames into the non-key video frames of the compressed video clip.

18. The method of claim 7, wherein:

each frame score is based on a histogram of the image of the video frame, from which a dynamic range for the image is obtained; and

the video analyzer assigns higher frame scores for the individual video frames whose images have a higher dynamic range than the video frames whose images have a lower dynamic range.

19. The method of claim 7, wherein:

each frame score is based on a measurement of a magnitude of optical flow of the image of the video frame; and

the video analyzer assigns higher frame scores to the video frames whose images have a lower magnitude of optical flow than the video frames whose images have a higher magnitude of optical flow.

10. A tangible computer storage medium including instructions executable by a microprocessor of a computing device for compressing a video clip, the instructions comprising:

instructions for analyzing at least one of an image and an audio portion of individual video frames of the video clip;

instructions for calculating frame scores for the individual video frames, each frame score based on at least one of an image attribute of the image of the video frame and an audio attribute of the audio portion of the video frame;

instructions for identifying key frames having a frame score that exceeds a threshold frame score; and

instructions for forming a compressed video clip comprising the images and audio portions of the key frames and the audio portions of the non-key frames, wherein the images of the non-key frames are not included in the compressed video clip.

11. The instructions of claim 10, wherein:

each frame score is based on a contrast measurement of the image of the video frame, which indicates a relative intensity difference between neighboring pixels of the image of the video frame; and

instructions for calculating frame scores for the individual video frames comprise instructions for assigning higher frame scores to the video frames whose images have a higher relative intensity difference between neighboring pixels than the video frames whose images have a lower relative intensity difference between neighboring pixels.

12. The instructions of claim 11, wherein each frame score is based on a change in pitch of the audio portion from the audio portion of a preceding video frame.

20. The instructions of claim 10, further comprising instructions for filling in image gaps of the non-key video frames of the compressed video clip with the images of the key frames.

21. The instructions of claim 10, wherein:

each frame is based on a histogram of the image of the video frame, from which a dynamic range for the image is obtained; and

instructions for calculating frame scores for the individual video frames comprise assigning higher frame scores to the video frames whose images have a higher dynamic range than the video frames whose images have a lower dynamic range.

22. The instructions of claim 10, wherein:

each frame score is based on a measurement of a magnitude of optical flow of the image of the video frame; and

instructions for calculating frame scores for the individual video frames comprise assigning higher frame scores to the video frames whose images have a lower magnitude of optical flow than the video frames whose images have a higher magnitude of optical flow.