Forming a Representation of a Video Item and Use Thereof

• US 20090007202A1
• Filed: 06/29/2007
• Published: 01/01/2009
• Est. Priority Date: 06/29/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Fan, Xiaodong, Williams, Hugh E., Hua, Xian-Sheng, Qian, Richard, Li, Hong-Qiao
• Granted Patent: US 8,503,523 B2
• Time in Patent Office: Days
• US Class Current: 725/105
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4828 for searching program descriptors retrieval of video data G06F16/739
• CPC Class Codes: H04N 21/8549 Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739
• CPC Class Codes: H04N 7/17309 Transmission or handling of upstream communications

Abstract

Functionality is described for forming a summary representation of a video item to help a user decide whether to obtain a full version of the video item. The functionality operates by: (a) receiving a video item; (b) dividing the video item into a plurality of segments; (c) extracting at least one key frame from each of the plurality of segments to form a plurality of key frames; and (d) organizing the video segments into one or more groups corresponding to one or more respective scenes based on the plurality of key frames, to thereby form the representation of the video item. The functionality can be used to communicate search results to a user, to provide a sample of the video item in a message, etc.

Claims

1. A method for forming a representation of a video item, comprising:

receiving a video item;

dividing the video item into a plurality of segments;

extracting at least one key frame from each of the plurality of segments to form a plurality of key frames; and

organizing the video segments into one or more groups corresponding to one or more respective scenes based on the plurality of key frames, to thereby form the representation of the video item.

2. The method of claim 1, wherein each of the plurality of video segments corresponds to a portion of the video item demarcated by a camera start-recording event and a camera stop-recording event.

3. The method of claim 1, wherein the dividing of the video item into a plurality of segments comprises:

determining a visual feature for each frame of the video item; and

detecting boundaries of the segments based on the visual feature.

4. The method of claim 3, wherein the visual feature relates to a manner in which a color histogram of a frame varies with respect to at least one neighboring frame.

5. The method of claim 1, wherein the dividing of the video item into a plurality of segments comprises:

determining at least one video segment that does not meet at least one prescribed criterion; and

omitting said at least one video segment.

6. The method of claim 1, wherein the extracting of at least one key frame for each of the video segments comprises, for a particular video segment:

determining a frame stability feature for each frame in the particular video segment;

determining a frame visual quality feature for each frame in the particular video segment; and

selecting a key frame for the particular video segment based on the frame stability feature and the visual quality feature.

7. The method of claim 6, wherein the frame stability feature relates to a manner in which a color histogram of a frame varies with respect to at least one neighboring frame.

8. The method of claim 6, wherein the frame visual quality feature relates to entropy of a color histogram of a frame.

9. The method of claim 6, further comprising determining a user attention feature for each frame in the particular video segment, wherein the user attention feature reflects a relative importance of subject matter in the frame to a user, wherein the selecting is additionally based on the user attention feature.

10. The method of claim 1, wherein the organizing comprises:

forming a pairwise visual similarity feature for each pair of key frames, wherein the pairwise visual similarity feature reflects a measure of visual similarity between the pair of key frames;

forming a pairwise semantic correlation feature for each pair of key frames, wherein the pairwise semantic correlation feature reflects a measure of correlation between the pair of key frames;

determining a distance between each pair of consecutive videos segments based on the pairwise visual similarity feature and the pairwise semantic correlation feature, to thereby produce distance information; and

organizing the video segments into said one or more groups based on the distance information.

11. The method of claim 10, wherein the pairwise visual similarity feature relates to a manner in which a color histogram of a key frame varies with respect to another key frame.

12. The method of claim 10, wherein the pairwise semantic similarity feature relates to a manner in which a color histogram of a key frame is correlated to the color histogram of another key frame.

13. The method of claim 1, further comprising selecting a video vignette from each of said one or more groups.

14. One or more machine-readable media containing machine-readable instructions for implementing the method of claim 1.

15. One or more computing devices, comprising:

one or more processors; and

memory to store computer-executable instructions that, when executed by the one or more processors, perform the method of claim 1.

16. A method for presenting a representation of a video item, comprising:

presenting a key frame associated with at least one scene of the video item;

receiving the user'"'"'s selection of the key frame; and

presenting a video vignette in response to the user'"'"'s selection of the key frame, wherein the video vignette is selected to give the user information to make a decision as to whether to receive the full video item.

17. The method of claim 16, further comprising receiving a search selection from the user, wherein the key frame is presented to the user in response to the receipt of the search selection.

18. The method of claim 16, further comprising:

in response to the user'"'"'s selection of the key frame, presenting additional key frames associated with additional respective scenes of the video item; and

receiving the user'"'"'s selection of the originally selected key frame or one or the additional key frames,

wherein the video vignette is presented in response to the user'"'"'s selection of the originally selected key frame or one of the additional key frames.

19. The method of claim 16, wherein the key frame is presented in a message received from another user.

20. A summary generation module for forming a representation of a video item, comprising:

a video segmentation module configured to divide a video item into a plurality of segments;

a key frame extraction module configured to extract at least one key frame from each of the plurality of segments to form a plurality of key frames; and

a grouping module configured to organize the video segments into one or more groups corresponding to one or more respective scenes based on the plurality of key frames, to thereby form the representation of the video item.