Space-Time Video Montage

• US 20080112684A1
• Filed: 11/14/2006
• Published: 05/15/2008
• Est. Priority Date: 11/14/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Tang, Xiaoou, Matsushita, Yasuyuki, Kang, Hong-Wen
• Granted Patent: US 8,000,533 B2
• Time in Patent Office: Days
• US Class Current: 386/52
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/785 using colour or luminescence
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence

Abstract

Systems and methods for space-time video montage are described. In one aspect, one or more arbitrary space-time volumes representing informative video portion(s) of at least one input video data sequence are identified. A video summary representing a montage of the at least one input video data sequence is generated for presentation to user from the one or more arbitrary space-time volumes.

Claims

1. A method at least partially implemented by a computing device, the method comprising:

identifying one or more arbitrary space-time volumes representing informative video portion(s) of at least one input video data sequence; and

generating a video summary montage of the input video data sequence, the video summary montage being based on the one or more arbitrary space-time volume, the video summary montage for presentation to a user.

2. The method of claim 1, wherein the video summary montage represents a spatial scale-down of the input video data sequence, a temporal scale-down of the input video data sequence, or a space-time scale down of the input video data sequence.

3. The method of claim 1, wherein the at least one input video data sequence is multiple input video data sequences, each input video data sequence representing a different video clip.

4. The method of claim 1, wherein the method further comprises:

analyzing spatio-temporal information distribution in the input video data sequence to locate the one or more arbitrary space-time volumes;

extracting, based on the analyzing, visually informative space-time portions of the input video data sequence;

representing respective ones of the visually informative space-time portions as respective volumetric saliency layers; and

wherein the video summary montage is based on an arrangement of at least a subset of the information associated with the volumetric saliency layers.

5. The method of claim 1, wherein the method further comprises presenting the video summary montage to a user.

6. The method of claim 1, wherein the method further comprises:

representing respective ones of the one or more arbitrary space-time volumes as respective volumetric saliency layers;

identifying, for each volumetric saliency layer, relative saliency distributions of pixels in the saliency layer; and

wherein the video summary montage is based on the relative saliency distributions of pixels across the volumetric saliency layers.

7. The method of claim 6, wherein identifying the relative saliency distributions further comprises, with respect to the input video data sequence, locating high saliency portions, mid-level saliency portions, and low-level-saliency portions.

8. The method of claim 6, wherein identifying the relative saliency distributions further comprises dilating high-saliency portions of the input video data sequence to simulate effect of the high-saliency portions on adjacent pixels.

9. The method of claim 6, wherein identifying the relative saliency distributions further comprises assigning negative saliency values to respective ones of the pixels in a least a subset of the volumetric saliency layers, the negative saliency values indicating preference of higher salient portions of pixels in one or more different layers of the volumetric saliency layers.

10. The method of claim 1, wherein the method further comprises:

representing respective ones of the one or more arbitrary space-time volumes as respective volumetric saliency layers;

identifying, for each volumetric saliency layer, relative saliency distributions of pixels in the saliency layer;

packing and merging data associated with the volumetric saliency layers into a 3-D video volume based on the relative saliency distributions of pixels in the saliency layers, the 3-D video volume being the video summary montage.

11. The method of claim 10, wherein each pixel in the 3-D video volume is associated with a respective saliency measurement.

12. The method of claim 10, wherein one or more of the volumetric saliency layers that have been packed and merged into the 3-D video volume overlap one or more different ones of the volumetric saliency layers.

13. The method of claim 10, wherein the packing is based on a first-fit algorithm and wherein the merging is based on a graph cut optimization algorithm.

14. The method of claim 10, wherein the packing further comprises positioning the volumetric saliency layers in the 3-D video volume to maximize overall saliency in the 3-D video volume.

15. The method of claim 10, wherein the merging is based on a set of soft constraints that maximize saliency, maximize continuity of high-saliency portions, and maximize color smoothness at scene boundaries of the volumetric saliency layers.

16. A tangible computer-readable data storage medium comprising computer-program instructions executable by a processor, the computer-program instructions when executed by the processor performing operations comprising:

extracting visually informative space-time portions from an input video data sequence;

representing respective ones of the visually informative space-time portions as respective volumetric saliency layers;

positioning at least a subset of the information associated with the volumetric saliency layers into a 3-D video volume to maximize saliency of pixels in the 3-D video volume;

merging data associated with the at least a subset of the information in the 3-D video volume to regulate continuity of high-saliency portions of the pixels and provide color coherence at boundaries between respective ones of pixels in the volumetric saliency layers; and

wherein the 3-D video volume represents a video summary of the input video data sequence for presentation to a user.

17. The computer-readable medium of claim 16, wherein the method further comprises presenting the video summary montage to a user.

18. The computer-readable medium of claim 16, wherein representing further comprises:

locating in the informative space-time portions, high saliency portions, mid-level saliency portions, and low-level-saliency portions;

dilating high-saliency portions of the informative space-time portions to simulate effect of the high-saliency portions on adjacent pixels; and

assigning negative saliency values to respective ones of the pixels in a least a subset of the volumetric saliency layers, the negative saliency values indicating preference of higher salient portions of pixels in one or more different layers of the volumetric saliency layers.

19. The computer-readable medium of claim 16, wherein one or more of the volumetric saliency layers overlap one or more different ones of the volumetric saliency layers in the 3-D video volume.

20. A computing device comprising:

a processor; and

a memory coupled to the processor, the memory comprising computer-program instructions executable by the processor, the computer-program instructions when executed by the processor for performing operations comprising;

evaluating an input video data sequence as 3-D data to cut off spatio-temporal redundancies, the evaluating comprising;

locating spatio-temporal informative video portions of the input video data sequence by evaluating texture levels;

representing the spatio-temporal informative video portions as volumetric saliency layers to depict pixel content saliency distributions;

packing and merging a subset of data associated with the volumetric saliency layers into a 3-D volume for presentation as a video summary montage to a user; and

presenting the 3-D volume as a video summary of the input video data sequence to user.