Video Collage Presentation

• US 20090003712A1
• Filed: 03/25/2008
• Published: 01/01/2009
• Est. Priority Date: 06/28/2007
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Mei, Tao, Hua, Xian-Sheng, Li, Shipeng
• Application Number:
  US12/055,267
• Publication Number:
  US 20090003712A1
• Time in Patent Office: Days
• US Class Current: 382/225
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/745 the internal structure of a single video sequence
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames

Abstract

A method, a computer-readable storage media, and a user interface describe techniques for creating a video collage synthesized from video content, selecting representative images from the video content, extracting and resizing regions of interest (ROI) from the representative images from the video content, and arranging the regions of interest on a canvas without seams while preserving a temporal structure of the video content. The described method, computer-readable storage, and user interface enhance the experience of the user in browsing a video collage that is compact.

Claims

1. A method for constructing a video collage, implemented at least in part by a computing device, the method comprising:

selecting representative images from a video content;

extracting and resizing regions of interest (ROI) from the representative images from the video content; and

arranging the regions of interest on a canvas and preserving a temporal structure of the regions of interest.

2. The method of claim 1, further comprising formulating an energy minimization equation to maximize representativeness of the video content and to minimize transition between the regions of interest.

3. The method of claim 1, wherein selecting representative images comprises measuring a saliency, a quality, and a distribution of a selected image, wherein the saliency is based on an importance of a visual information embedded in a selected image.

4. The method of claim 1, wherein resizing the regions of interest comprises using a bilinear interpolation based on a saliency of an image, such that the saliency is based on an importance of a visual information embedded in the image.

5. The method of claim 1, wherein arranging the regions of interest comprises the ROI within a same sub-shot is blending based on a camera motion, the ROI do not overlap, and a neighboring ROI are in a seamless transition.

6. The method of claim 1, wherein the temporal structure of the video content is consistent with a spatial layout of a selected region of interest, wherein the spatial layout includes a left to a right layout and a top to a down layout.

7. The method of claim 1, wherein arranging the regions of interest comprises arbitrary shaped regions of interest with design styles that include a book, a diagonal, or a spiral.

8. The method of claim 1, further comprising using a Gaussian distribution to avoid overlapping the regions of interest.

9. The method of claim 1, further comprising the regions of interest within a same sub-shot is blended based on a camera motion, wherein the camera motion includes panning by horizontally blending and tilting by vertically blending the images from the same sub-shot.

10. A computer-readable storage media comprising computer-executable instructions that, when executed, perform the method as recited in claim 1.

11. A computer-readable storage media comprising computer-readable instructions executed on a computing device, the computer-readable instructions comprising instructions for:

utilizing a video content to select representative images from the video content;

generating a video collage from the video content by extracting and resizing regions of interest (ROI) from representative images, wherein the ROI is based on an importance of a visual information embedded in the representative images;

preserving a temporal structure of the video content; and

creating the video collage with the regions of interest on a canvas and in a compact layout.

12. The computer-readable storage media of claim 11, further comprising formulating an energy minimization equation to find a λ

to minimize an energy or cost E(λ

) such that

E(λ

)=ω

₁E_rep(λ

)+ω

₂E_trans(λ

)

Subject to Σ

_i=1^Mλ

_i=N

where Erep(λ

) denotes a cost from representativeness of λ

,E_trans(λ

) denotes the cost of any transition that is not visually smooth, ω

₁ and ω

₂ are two predefined weights controlling a relative strength of each energy term.

13. The computer-readable storage media of claim 11, further comprising formulating an equation for representing cost to determine how to select images representing video content, wherein the equation includes:

E_rep(λ

)=−

(α

A(λ

)+β

Q(λ

)+γ

D(λ

)),

wherein α

+β

+γ

=1,0≦

α

,β

, γ

≦

1, and A(λ

),Q(λ

) and D(λ

) measures a saliency, a quality and a distribution of the selected images, respectively.

14. The computer-readable storage media of claim 11, wherein resizing regions of interest comprises formulating an equation:

$E_{\mathrm{rep}}<br><br>\left(\mathrm{\lambda <br><br>}\right)=-\underset{i=1}{\overset{M}{\sum <br><br>}}<br><br>\left[\mathrm{\alpha <br><br>}<br><br><br><br>A<br><br>\left(I_i,R_i\right)+\mathrm{\beta <br><br>}<br><br>\left(C<br><br>\left(I_i,R_i\right)-B<br><br>\left(I_i,R_i\right)\right)\right]·<br><br>\frac{\mathrm{\varepsilon <br><br>}<br><br><br><br>A<br><br>\left(I_i,R_i\right)}{A_{\max }}-\mathrm{\gamma <br><br>}<br><br><br><br>D<br><br>\left(\mathrm{\lambda <br><br>}\right)$

where A(I_i,R_i) measures a saliency or importance of Ii;

a quality of I_i, Q(Ii,R_i), is derived from a color contrast C(I_i,R_i) and a blurring degree B(I_i, R_i);

Amax is a maximal saliency in λ

;

ε

(1≦

ε

≦

2) is a constant to control a resizing of ROI of I_i.

15. The computer-readable storage media of claim 14, wherein D(λ

) measures a temporal distribution of λ

, wherein D(λ

) can be defined as

$D<br><br>\left(\mathrm{\lambda <br><br>}\right)=-\frac{1}{\log <br><br><br><br>N}<br><br>\underset{i=1,{\mathrm{\lambda <br><br>}}_i\ne <br><br>0}{\overset{N-1}{\sum <br><br>}}<br><br>p<br><br>\left(I_i,R_i\right)·<br><br>\log <br><br><br><br>p<br><br>\left(I_i,R_i\right)$

wherein p(I_i, R_i)=(interval between I_i and I_i+1)/(a total duration of a video).

16. The computer-readable storage media of claim 11, wherein creating the video collage comprises minimizing a transition energy E_trans (λ

) by formulating an equation;

$E_{\mathrm{trans}}<br><br>\left(\mathrm{\lambda <br><br>}\right)=\underset{p,q\in <br><br>C}{\sum <br><br>}<br><br>\left(<br><br>R_{L<br><br>\left(p\right)}^{\mathrm{\prime <br><br>}}<br><br>\left(p\right)-R_{L<br><br>\left(q\right)}^{\mathrm{\prime <br><br>}}<br><br>\left(p\right)<br><br>+<br><br>R_{L<br><br>\left(p\right)}^{\mathrm{\prime <br><br>}}<br><br>\left(q\right)-R_{L<br><br>\left(q\right)}^{\mathrm{\prime <br><br>}}<br><br>\left(q\right)<br><br>\right)$

wherein R′

_L(p)(q) denotes a color of pixel q(q ∈

C) in a resized ROI R′

_L(p).

17. The computer-readable storage media of claim 11, wherein the ROI is resized according to a saliency to emphasize meaningful highlights using equation:

$\mathrm{size}<br><br>\left(R_i^{\mathrm{\prime <br><br>}}\right)=\mathrm{size}<br><br>\left(R_i\right)<br><br>\frac{\mathrm{\varepsilon <br><br>}<br><br><br><br>A<br><br>\left(R_i\right)}{A_{\max }}$

wherein size(R_i) denotes a size of an original ROI, size(R′

_i) denotes a size of a resized ROI, and Amax denotes a maximal saliency in λ

.

18. A user interface having computer-readable instructions that, when executed by a computing device, cause the computing device to perform acts comprising:

designing a video collage for video browsing;

generating the video collage in a first panel with regions of interest from representative images on a canvas without seams;

presenting access to the video collage in the first panel to play a corresponding video content in a second panel, wherein the video collage in the first panel is shown in a two dimensional static collage; and

presenting access to the video collage in the first panel to play a corresponding video clip in the first panel, wherein the video collage in the first panel is shown in a two dimensional dynamic collage.

19. The user interface of claim 18, wherein the instructions further cause presenting access to the video collage in the first panel to play a corresponding video content in a third panel, wherein the video collage in the first panel is shown in a one dimensional static collage.

20. The user interface of claim 18, wherein the instructions further cause presenting access to the video collage in the first panel to play a corresponding video clip in a third panel, wherein the video collage in the first panel is shown in a one dimensional dynamic collage.

21. The user interface of claim 18, wherein the instructions further cause generating key frames in a fourth panel by clicking on a specific key-frame to access the corresponding video content in the second panel.

22. A method for constructing a video collage, implemented at least in part by a computing device, the method comprising:

selecting images from a photo collection;

extracting and resizing the images from the photo collection; and

arranging the images on a canvas according to a timestamp.