MULTI-VIDEO SYNTHESIS

• US 20100149419A1
• Filed: 12/12/2008
• Published: 06/17/2010
• Est. Priority Date: 12/12/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Li, Teng, Mei, Tao, Hua, Xian-Sheng, Li, Shipeng
• Granted Patent: US 8,207,989 B2
• Time in Patent Office: Days
• US Class Current: 348/584
• CPC Class Codes: G11B 27/036 Insert-editing

Abstract

Embodiments that provide multi-video synthesis are disclosed. In accordance with one embodiment, multi-video synthesis includes breaking a main video into a plurality of main frames and break a supplementary video into a plurality of supplementary frames. The multi-video synthesis also includes assigning one or more supplementary frames into each of a plurality of states of a Hidden Markov Model (HMM), where each of the plurality of states corresponding to one or more main frames. The multi-video synthesis further includes determining optimal frames in the plurality of main frames for insertion of the plurality of supplementary frames based on the plurality of states and visual properties. The optimal frames include optimal insertion positions. The multi-video synthesis additionally includes inserting the plurality of supplementary frames into the optimal insertion positions to form a synthesized video.

Claims

1. A system, comprising:

a division component to break a main video into a plurality of main frames, and to break a supplementary video into a plurality of supplementary frames;

a modeling component to assign one or more supplementary frames into each of a plurality of states of a Hidden Markov Model (HMM), and correspond each of the plurality of states to one or more of the plurality of main frames;

a detection component to determine optimal frames that include optimal insertion positions in the plurality of main frames for insertion of the plurality of supplementary frames based on the plurality of states and visual properties;

an insertion component to insert the plurality of supplementary frames into the optimal insertion positions to form a synthesized video; and

a blend component to blend the inserted supplementary frames and the optimal frames of the synthesized video.

2. The system of claim 1, further comprising:

an informativeness measurement component to compute expected losses of information from the plurality of the main frames due to insertion of the plurality of the supplementary frames,

wherein the visual properties include the expected losses computed by the informativeness measurement component.

3. The system of claim 1, further comprising:

a consistency measurement component to compute expected visual consistencies between the plurality of the main frames and the plurality of the supplementary frames,

wherein the visual properties include the expected visual consistencies computed by the consistency measurement component.

4. The system of claim 1, further comprising:

a visual naturalness component to compute expected visual naturalness between a first boundary areas in the plurality of the main frame and a second boundary areas in the plurality of supplementary frames,

wherein the visual properties include the expected visual naturalness computed by the visual naturalness component.

5. The system of claim 1, wherein the detection component is to determine the optimal insertion positions included in the optimal frames via a Viterbi beam search.

6. The system of claim 1, wherein the detection component is to determine the optimal frames via a beam search for an optimal insertion path based on the states and the visual properties, wherein the optimal insertion path includes spatially stable optimal insertion positions that correspond to the optimal frames

7. The system of claim 1, wherein the blend component is to blend the inserted supplementary frames and the optimal frames using probabilistic matting.

8. The system of claim 1, wherein the detection component is to determine the optimal frames by formulating insertion of the plurality of the supplementary frames into the optimal frames as a Maximum a Posterior (MAP) problem.

9. The system of claim 1, wherein the detection component is to determine the optimal frames by treating each of the main frames and each of the supplementary frames as including two spatial coordinates and a time coordinate.

10. A computer readable medium storing computer-executable instructions that, when executed, cause one or more processors to perform acts comprising:

breaking a main video into a plurality of main frames and break a supplementary video into a plurality of supplementary frames;

assigning one or more supplementary frames into each of a plurality of states of a Hidden Markov Model (HMM), each of the plurality of states corresponding to one or more main frames;

determining optimal frames that include optimal insertion positions in the plurality of main frames for insertion of the plurality of supplementary frames by performing a beam search for an optimal insertion path based on the plurality of states and visual properties; and

inserting the plurality of supplementary frames into the optimal insertion positions to form a synthesized video.

11. The computer readable medium of claim 10, wherein the optimal insertion path includes spatially stable optimal insertion positions that correspond to the optimal frames.

12. The computer readable medium of claim 10, wherein the computer-executable instructions, when executed, further cause the one or more processors to perform acts comprising computing expected losses of information from the plurality of the main frames due to insertion of the plurality of the supplementary frames, wherein the visual properties include the expected losses.

13. The computer readable medium of claim 10, wherein the computer-executable instructions, when executed. Further cause the one or more processors to perform acts comprising computing expected visual consistencies between the plurality of the main frames and the plurality of the supplementary frames, wherein the visual properties include the expected visual consistencies.

14. The computer readable medium of claim 10, wherein the computer-executable instructions, when executed, further cause the one or more processors to perform acts comprising computing expected visual naturalness between the first boundary areas in the plurality of the main frame and the second boundary areas in the plurality of supplementary frames, wherein the visual properties include the expected visual naturalness.

15. The computer readable medium of claim 10, wherein the computer-executable instructions, when executed, further cause the one or more processors to perform acts comprising blending the inserted supplementary frames and the optimal frames of the synthesized video using probabilistic matting.

16. A method, comprising:

breaking a main video into a plurality of main frames and break a supplementary video into a plurality of supplementary frames;

assigning one or more supplementary frames into each of a plurality of states of a Hidden Markov Model (HMM), each of the plurality of states corresponding to one or more main frames;

determining optimal frames that include optimal insertion positions in the plurality of main frames for insertion of the plurality of supplementary frames based on the plurality of states and visual properties, the visual properties including informativeness, consistency, and naturalness;

inserting the plurality of supplementary frames into the optimal insertion positions to form a synthesized video; and

blending the inserted supplementary frames and the optimal frames of the synthesized video.

17. The method of claim 16, further comprising obtaining the informativeness visual property by computing expected losses of information from the plurality of the main frames due to insertion of the plurality of the supplementary frames.

18. The method of claim 16, further comprising obtaining the consistency visual property by computing expected visual consistencies between the plurality of the main frames and the plurality of the supplementary frames.

19. The method of claim 16, further comprising obtaining the visual naturalness visual property by computing expected visual naturalness between the first boundary areas in the plurality of the main frame and the second boundary areas in the plurality of supplementary frames.

20. The method of claim 16, wherein the determining includes determining the optimal frames via a Viterbi beam search for an optimal insertion path based on the states and the visual properties, wherein the optimal insertion path includes spatially stable optimal insertion positions that correspond to the optimal frames.