Creating a multimedia presentation from full motion video using significance measures

• US 6,892,351 B2
• Filed: 01/17/2002
• Issued: 05/10/2005
• Est. Priority Date: 12/17/1998
• Status: Expired due to Fees

• Current Assignee: Wirefree Services Belgium S.A.
• Original Assignee: Newstakes, Inc.
• Inventors: Vasudevan, Vinod V., Singh, Gurminder, Jesudoss, Manoranjan D.
• Primary Examiner(s): Luu, Sy D.
• Application Number:
  US10/051,631
• Publication Number:
  US 20020140721A1
• Time in Patent Office: 1,209 Days
• Field of Search: 715/716, 715/723, 715730-732, 715/500.1, 358/448, 358/467, 345/716, 345/723, 345730-732
• US Class Current: 715/716
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence

Abstract

Methods apparatus and computer programs products for creating a multimedia presentation having video frames and video segments with synchronized audio are disclosed. Audio-video data is separated into an audio stream and the video sequence. The video sequence is divided into video segments, with each video segment comprising a group of frames. For each video segment an audio significance measure is calculated using the audio stream. Using the audio significance measure, the video segment is selected in its entirety or, alternatively, one or more slide frames are extracted from the corresponding group of frames. The selection of video segments or slide frames can also be performed on the basis of an additional video significance measure. The audio stream and selected video segment and slide frames are synchronized, then synchronously reproduced as the multimedia presentation.

Claims

1. A method for creating a multimedia presentation having video frames and video segments with synchronized audio, said method comprising the steps of:

(g) receiving audio-video data;

(h) separating said audio-video data into an audio stream and a video sequence;

(i) dividing said video sequence into video segments, each of said video segments comprising a group of slide frames;

(j) for each of said video segments;

(d1) calculating an audio significance measure using said audio stream related to each of said video segments;

(d2) using at least said audio significance measure, selecting either of each said video segments in its entirety or extracting at least one slide frame from said corresponding group of frames;

(k) synchronizing said audio stream and selected video segments and said at least one slide frame; and

(l) synchronously reproducing selected video segments and said at least one slide frame and said audio stream as said multimedia presentation.

2. The method according to claim 1, wherein, in step (d1), said audio significance measure is calculated from a measure of activity within said audio stream.

3. The method according to claim 2, wherein said measure of activity is determined from a frequency domain representation of said audio stream.

4. The method according to claim 2, wherein said measure of activity is calculated from one or more of power, peak-frequency and frequency spread.

5. The method according to claim 4, wherein said measure of activity is calculated as power+peak frequency+spread.

6. The method according to claim 4, wherein said measure of activity is calculated as 2×

power+peak frequency+0.5×

spread.

7. The method of claim 1, wherein, in step (d2), said selection is based upon a comparison of said audio significance measure with an activity threshold, such that if said threshold is exceeded, then said video segment is selected, else if not exceeded, said at least one slide frame is selected.

8. A method for creating a multimedia presentation having video frames and video segments with synchronized audio, said method comprising the steps of:

(g) receiving audio-video data;

(h) separating said audio-video data into an audio stream and a video sequence;

(i) dividing said video sequence into video segments, each of said video segments comprising a group of slide frames;

(j) for each of said video segments;

(d1) extracting at least one representative frame from the corresponding said group of frames;

(d2) calculating a video significance measure using the extracted at least one frame;

(d3) calculating an audio significance measure using the audio stream related to said each of said video segment;

(d4) using said video and audio significance measures, selecting either each of the video segments in its entirely or extracting at least one slide frame from said group of frames;

(k) synchronizing said audio stream and the selected video segments and the extracted said at least one slide frame; and

(l) synchronously reproducing the selected video segment and the extracted slide frame and said audio stream.

9. The method according to claim 8, wherein, in step (d1), said audio significance measure is calculated from a measure of activity within said audio stream.

10. The method according to claim 9, wherein said measure of activity is determined from a frequency domain representation of said audio stream.

11. The method according to claim 8, wherein said measure of activity is calculated from one or more of power, peak-frequency and frequency spread.

12. The method according to claim 11, wherein said measure of activity is calculated as power+peak frequency+spread.

13. The method according to claim 11, wherein said measure of activity is calculated as 2×

power+peak frequency+0.5×

spread.

14. The method according to claim 8, wherein said video significance measure is determined from a level of relative movement between said frames.

15. The method according to claim 14, wherein said frames comprise objects and said level of relative movement is determined from a direction and magnitude of motion (u_i, v_i) of each object in said frames to derive an activity value of the frame.

16. The method according to claim 15, wherein said activity value is determined from the standard deviation of the direction and magnitude (u_i, v_i) of each object.

17. The method of claim 16, wherein said activity value is determined according to the expression:

$\mathrm{Activity}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{value}==\underset{i=1}{\overset{i=k}{\sum <br><br>}}<br><br>\sqrt{{\left[u_i-u\right]}^2+{\left[v_i-v\right]}^2}$

wherein (u,v) is the average direction and magnitude of motion of all objects in said frame.

18. The method of claim 8, wherein, in step (d4), said selection is based upon a comparison of said audio significance measure with an activity threshold, such that if said threshold is exceeded, then said video segment is selected, else if not exceeded, said at least one slide frame is selected.

19. The method of claim 18, wherein said combined significance measure is equal to one of the audio significance measure, the video significance measure, or an average of said audio and video significance measures depending upon the level of audio and video activity.

20. Apparatus for creating a multimedia presentation having video frames and video segments with synchronized audio, said apparatus comprising:

input means for receiving audio-video data;

means for separating said audio-video data into an audio stream and a video sequence;

means for dividing said video sequence into video segments, each of said video segments comprising a group of slide frames;

processor means which, for each said video segments, calculates an audio significance measure using the audio stream related to each of the video segments, and, using at least said audio significance measure, selects either each of the video segments in its entity or extracts at least one slide frame from said corresponding group of frames;

means for synchronizing said audio stream and selected video segments and extracted slide frames; and

means for synchronously reproducing selected video segments and extracted slide frames and said audio stream as said multimedia presentation.

21. Apparatus for creating a multimedia presentation having video frames and video segments with synchronized audio, comprising:

means for receiving audio-video data;

means for separating said audio-video data into an audio stream and a video sequence;

means for dividing said video sequence into video segments, each of said video segments comprising a group of slide frames;

processor means which, for each of said video segments, extracts at least one representative frame from the corresponding group of frames, calculates a video significance measure using the extracted at least one frame, calculates an audio significance measure using the audio stream related to said each of said video segments, and, using said video and audio significance measures, selects either said each of said video segments in its entirely or extracts at least one slide frame from the group of frames;

means for synchronizing said audio stream and selected video segments and extracted slide frames; and

means for synchronously reproducing the selected video segments and the extracted slide frames and said audio stream.

22. A computer program product including a computer readable medium incorporating a computer program for creating a multimedia presentation having video frames and video segments with synchronized audio, said computer program having:

input code means for receiving audio-video data;

code means for separating said audio-video data into an audio stream and a video sequence;

code means for dividing said video sequence into video segments, each of said video segments comprising a group of slide frames;

processing code means which, for each of said video segments, calculates an audio significance measure using the audio stream related to said each of said video segments, and, using at least said audio significance measure, selects either said each of said video segments in its entity or extracts at least one slide frame from said corresponding group of frames;

code means for synchronizing said audio stream and selected video segments and extracted slide frames; and

code means for synchronously reproducing selected video segments and extracted slide frames and said audio stream as said multimedia presentation.

23. A computer program product including a computer readable medium incorporating a computer program for creating a multimedia presentation having video frames and video segments with synchronized audio, said computer program having:

code means for receiving audio-video data;

code means for separating said audio-video data into an audio stream and a video sequence;

code means for dividing said video sequence into video segments, each of said video segments comprising a group of slide frames;

processing code means which, for said each of said video segments, extracts at least one representative frame from the corresponding said group of frames, calculates a video significance measure using the at least one extracted frame, calculates an audio significance measure using the audio stream related to said each of said video segments, and, using said video and audio significance measures, selects either said each of said video segments in its entirety or extracts at least one slide frame from said group of frames;

code means for synchronizing said audio stream and selected video segments and extracted slide frames; and

code means for synchronously reproducing the selected video segments and extracted slide frames and said audio stream.