Creating a slide presentation from full motion video

• US 6,342,904 B1
• Filed: 12/17/1998
• Issued: 01/29/2002
• Est. Priority Date: 12/17/1998
• Status: Expired due to Fees

• Current Assignee: Wirefree Services Belgium S.A.
• Original Assignee: Newstakes, Inc.
• Inventors: Singh, Gurminder, Vasudevan, Vinod V., Jesudoss, Manoranjan D.
• Primary Examiner(s): Bayerl, Raymond J.
• Assistant Examiner(s): Luu, Sy D.
• Application Number:
  US09/215,004
• Time in Patent Office: 1,139 Days
• Field of Search: 345/328, 345/327, 345/302, 345/723, 345/716, 345/730, 358/448, 358/467, 707/500.1
• US Class Current: 715/723
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence

Abstract

A method and apparatus is disclosed for converting audio-video data from a full motion video format to a slide show presentation with synchronised sound. The full motion video is received from a source, separated into an audio stream and a video sequence, the video sequence is divided into a number of shorter video segments, key frames are extracted for each segment and a significance measure is calculated for each frame. A database is created wherein the extracted data is stored for subsequent (off-line) processing and reproduction. The system may have more than one system retrieving data from the database, selecting slide frames and subsequently displaying a slide presentation.

Claims

1. A method for converting full motion video into a slide show with synchronized audio, said method comprising the steps of:

(a) receiving audio-video data;

(b) separating said audio-video data into an audio stream and a video sequence;

(c) dividing said video sequence into video segments, each of said video segments comprising a group of frames;

(d) for each of said video segments

(d1) extracting at least one representative frame from the corresponding said group of frames;

(d2) calculating a significance measure using said at least one representative frame;

(d3) using said significance measure, selecting at least one slide frame from said group of frames;

(e) synchronizing said audio stream and said at least one slide frame; and

(f) synchronously reproducing said at least one slide frame and said audio stream.

2. A method according to claim 1 wherein stop (c) comprises dividing said video sequence into video segments by detecting scene changes in said video sequence.

3. A method according to claim 1 wherein step (d2) comprises calculating a (first) significance measure for each frame.

4. A method according to claim 3 wherein step (d2) further comprises calculating a second significance measure for said video segment using said first significance measures.

5. A method according to claim 4 wherein said second significance measure is used to determine a number of said frames to be selected as slide frames for said video segment.

6. A method according to claim 5 wherein said number of frames is determined by applying one of a logistic function or a mono-tonically increasing function to said second significance measures.

7. A method according to claim 4 wherein step (d2) further comprises calculating a third significance measure for said representative frames using said first significance measures.

8. A method according to claim 7 wherein step (d3) selects said slide frames using said third significance measures.

9. A method according to claim 7 wherein said third significance measure of each of said representative frames are a function of said first significance measure of said frames surrounding said representative frame.

10. A method according to claim 9 further comprising the step, after step (e1) and before step (e), of:

(e2) retrieving said audio stream and said slide frames from said storage means.

11. A method according to claim 7 wherein said third significance measures of each of said representative frames are said first significance measures of sad representative frames.

12. A method according to claim 3 wherein step (d3) selects at least one said slide frame using said first significance measures.

13. A method according to claim 3 wherein step (d3) selects a predetered number of said slide frames using said fire significance measures.

14. A method according to claim 3 wherein said first significance measures are determined from a level of relative movement between said frames.

15. A method according to claim 14 wherein said frames comprise objects and said level of relative movement is determined from a direction and magnitude of motion (ui, vi) of each object in said frames to derive an activity value of the frame.

16. A method according to claim 15 wherein said activity value is determined from the standard deviation of the direction and magnitude of motion (ui, vi) of each object.

17. A method according to claim 15 wherein said activity value is determined according to the expression:

$\mathrm{Activity}=\underset{i=1}{\overset{k}{\sum <br><br>}}<br><br>\sqrt{{\left[u_i-u\right]}^2+{\left[v_i-u\right]}^2}$

wherein (u,v) is the average direction and magnitude of motion of all objects in said frame.

18. A method according to claim 15 wherein said first significance measures of each of said frames are the activity values of said frames.

19. A method according to claim 15 wherein said second significance measure for each said segment is determined by averaging the activity values of said frames.

20. A method according to claim 3 wherein step (d3) is performed by selecting the slide frames such that said slide frames are substantially equally spaced in time.

21. A method according to claim 1, further comprising the step, before step (e) of:

(e1) storing said audio stream and said slide frames on a storage means for later retrieval.

22. A method according to claim 1, wherein step (e) is performed by determining a frame number of the slide frame to be shown from a time identifier retrieved from said audio stream.

23. A method according to claim 1, further comprising the step, after step (e) and before step (f), of:

(f1) receiving an user request to change a criteria for selecting said slide frames and in response thereto, using said significance measure, select for each said video segment at least one substitute slide frame from said group of frames; and

(f2) synchronising said audio stream and said slide frames.

24. A method for creating a database for use in converting full motion video into a slide show with synchronized audio, said method comprising the steps of:

(a) receiving audio-video data;

(b) separating said audio-video data into an audio stream and a video sequence;

(c) dividing said video sequence into video segments, each of said video segments comprising a group of frames;

(d) for each of said video segments

(d1) extracting at least one representative frame from the corresponding said group of frames;

(d2) calculating a significance measure using said at least one representative frame;

(d3) selecting at least one slide frame from said group of frames using said significance measures; and

(e) storing at least an identifier to each of said segments, said significance measure and an identifier to said audio stream.

25. A method according to claim 24, the method comprises the further step of storing an identifier to said extracted representative frames.

26. A method for displaying a slide show with synchronized audio, said method comprising the steps of:

(a) retrieving at least an identifier to each of a plurality of video segments, significance measures and an identifier to an audio stream, wherein each of said video segments comprising a group of frames;

(b) for each of said video segments

(b1) extracting at least one representative frame from the corresponding said group of frames;

(b2) calculating a significance measure using said at least one representative frame;

(b3) using said significance measure, selecting at least one slide frame from said group of frames;

(c) synchronizing said audio stream and said slide frame; and

(d) synchronously reproducing said slide frame and said audio stream.

27. Apparatus for creating a database for use in converting full motion video into a slide show with synchronized audio, said apparatus comprising:

input means for receiving audio-video data;

audio extraction means for separating said audio-video data into an audio stream and a video sequence;

video dividing means for dividing said video sequence into video segments, each of said video segments comprising a group of frames;

key frame extraction means for extracting, for each of said video segments at least one representative frame from the corresponding said group of frames of each of said video segments;

significance calculation means for calculating significance measures using said at least one representative frame;

selection means for selecting at least one slide frame from said group of frames using said significance measures; and

database storage means for storing said database, the database including at least an identifier to each of said segments, said significance measures and an identifier to said audio stream.

28. Apparatus for displaying a slide show with synchronized audio, said apparatus comprising:

database retrieval means for retrieving a database, the database including at least an identifier to each of a plurality of video segments, significance measures and an identifier to an audio stream, wherein each of said video segments comprising a group of frames;

extracting means for extracting at least one representative frame from the corresponding said group of frames;

calculating means for calculating a significance measure using said at least one representative frame;

selection means for selecting, for each of said video segments, at least one slide frame from said group of frames using said significance measure; and

synchronization means for synchronising said audio stream and said at least one slide frame; and

output means for synchronously reproducing said at least one slide frame and said audio stream.

31. Apparatus according to claim 28, wherein said database is retrieved from a network.

29. Apparatus according to clam 28, said apparatus further comprising:

presentation storage means for storing said audio stream and said slide frames for later retrieval.

30. Apparatus for presenting a slide show with synchronized audio, said apparatus comprising:

presentation retrieval means for retrieving from a storage means at least an identifier to each of a plurality of video segments, significance measures and an identifier to an audio stream, wherein each of said video segments comprising a group of frames;

extracting means for extracting at least one representative frame from the corresponding said group of frames;

calculating means for calculating a significance measure using said at least one representative frame;

selection means for selecting, for each of said video segments, at least one slide frame from said group of frames using said significance measure; and

synchronization means for synchronising said audio stream and said at least one slide frame; and

output means for synchronously reproducing said at least one slide frame and said audio stream.

32. A computer program product including a computer readable medium incorporating a computer program for creating a database for use in converting full motion video into a slide show with synchronized audio, said computer program product comprising:

input means for receiving audio-video data;

audio extraction means for separating said audio-video data into an audio stream and a video sequence;

video dividing means for dividing said video sequence into video segments, each of said video segments comprising a group of frames;

key frame extraction means for extracting, for each of said video segments, at least one representative frame from the corresponding said group of frames of each of said video segments;

significance calculation means for calculating significance measures using said at least one representative frame;

selection means for selecting at least one slide frame from said group of frames using said significance measures; and

database storage means for storing said database, the database including at least an identifier to each of said segments, said significance measures and an identifier to said audio stream.

33. A computer program product including a computer readable medium incorporating a computer program for creating a database for displaying a slide show with synchronized audio, said computer program product comprising:

database retrieval means for retrieving a database, the database including at least an identifier to each of a plurality of video segments, significance measures and an identifier to an audio stream, wherein each of said video segments comprising a group of frames;

extracting means for extracting at least one representative frame from the corresponding said group of frames;

calculating means for calculating a significance measure using said at least one representative frame;

selection means for selecting, for each of said video segments, at least one slide frame from said group of frames using said significance measurer; and

synchronization means for synchronising said audio stream and said at least one slide frame; and

output means for synchronously reproducing said at least one slide frame and said audio stream.

34. A computer program product including a computer readable medium incorporating a computer program for creating a database for converting full motion video into a slide show with synchronized audio, said computer program product comprising:

means for receiving audio-video data;

means for separating said audio-video data into an audio stream and a video sequence;

means for dividing said video sequence into video segments, each of said video segments comprising a group of frames;

means for processing each said video segments by

(1) extracting at least one representative frame from the corresponding said group of frames;

(2) calculating a significance measure using said at least one representative frame;

(3) using said significance measure, selecting at least one slide frame from said group of frames;

means for synchronising said audio stream and said at least one slide frame; and

means for synchronously reproducing said at least one slide frame and said audio stream.

35. A computer program product including a computer readable medium incorporating a computer program for creating a database for use in converting full motion video into a slide show with synchronized audio, said computer program product comprising:

means for receiving audio-video data;

means for separating said audio-video data into an audio stream and a video sequence;

means for dividing said video sequence into video segments, each of said video segments comprising a group of frames;

means for processing each of said video segments by

(1) extracting at least one representative frame from the corresponding said group of frames;

(2) calculating a significance measure using said at least one representative frame;

(3) selecting at least one slide frame from said group of frames using said significance measure; and

means for storing at least an identifier to each of said segments, said significance measure and an identifier to said audio stream.

36. A computer program product including a computer readable medium incorporating a computer program for displaying a slide show with synchronized audio, said computer program product comprising:

means for retrieving at least an identifier to each of a plurality of video segments, significance measures and an identifier to an audio stream, wherein each of said video segments comprising a group of frames;

means for processing each of said video segments by

(1) extracting at least one representative frame from the corresponding said group of frames;

(2) calculating a significance measure using said at least one representative frame;

(3)using said significance measurer, selecting at least one slide frame from said group of frames;

means for synchronising said audio stream and said at least one slide frame; and

means for synchronously reproducing said at least one slide frame and said audio stream.

37. A method for converting full motion video into a slide show with synchronized audio, said method comprising the steps of:

(a) receiving audio-video data;

(b) separating said audio-video data into an audio stream and a video sequence;

(c) dividing said video sequence into video segments, each of said video segments comprising a group of frames;

(d) for each of said video segments

(d1) extracting at least one representative frame from the corresponding said group of frames;

(d2) calculating a first significance measure using said at least one representative frame, and calculating a second significance measure for said video segment using said first significance measures;

(d3) using said significance measure, selecting at least one slide frame from said group of frames;

(e) synchronizing said audio stream and said at least one slide frame;

(f) synchronously reproducing said at least one slide frame and said audio stream;

(g) wherein said second significance measure is used to determine a number of said at least one slide frame to be selected as slide frames for said video segment.

38. A method according to claim 37 wherein step (d2) further comprises calculating a third significance measure for said representative frames using said first significance measures.

39. A method according to claim 37 wherein said number of frames is determined by applying one of a logistic function or a mono-tonically increasing function to said second significance measures.

40. A computer program product including a computer readable medium incorporating a computer program for creating a database for use in converting full motion video into a slide show with synchronized audio, said computer program product comprising:

input means for receiving audio-video data;

audio extraction means for separating said audio-video data into an audio stream and a video sequence;

video dividing means for dividing said video sequence into video segments, each of said video segments comprising a group of frames;

key frame extraction means for extracting, for each of said video segments, at least one representative frame from the corresponding group of frames of each of said video segments;

significance calculation means for calculating a first significance measure using said at least one representative frame, and for calculating a second significance measure for said video segment using said first significance measure, and using said significance measures to select at least one slide frame from said group of frames and

database storage means for storing said database, the database including at least an identifier to each of said segments, said significance measures and an identifier to said audio stream.

41. A computer program product including a computer readable medium incorporating a computer program for converting full motion video into a slide show with synchronized audio, said computer program product comprising:

means for receiving audio-video data;

means for separating said audio-video data into an audio stream and a video sequence;

means for dividing said video sequence into video segments, each of said video segments comprising a group of frames;

means for processing each of said video segments by

(1) extracting at least one representative frame from the corresponding said group of frames;

(2) calculating a first significance measure using said at least one representative frame, and calculating a second significance measure for said video segment using said first significance measure;

(3) using said significance measures, selecting at least one slide frame from said group of frames;

means for synchronising said audio stream and said at least one slide frame;

means for synchronously reproducing said at least one slide frame and said audio stream; and

wherein said second significance measure is used to determine a number of said at least one slide frames to be selected as slide frames for said video segment.