Automatic generation of multimedia presentation

• US 20040054542A1
• Filed: 09/13/2002
• Published: 03/18/2004
• Est. Priority Date: 09/13/2002
• Status: Active Grant

• Current Assignee: Fuji Xerox Company Limited (Fujifilm Holdings Corporation)
• Original Assignee: Fuji Xerox Company Limited (Fujifilm Holdings Corporation)
• Inventors: Schilit, William N., Cooper, Matthew L., Foote, Jonathan T., Hilbert, David Michael
• Granted Patent: US 7,383,509 B2
• Time in Patent Office: Days
• US Class Current: 704/500
• CPC Class Codes: G09B 5/00 Electrically-operated educational appliances working with questions and answers G09B7/00; simulators G09B9/00; advertising or displaying in general G09F
• CPC Class Codes: G09B 7/00 Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads

Abstract

The present invention provides a system and method for automatically combining image and audio data to create a multimedia presentation. In one embodiment, audio and image data are received by the system. The audio data includes a list of events that correspond to points of interest in an audio file. The audio data may also include an audio file or audio stream. The received images are then matched to the audio file or stream using the time. In one embodiment, the events represent times within the audio file or stream at which there is a certain feature or characteristic in the audio file. The audio events list may be processed to remove, sort or predict or otherwise generate audio events. Images processing may also occur, and may include image analysis to determine image matching to the event list, deleting images, and processing images to incorporate effects. Image effects may include cropping, panning, zooming and other visual effects.

Claims

1. A method for automatically generating a multimedia presentation, comprising:

receiving a list of events, the events containing time information regarding events occurring in audio data;

receiving images;

automatically generating a multimedia presentation that includes a presentation of the images, the presentation of the images automatically synchronized with the audio data using the events list.

2. The method of claim 1 further comprising processing the images, wherein processing the images includes automatically determining the number of images to use in the multimedia presentation.

3. The method of claim 2 wherein automatically determining the number of images to use includes deleting events when there are more events than images.

4. The method of claim 3 wherein deleting events includes:

sorting the events according to a numerical value associated with each event; and

deleting events according to the numerical value associated with each event.

5. The method of claim 4 wherein the numerical value is an importance score.

6. The method of claim 2 wherein automatically determining the number of images to use includes deleting images if there are more images than events.

7. The method of claim 6 wherein deleting images includes:

detecting a cluster of images within the received images, the cluster including a key frame; and

deleting an image within the detected cluster of images, the deleted image most similar to the cluster keyframe.

8. The method of claim 6 wherein deleting images includes:

automatically determining matching image features to match to events in the events list;

analyzing images to detect features; and

deleting images based on the matching image features and detected features.

9. The method of claim 1 further comprising processing the images, wherein processing the images includes automatically determining the order of images to use in the multimedia presentation.

10. The method of claim 9 wherein each of the images has at least one image feature, each image feature associated with a feature type, each event within the events list associated with an event type, wherein automatically determining the order includes:

(a) matching a feature type to an event type;

(b) automatically analyzing the images to detect image features for each image; and

(c) automatically determining the order of the images by matching an image to an event in the event list, wherein a feature of the matched image is associated with the matched feature type from step (a) and the matched event is associated with the matched event type from step (a).

11. The method of claim 10 wherein image feature types include brightness, color scheme, and association with a cluster.

12. The method of claim 10 wherein event types include amplitude peaks, loudness, bass, treble, crescendos, and diminuendos.

13. The method of claim 1 further comprising processing the images, wherein processing the images includes automatically determining an effect to incorporate into an image displayed in the multimedia presentation.

14. The method of claim 13 wherein determining an effect includes configuring the effect to begin upon the occurrence of an event in the event list.

15. The method of claim 13 wherein determining an effect includes configuring the effect to end upon the occurrence of an event in the event list.

16. The method of claim 13 wherein an effect includes panning a viewing window within an image.

17. The method of claim 16 wherein the image is received in a first format having a first aspect ratio, the image displayed in the multimedia presentation in a second format having a second aspect ratio, the viewing window configured to be in the second aspect ratio.

18. The method of claim 16 wherein panning includes panning from an edge to a hot spot.

19. The method of claim 18 wherein panning includes:

detecting a hot spot within the image;

determining an edge of the image furthest from the hot spot;

panning the viewing window from the edge furthest from the hot spot to an edge closest to the hot spot.

20. The method of claim 16 wherein panning a viewing window includes:

detecting the angular distribution of spatial frequency from a location within the image; and

panning the viewing window from the location along a direction having the highest spatial frequency.

21. The method of claim 13 wherein the effect includes zooming.

22. The method of claim 20 wherein zooming includes:

detecting a hot spot within the image; and

performing a zoom operation on the hot spot.

23. The method of claim 20 wherein zooming includes:

detecting symmetry within the image, the symmetry having a center; and

performing a zoom operation on the center.

24. A computer program product for execution by a server computer for automatically generating a multimedia presentation, comprising:

computer code for receiving a list of events, the events containing time information regarding events occurring in audio data;

computer code for receiving images; and

computer code for automatically generating a multimedia presentation that includes a presentation of the images, the presentation of the images automatically synchronized with the audio data using the events list.

25. The computer program product of claim 24 farther including computer code for automatically determining the number of images to use, wherein computer code for automatically determining the number of images to use includes:

computer code for sorting the events according to a importance score associated with each event; and

computer code for deleting events according to the importance score associated with each event.

26. The computer program product of claim 24 farther including computer code for deleting images, wherein computer code for deleting images includes:

computer code for detecting a cluster of images within the received images, the cluster including a key frame; and

computer code for deleting an image within the detected cluster of images, the deleted image most similar to the cluster keyframe.

27. The computer program product of claim 24 further including computer code for deleting images, wherein computer code for deleting images includes:

computer code for automatically determining matching image features to match to events in the events list;

computer code for analyzing images to detect features; and

computer code for deleting images based on the matching image features and detected features.

28. The computer program product of claim 24 further comprising computer code for processing the images, wherein computer code for processing the images includes automatically determining the order of images to use in the multimedia presentation.

29. The computer program product of claim 28 wherein each of the images has at least one image feature, each image feature associated with a feature type, each event within the events list associated with an event type, wherein computer code for automatically determining the order includes:

(a) computer code for matching a feature type to an event type;

(b) computer code for automatically analyzing the images to detect image features for each image; and

(c) computer code for automatically determining the order of the images by matching an image to an event in the event list, wherein a feature of the matched image is associated with the matched feature type from step (a) and the matched event is associated with the matched event type from step (a).

30. The computer program product of claim 24 further comprising computer code for processing the images, wherein computer code for processing the images includes automatically determining an effect to incorporate into an image displayed in the multimedia presentation.

31. The computer program product of claim 30 wherein an effect includes panning a viewing window within an image, the image received in a first format having a first aspect ratio, the image displayed in the multimedia presentation in a second format having a second aspect ratio, the viewing window configured to be in the second aspect ratio.

32. The computer program product of claim 31 wherein computer code for panning includes panning from an edge to a hot spot.

33. The computer program product of claim 31 wherein computer code for panning a viewing window includes:

computer code for detecting the angular distribution of spatial frequency from a location within the image; and

computer code for panning the viewing window from the location along a direction having the highest spatial frequency.

34. The computer program product of claim 30 wherein the effect includes zooming.