Synthesizing a presentation of a multimedia event

• US 9,106,804 B2
• Filed: 09/26/2008
• Issued: 08/11/2015
• Est. Priority Date: 09/28/2007
• Status: Active Grant

• Current Assignee: Gracenote, Inc. (RR Donnelley & Sons Company)
• Original Assignee: Gracenote, Inc. (RR Donnelley & Sons Company)
• Inventors: Roberts, Dale T., Cook, Randall E., Cremer, Markus K.
• Primary Examiner(s): Tran, Quoc A
• Application Number:
  US12/239,082
• Publication Number:
  US 20090087161A1
• Time in Patent Office: 2,510 Days
• Field of Search: 715/203, 715723-732, 382/173
• US Class Current: 1/1
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
• CPC Class Codes: G06F 40/103 Formatting, i.e. changing of presentation of documents automatic justification G06F40/189; automatic line break hyphenation G06F40/191
• CPC Class Codes: G06F 40/106 Display of layout of documents; Previewing
• CPC Class Codes: G06F 40/14 Tree-structured documents parsing G06F40/205; validation G06F40/226
• CPC Class Codes: G06F 40/174 Form filling; Merging
• CPC Class Codes: G06F 40/177 of tables; using ruled lines
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/036 Insert-editing
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/2743 Video hosting of uploaded data from client
• CPC Class Codes: H04N 21/8549 Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
• CPC Class Codes: H04N 7/17336 Handling of requests in head-ends

Abstract

Example embodiments of a media synchronization system and method for synthesizing a presentation of a multimedia event are generally described herein. In some example embodiments, the media synchronization system includes a media ingestion module to access a plurality of media clips received from a plurality of client devices, a media analysis module to determine a temporal relation between a first media clip from the plurality of media clips and a second media clip from the plurality of media clips, and a content creation module to align the first media clip and the second media clip based on the temporal relation, and to combine the first media clip and the second media clip to generate the presentation.

Claims

1. A media synchronization system to synthesize a presentation of a multimedia event, the system comprising:

a media ingestion module configured to access a plurality of media clips received from a plurality of client devices that include a first client device from which a first media clip with a first audio waveform is received and a second client device from which a second media clip with a second audio waveform is received;

processing circuitry configured by a media analysis module to determine a temporal overlap of the first media clip received from the first client device with the second media clip received from the second client device, the temporal overlap of the first and second media clips being determined by matching a first fingerprint of at least part of the first media clip with a second fingerprint of at least part of the second media clip; and

a content creation module configured to merge the first media clip and the second media clip into a group of overlapping media clips based on the temporal overlap determined by the matching of the first fingerprint with the second fingerprint, and to generate the presentation to include the group formed by merging the first and second media clips based on the temporal overlap.

2. The system of claim 1, the media ingestion module being configured to assign an identifier to the first media clip.

3. The system of claim 1, the plurality of media clips comprising at least one of video data, audio data, image data, or text data.

4. The system of claim 1, the plurality of media clips comprising a media clip recorded by a user.

5. The system of claim 1, the media analysis module further configuring the processing circuitry to generate recognition data associated with the first media clip.

6. The system of claim 1, the media analysis module further configuring the processing circuitry to group at least some of the plurality of media clips based on fingerprint matching.

7. The system of claim 1, the first media clip comprising audio data associated with video data, the content creation module being configured to separately align the audio data and the video data.

8. The system of claim 7, the content creation module being configured to separately align the audio data and the video data based on at least one of fingerprint matching, metadata matching, or lack of matching.

9. The system of claim 1, the content creation module being configured to combine the first media clip and the second media clip based on at least one of automatic combining or user-supervised combining.

10. The system of claim 1, the content creation module being configured to generate a media effect to be combined with the first media clip.

11. The system of claim 1 further comprising a content publishing module configured to publish the presentation.

12. The system of claim 11, the content publishing module being further configured to publish the presentation to at least one of a mobile device, a stationary device, a public network, or a nonpublic network.

13. The system of claim 1, wherein:

each of the plurality of media clips is generated from an event by the plurality of client devices; and

the media analysis module configures the processing circuitry to identify the first media clip and the second media clip as being generated from the event.

14. The system of claim 1, wherein:

the first media clip and the second media clip are taken from the same event; and

the media analysis module configures the processing circuitry to compute the first fingerprint from the first media clip taken from the same event and compute the second fingerprint from the second media clip taken from the same event.

15. The system of claim 1, wherein:

the first media clip and the second media clip are separately recorded views of a single event; and

the content creation module is configured to merge the first media clip and the second media clip based on a first audio fingerprint of the first media clip and on a second audio fingerprint of the second media clip.

16. The system of claim 1, wherein:

the first media clip includes first footage filmed during a concert at a performance hall by the first client device located in a first part of the performance hall;

the second media clip includes second footage filmed during the concert at the performance hall by the second client device located in a second part of the performance hall.

17. The system of claim 1, wherein:

the first media clip includes first footage of a concert at a performance hall filmed by the first client device from a first angle within the performance hall during a first period of time during the concert;

the second media clip includes second footage of the concert at the performance hall filmed by the second client device from a second angle within the performance hall during a second period of time that overlaps the first period of time during the concert.

18. A computerized method for synthesizing a presentation of a multimedia event, the method comprising:

accessing a plurality of media clips received from a plurality of client devices that include a first client device from which a first media clip with a first audio waveform is received and a second client device from which a second media clip with a second audio waveform is received;

determining a temporal overlap of the first media clip received from the first client device with the second media clip received from the second client device, the determining being performed by processing circuitry, the temporal overlap of the first and second media clips being determined by matching a first fingerprint of at least part of the first media clip with a second fingerprint of at least part of the second media clip;

merging the first media clip and the second media clip into a group of overlapping media clips based on the temporal overlap determined by the matching of the first fingerprint with the second fingerprint; and

generating the presentation to include the group formed by the merging of the first and second media clips based on the temporal overlap.

19. The method of claim 18 further comprising assigning an identifier to the first media clip.

20. The method of claim 18, the plurality of media clips comprising at least one of video data, audio data, image data, or text data.

21. The method of claim 18, the plurality of media clips comprising a media clip recorded by a user.

22. The method of claim 18 further comprising processing the first media clip to generate recognition data associated with the first media clip.

23. The method of claim 18, the first media clip comprising audio data associated with video data, the merging comprising separately aligning the audio data and the video data.

24. The method of claim 23, the separate aligning of the audio data and the video data comprising separately aligning the audio data and the video data based on at least one of fingerprint matching, metadata matching, or lack of matching.

25. The method of claim 18, the combining comprising generating a media effect to be combined with the first media clip.

26. The method of claim 18, further comprising publishing the presentation.

27. The method of claim 26, the publishing comprising publishing to at least one of a mobile device, a stationary device, a public network, or a nonpublic network.

28. The method of claim 26, the publishing comprising displaying an advertisement.

29. The method of claim 18, wherein:

each of the plurality of media clips is generated from an event by the plurality of client devices; and

the determining of the temporal overlap of the first media clip and the second media clip includes identifying the first media clip and the second media clip as being generated from the event.

30. The method of claim 18, wherein:

the first media clip and the second media clip are taken from the same event; and

the method further comprises

computing the first fingerprint from the first media clip taken from the same event and computing the second fingerprint from the second media clip taken from the same event.

31. The method of claim 18, wherein:

the first media clip and the second media clip are separately recorded views of a single event; and

the merging of the first media clip and the second media clip is based on a first audio fingerprint of the first media clip and on a second audio fingerprint of the second media clip.

32. The method of claim 18, wherein:

the first media clip includes first footage filmed during a concert at a performance hall by the first client device located in a first part of the performance hall;

the second media clip includes second footage filmed during the concert at the performance hall by the second client device located in a second part of the performance hall.

33. The method of claim 18, wherein:

the first media clip includes first footage of a concert at a performance hall filmed by the first client device from a first angle within the performance hall during a first period of time during the concert;

the second media clip includes second footage of the concert at the performance hall filmed by the second client device from a second angle within the performance hall during a second period of time that overlaps the first period of time during the concert.

34. A non-transitory machine-readable medium comprising instructions, which when implemented by one or more processors, cause the one or more processors to perform operations comprising:

accessing a plurality of media clips received from a plurality of client devices that include a first client device from which a first media clip with a first audio waveform is received and a second client device from which a second media clip with a second audio waveform is received;

determining a temporal overlap of the first media clip received from the first client device with the second media clip received from the second client device, the determining being performed by the one or more processors, the temporal overlap of the first and second media clips being determined by matching a first fingerprint of at least part of the first media clip with a second fingerprint of at least part of the second media clip;

merging the first media clip and the second media clip into a group of overlapping media clips based on the temporal overlap determined by the matching of the first fingerprint with the second fingerprint; and

generating the presentation to include the group formed by the merging of the first and second media clips based on the temporal overlap.

35. The non-transitory machine-readable medium of claim 34, wherein:

each of the plurality of media clips is generated from an event by the plurality of client devices; and

the determining of the temporal overlap of the first media clip and the second media clip includes identifying the first media clip and the second media clip as being generated from the event.

36. The non-transitory machine-readable medium of claim 34, wherein:

the first media clip and the second media clip are taken from the same event; and

the operations further comprise

computing the first fingerprint from the first media clip taken from the same event and computing the second fingerprint from the second media clip taken from the same event.

37. The non-transitory machine-readable medium of claim 34, wherein:

the first media clip and the second media clip are separately recorded views of a single event; and

the merging of the first media clip and the second media clip is based on a first audio fingerprint of the first media clip and on a second audio fingerprint of the second media clip.

38. The non-transitory machine-readable medium of claim 34, wherein:

the first media clip includes first footage filmed during a concert at a performance hall by the first client device located in a first part of the performance hall;

the second media clip includes second footage filmed during the concert at the performance hall by the second client device located in a second part of the performance hall.

39. The non-transitory machine-readable medium of claim 34, wherein:

the first media clip includes first footage of a concert at a performance hall filmed by the first client device from a first angle within the performance hall during a first period of time during the concert;

the second media clip includes second footage of the concert at the performance hall filmed by the second client device from a second angle within the performance hall during a second period of time that overlaps the first period of time during the concert.