Systems and methods for intelligently synchronizing events in visual content with musical features in audio content

• US 10,560,657 B2
• Filed: 05/03/2019
• Issued: 02/11/2020
• Est. Priority Date: 11/07/2016
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Noel, Jean-Baptiste
• Primary Examiner(s): Harvey, David E
• Application Number:
  US16/402,601
• Publication Number:
  US 20190289254A1
• Time in Patent Office: 284 Days
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: G11B 27/11 by using information not detectable on the record carrier
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 5/91 Television signal processing therefor
• CPC Class Codes: H04N 5/9305 involving the mixing of the reproduced video signal with a non-recorded signal, e.g. a text signal

Abstract

Systems for synchronizing events or transitions in visual content with musical features in audio content are configured to obtain audio and visual content; determine a minimum, maximum, and/or a target display duration for items of visual content; determine a first playback-time in the first audio content to associate with a start-of-display time for the first visual content; identify a first timeframe in the first audio content corresponding to a range of acceptable end-of-display times for the first visual content; identify musical features within the first timeframe; identify a candidate musical feature among the identified musical features in accordance with a hierarchy; and/or define a first candidate end-of-display time that aligns with the playback time of the candidate musical feature. A set of candidate end-of-display times are defined for multiple visual content items in a single multimedia project, the set identified by seeking a solution that increases rank among the hierarchy.

Claims

1. A system for synchronizing visual content with musical features in audio content, the system comprising:

one or more physical computer processors configured by computer readable instructions to;

obtain a first media file containing first visual information defining first visual content;

obtain a minimum display duration, a maximum display duration, and a target display duration for display of the first visual content;

obtain an audio file containing audio information representing audio content;

determine a first playback-time in the audio content to associate with a start-of-display time for display of the first visual content;

identify a range of acceptable end-of-display times for the first visual content based on the start-of-display time, the minimum display duration, and/or the maximum display duration of the first visual content, the range of acceptable end-of-display times for the first visual content corresponding to a first timeframe in the audio content;

identify a target end-of-display time of the first visual content based on the start-of-display time and the target display duration of the first visual content;

identify relative importance of musical features within the audio content during the first timeframe based on a musical feature hierarchy;

select a first musical feature from among the musical features within the audio content during the first timeframe based on (1) its relative importance in accordance with the musical feature hierarchy, and (2) its relative proximity in time to the target end-of-display time of the first visual content as compared to other musical features; and

determine a first end-of-display time for the first visual content that aligns with a playback time of the first musical feature.

2. The system of claim 1, wherein the start-of-display time for the first visual content is defined by a user.

3. The system of claim 1, wherein the one or more physical computer processors are further configured by computer readable instructions to:

obtain a second media file containing second visual information defining second visual content;

determine a minimum display duration, a maximum display duration, and a target display duration for display of the second visual content;

determine a second playback-time in the audio content to associate with a start-of-display time for display of the second visual content;

identify a range of acceptable end-of-display times for the second visual content based on the start-of-display time, the minimum display duration, and/or the maximum display duration of the second visual content, the range of acceptable end-of-display times for the second visual content corresponding to a second timeframe in the audio content;

identify a target end-of-display time of the second visual content based on the start-of-display time and the target display duration of the second visual content;

identify relative importance of musical features within the audio content during the second timeframe based on the musical feature hierarchy;

select a second musical feature from among the musical features within the audio content during the second timeframe based on (1) its relative importance in accordance with the musical feature hierarchy, and (2) its relative proximity in time to the target end-of-display time of the second visual content as compared to other musical features; and

determine a second end-of-display time for the second visual content that aligns with a playback time of the second musical feature.

4. The system of claim 3, wherein the second timeframe in the audio content is adjacent to the first timeframe.

5. The system of claim 3, wherein a combination of a first duration from the start-of- display time of the first visual content until the first end-of-display time for the first visual content and a second duration from the start-of-display-time of the second visual content until the second end-of-display time for the second visual content matches a total playback time of the audio content.

6. The system of claim 3, wherein the first musical feature and the second musical feature are selected such that an average of the proximity in time between the playback time of the first musical feature and the target-end of display time for the first visual content, and proximity in time between the playback time of the second musical feature and the target-end of display time for the second visual content is minimized.

7. The system of claim 3, wherein the start-of-display time and the end-of-display time for the first visual content is defined by a user.

8. A method for synchronizing transitions in visual content with musical features in audio content, the method comprising:

obtaining a first media file containing first visual information defining first visual content;

determining a minimum display duration, a maximum display duration, and a target display duration for display of the first visual content;

obtaining an audio file containing audio information representing audio content;

determining a first playback-time in the audio content to associate with a start-of-display-time for display of the first visual content;

identifying a range of acceptable end-of-display times for the first visual content based on the start-of-display time, the minimum display duration, and/or the maximum display duration of the first visual content, the range of acceptable end-of-display times for the first visual content corresponding to a first timeframe in the audio content;

identifying a target end-of-display time of the first visual content based on the start-of-display time and the target display duration of the first visual content;

identifying relative importance musical features within the audio content during the first timeframe based on a musical feature hierarchy;

selecting a first musical feature from among the musical features within the audio content during the first timeframe based on (1) its relative importance in accordance with the musical feature hierarchy, and (2) its relative proximity in time to the target end-of-display time of the first visual content as compared to other musical features; and

determining a first end-of-display time for the first visual content that aligns with a playback time of the first musical feature.

9. The method of claim 8, wherein start-of-display time for the first visual content is defined by a user.

10. The method of claim 8, further comprising:

obtaining a second media file containing second visual information defining second visual content;

determining a minimum display duration, a maximum display duration, and a target display duration for display of the second visual content;

determining a second playback-time in the audio content to associate with a start-of-display-time for display of the second visual content;

identifying a range of acceptable end-of-display times for the second visual content based on the start-of-display time, the minimum display duration, and/or the maximum display duration of the second visual content, the range of acceptable end-of-display times for the second visual content corresponding to a second timeframe in the audio content;

identifying a target end-of-display time of the second visual content based on the start-of-display time and the target display duration of the second visual content;

identifying relative importance of musical features within the audio content during the second timeframe based on the musical feature hierarchy;

selecting a second musical feature from among the musical features within the audio content during the second timeframe based on (1) its relative importance in accordance with the musical feature hierarchy, and (2) its relative proximity in time to the target end-of-display time of the second visual content as compared to other musical features; and

determining a second end-of-display time for the second visual content that aligns with a playback time of the second musical feature.

11. The method of claim 10, wherein the second timeframe in the audio content is adjacent to the first timeframe.

12. The method of claim 10, wherein a combination of a first duration from the start-of-display time of the first visual content until the first end-of-display time for the first visual content and a second duration from the start-of-display-time of the second visual content until the second end-of-display time for the second visual content matches a total playback time of the audio content.

13. The method of claim 10, wherein the first musical feature and the second musical feature are selected such that an average of the proximity in time between the playback time of the first musical feature and the target-end of display time for the first visual content, and proximity in time between the playback time of the second musical feature and the target-end of display time for the second visual content is minimized.

14. The method of claim 10, wherein the start-of-display time and the end-of-display time for the first visual content is defined by a user.

15. A system for synchronizing transitions in visual content with musical features in audio content, the system comprising:

one or more physical computer processors configured by computer readable instructions to;

obtain multiple media files containing visual information representing visual content;

define a display order for the visual content of the multiple media files;

define a minimum display duration, a maximum display duration, and a target display duration for display of the visual content of the multiple media files;

obtain an audio file containing audio information representing audio content;

define a first playback-time in the audio content as a start-of-display time for display of the visual content in accordance with the display order;

identify a range of acceptable end-of-display times for individual ones of the visual content of the multiple media files based on the start-of-display time, the minimum display duration, and/or the maximum display duration for the individual ones of the visual content of the multiple media files, the ranges of acceptable end-of-display times for the visual content of the multiple media files corresponding to timeframes in the audio content;

identify a target end-of-display times of the individual ones of the visual content of the multiple media files based on the start-of-display time and the target display duration of the individual ones of the visual content of the multiple media files;

identify relative importance of musical features within the audio content during the timeframes based on a musical feature hierarchy;

select a musical feature within individual ones of the timeframes based on (1) its relative importance as compared to the other identified musical features, and (2) its relative proximity in time to a corresponding target end-of-display time of the visual content as compared to other musical features; and

determine an end-of-display time for the individual ones of the visual content of the multiple media files, wherein the end-of-display time aligns with a playback time of the musical feature within a corresponding timeframe.

16. The system of claim 15, wherein the start-of-display time is defined by a user.

17. The system of claim 15, wherein the multiple media files include a first media file and a second media file, the first media file containing first visual information defining first visual content and the second media file containing second visual information defining second visual content.

18. The system of claim 17, wherein:

a first musical feature is selected for the first visual content and a second musical feature is selected for the second visual content; and

the first musical feature and the second musical feature are selected such that an average of the proximity in time between a playback time of the first musical feature and the target-end of display time for the first visual content, and proximity in time between a playback time of the second musical feature and the target-end of display time for the second visual content is minimized.

19. The system of claim 17, wherein the timeframes in the audio content include a first timeframe for the first visual content and a second timeframe for the second visual content, the second timeframe adjacent to the first timeframe.

20. The system of claim 17, wherein a combination of a first duration from the start-of-display time of the first visual content until the first end-of-display time for the first visual content and a second duration from the start-of-display-time of the second visual content until the second end-of-display time for the second visual content matches a total playback time of the audio content.