Systems and methods for synchronization based on audio track changes in video editing

• US 9,794,632 B1
• Filed: 04/07/2016
• Issued: 10/17/2017
• Est. Priority Date: 04/07/2016
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Matias, Joven
• Primary Examiner(s): Shibru, Helen
• Application Number:
  US15/093,707
• Time in Patent Office: 558 Days
• Field of Search: 386241, 386337, 386285, 386287, 386290, 386201
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/036 Insert-editing
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 21/4302 Content synchronisation processes, e.g. decoder synchronisation
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/47205 for manipulating displayed content, e.g. interacting with MPEG-4 objects, editing locally

Abstract

Video clips may be automatically edited to be synchronized for accompaniment by audio tracks. A first version of a video clip may be made up from stored video content. The first version of the video clip may include one occurrences of video events. An audio track may include audio event markers. The first version of the video clip may be synchronized so that moments within the video clip corresponding to occurrences of video events are aligned with moments within the audio track corresponding to audio event markers. Changes to the audio track may be determined. A second version of the video clip may be synchronized so that moments within the video clip corresponding to occurrences of video events are aligned with moments within the changed audio track corresponding to audio event markers.

Claims

1. A system that automatically edits video clips to synchronize accompaniment by different musical tracks, the system comprising:

one or more storage media storing video content and first instructions defining a first version of a video clip made up from the stored video content, the first instructions indicating specific portions of the video content to be included in the first version of the video clip and an order in which the specific portions of the video content should be presented, the specific portions of the video content including a first portion and a second portion, the second portion following the first portion in the first version of the video clip, wherein;

the first version of the video clip includes one or more occurrences of video events, the video events corresponding to particular visuals within the video clip, the individual occurrences of the video events corresponding to different moments within the first version of the video clip;

the first portion of the video content includes a first video event occurring at a first moment within the first version of the video clip;

the second portion of the video content includes a second video event occurring at a second moment within the first version of the video clip; and

the first version of the video clip is synchronized with a first musical track, the first musical track providing an accompaniment for the video clip, the first musical track characterized by first musical track audio event markers including a first audio event marker occurring at a third moment within the first musical track and a second audio event marker occurring at a fourth moment within the first musical track, the fourth moment occurring later in the first musical track than the third moment, the individual first musical track audio event markers corresponding to different moments within the first musical track and characterizing audio characteristics of the first musical track at the different moments,

wherein;

the first version of the video clip is synchronized with the first musical track such that the first moment corresponding to the first video event is aligned to the third moment corresponding to the first audio event marker and the second moment corresponding to the second video event is aligned to the fourth moment corresponding to the second audio event marker; and

a boundary between the first portion of the video content and the second portion of the video content in the first version of the video clip is located at a beat of the first musical track at or near a mid-point of the first video event and the second video event; and

one or more physical processors configured by machine readable instructions to;

determine that a second musical track has been selected to replace the first musical track as the accompaniment for the video clip; and

determine second instructions defining a second version of the video clip that is synchronized with the second musical track so that one or more moments within the video clip corresponding to one or more of the occurrences of the video events are aligned with one or more moments within the second musical track corresponding to one or more second musical track audio event markers.

2. The system of claim 1, wherein the first musical track has a first duration and the second musical track has a second duration different from the first duration.

3. The system of claim 1, wherein the first musical track has a first tempo and the second musical track has a second tempo different from the first tempo.

4. The system of claim 1, wherein the first musical track has a first number of the first musical track audio event markers and the second musical track has a second number of the second musical track audio event makers, the first number being different from the second number.

5. The system of claim 1, wherein one or more of the musical track audio event markers indicate one of more of a beat, a tempo, a rhythm, an instrument, a volume, a vocal, a frequency a style, a start, and/or an end of the audio track at corresponding moments within the audio track.

6. The system of claim 1, wherein the boundary between the first portion of the video content and the second portion of the video content in the first version of the video clip defines a first duration of the first portion and a second duration of the second portion in the first version of the video clip.

7. The system of claim 1, wherein:

the one or more second musical track audio event markers include a third audio event marker occurring at a fifth moment within the second musical track and a fourth audio event marker occurring at a sixth moment within the second musical track, the sixth moment occurring later in the second musical track than the fifth moment; and

the second version of the video clip is synchronized with the second musical track such that the first moment corresponding to the first video event is aligned to the fifth moment corresponding to the third audio event marker and the second moment corresponding to the second video event is aligned to the sixth moment corresponding to the fourth audio event marker.

8. The system of claim 7, wherein:

the second version of the video clip is synchronized with the second musical track such that the boundary between the first portion of the video content and the second portion of the video content in the second version of the video clip is located at a beat of the second musical track at or near the mid-point of the first video event and the second video event.

9. The system of claim 1, wherein the first musical track audio event markers are ranked.

10. The system of claim 9, wherein the first musical track audio event markers are aligned to the video events based on the ranking.

11. The system of claim 9, wherein the first musical track audio event markers are aligned to the video events based on a reverse of the ranking.

12. The system of claim 1, wherein the beat of the first musical track at or near the mid-point of the first video event and the second video event includes a down beat nearest to the mid-point.

13. A method for automatically editing video clips to synchronize accompaniment by different musical tracks, the method comprising:

accessing one or more storage media storing video content and first instructions defining a first version of a video clip made up from the stored video content, the first instructions indicating specific portions of the video content to be included in the first version of the video clip and an order in which the specific portions of the video content should be presented, the specific portions of the video content including a first portion and a second portion, the second portion following the first portion in the first version of the video clip, wherein;

the first version of the video clip includes one or more occurrences of video events, the video events corresponding to particular visuals within the video clip, the individual occurrences of the video events corresponding to different moments within the first version of the video clip;

the first portion of the video content includes a first video event occurring at a first moment within the first version of the video clip;

the second portion of the video content includes a second video event occurring at a second moment within the first version of the video clip; and

the first version of the video clip is synchronized with a first musical track, the first musical track providing an accompaniment for the video clip, the first musical track characterized by first musical track audio event markers including a first audio event marker occurring at a third moment within the first musical track and a second audio event marker occurring at a fourth moment within the first musical track, the fourth moment occurring later in the first musical track than the third moment, the individual first musical track audio event markers corresponding to different moments within the first musical track and characterizing audio characteristics of the first musical track at the different moments,

wherein;

the first version of the video clip is synchronized with the first musical track such that the first moment corresponding to the first video event is aligned to the third moment corresponding to the first audio event marker and the second moment corresponding to the second video event is aligned to the fourth moment corresponding to the second audio event marker; and

a boundary between the first portion of the video content and the second portion of the video content in the first version of the video clip is located at a beat of the first musical track at or near a mid-point of the first video event and the second video event;

determining that a second musical track has been selected to replace the first musical track as the accompaniment for the video clip; and

determining second instructions defining a second version of the video clip that is synchronized with the second musical track so that the one or more moments within the video clip corresponding to the one or more occurrences of the video events are aligned with one or more moments within the second musical track corresponding to one or more second musical track audio event markers.

14. The method of claim 13, wherein the first musical track has a first duration and the second musical track has a second duration different from the first duration.

15. The method of claim 13, wherein the first musical track has a first tempo and the second musical track has a second tempo different from the first tempo.

16. The method of claim 13, wherein the first musical track has a first number of the first musical track audio event markers and the second musical track has a second number of the second musical track audio event makers, the first number being different from the second number.

17. The method of claim 13, wherein one or more of the musical track audio event markers indicate one of more of a beat, a tempo, a rhythm, an instrument, a volume, a vocal, a frequency a style, a start, and/or an end of the audio track at corresponding moments within the audio track.

18. The method of claim 13, wherein the boundary between the first portion of the video content and the second portion of the video content in the first version of the video clip defines a first duration of the first portion and a second duration of the second portion in the first version of the video clip.

19. The method of claim 13, wherein:

the one or more second musical track audio event markers include a third audio event marker occurring at a fifth moment within the second musical track and a fourth audio event marker occurring at a sixth moment within the second musical track, the sixth moment occurring later in the second musical track than the fifth moment; and

the second version of the video clip is synchronized with the second musical track such that the first moment corresponding to the first video event is aligned to the fifth moment corresponding to the third audio event marker and the second moment corresponding to the second video event is aligned to the sixth moment corresponding to the fourth audio event marker.

20. The method of claim 19, wherein:

the second version of the video clip is synchronized with the second musical track such that the boundary between the first portion of the video content and the second portion of the video content in the second version of the video clip is located at a beat of the second musical track at or near the mid-point of the first video event and the second video event.