Systems and methods for audio track selection in video editing

• US 10,341,712 B2
• Filed: 11/03/2017
• Issued: 07/02/2019
• Est. Priority Date: 04/07/2016
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Matias, Joven
• Primary Examiner(s): Tran, Thai Q
• Assistant Examiner(s): Smith, Stephen R
• Application Number:
  US15/802,674
• Publication Number:
  US 20180054648A1
• Time in Patent Office: 606 Days
• Field of Search: 386240, 386285, 386278, 386282, 715769, 715723, 84625
• CPC Class Codes: G11B 27/005 Reproducing at a different information rate from the information rate of recording for television signals H04N5/783
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/036 Insert-editing
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 21/4302 Content synchronisation processes, e.g. decoder synchronisation
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/47205 for manipulating displayed content, e.g. interacting with MPEG-4 objects, editing locally

Abstract

Video clips may be automatically edited to be synchronized for accompaniment by audio tracks. A preliminary version of a video clip may be made up from stored video content. Occurrences of video events within the preliminary version may be determined. A first audio track may include audio event markers. A first revised version of the video clip may be synchronized so that moments within the video clip corresponding to occurrences of video events are aligned with moments within the first audio track corresponding to audio event markers. Other audio tracks similar to the first audio track may be identified based on audio characteristic parameters. Other revised versions of the video clip may be synchronized.

Claims

1. A system that automatically edits video clips to synchronize accompaniment by different musical tracks, the system comprising:

one or more non-transitory storage media storing video content and first instructions defining a first edit of the video content, the first instructions indicating specific portions of the video content included in the first edit of the video content and an order of the specific portions of the video content within the first edit of the video content, the specific portions of the video content including a first portion and a second portion, the second portion following the first portion in the first edit of the video content, wherein;

the first edit of the video content includes one or more occurrences of video events, the individual occurrences of the video events corresponding to different moments within the first edit of the video content;

the first portion of the video content includes a first video event occurring at a first moment within the first edit of the video content;

the second portion of the video content includes a second video event occurring at a second moment within the first edit of the video content; and

the first edit of the video content is synchronized with a musical track, the musical track providing an accompaniment for the first edit of the video content, the musical track characterized by audio event markers including a first audio event marker occurring at a third moment within the musical track and a second audio event marker occurring at a fourth moment within the musical track, the fourth moment occurring later in the musical track than the third moment, the individual audio event markers corresponding to different moments within the musical track,

the first edit of the video content is synchronized with the musical track such that the first moment corresponding to the first video event is aligned to the third moment corresponding to the first audio event marker and the second moment corresponding to the second video event is aligned to the fourth moment corresponding to the second audio event marker; and

a boundary between the first portion of the video content and the second portion of the video content in the first edit of the video content is located at a given audio event of the musical track at or near a mid-point between the first video event and the second video event; and

one or more physical processors configured by machine readable instructions to;

determine a change to the musical track; and

determine second instructions defining a second edit of the video content that is synchronized with the changed musical track so that one or more moments within the second edit of the video content corresponding to one or more of the occurrences of the video events are aligned with one or more moments within the changed musical track corresponding to one or more of the audio event markers, wherein determination of the second instructions includes;

identifying the mid-point between the first video event and the second video event;

identifying an audio event of the changed musical track that is nearest to the mid-point; and

shifting the boundary between the first portion of the video content and the second portion of the video content in the second edit of the video content to be located at the audio event of the changed musical track that is nearest to the mid-point.

2. The system of claim 1, wherein the musical track has a first duration and the changed musical track has a second duration different from the first duration.

3. The system of claim 1, wherein the musical track has a first tempo and the changed musical track has a second tempo different from the first tempo.

4. The system of claim 1, wherein the musical track has a first number of the audio event markers and the changed musical track has a second number of the audio event makers, the first number being different from the second number.

5. The system of claim 1, wherein the boundary between the first portion of the video content and the second portion of the video content in the first edit of the video content defines a first duration of the first portion and a second duration of the second portion.

6. The system of claim 1, wherein the audio event markers are ranked.

7. The system of claim 6, wherein the audio event markers are aligned to the video events based on the ranking.

8. The system of claim 6, wherein the audio event markers are aligned to the video events based on a reverse of the ranking.

9. The system of claim 1, wherein one or more of the audio event markers indicate one of more of a beat, a tempo, a rhythm, an instrument, a volume, a vocal, a frequency a style, a start, and/or an end of the musical track at corresponding moments within the musical track.

10. The system of claim 1, wherein the given audio event of the musical track at or near the mid-point of the first video event and the second video event includes a beat nearest to the mid-point.

11. A method for automatically editing video clips to synchronize accompaniment by different musical tracks, the method performed by a computing system including one or more physical processors, the method comprising:

accessing, by the computing system, one or more non-transitory storage media storing video content and first instructions defining a first edit of the video content, the first instructions indicating specific portions of the video content included in the edit of the video content and an order of the specific portions of the video content within the first edit of the video content, the specific portions of the video content including a first portion and a second portion, the second portion following the first portion in the first edit of the video content, wherein;

the first edit of the video content includes one or more occurrences of video events, the individual occurrences of the video events corresponding to different moments within the first edit of the video content;

the first portion of the video content includes a first video event occurring at a first moment within the first edit of the video content;

the second portion of the video content includes a second video event occurring at a second moment within the first edit of the video content; and

the first edit of the video content is synchronized with a musical track, the musical track providing an accompaniment for the edit of the video content, the musical track characterized by audio event markers including a first audio event marker occurring at a third moment within the musical track and a second audio event marker occurring at a fourth moment within the musical track, the fourth moment occurring later in the musical track than the third moment, the individual audio event markers corresponding to different moments within the musical track,

the first edit of the video content is synchronized with the musical track such that the first moment corresponding to the first video event is aligned to the third moment corresponding to the first audio event marker and the second moment corresponding to the second video event is aligned to the fourth moment corresponding to the second audio event marker; and

a boundary between the first portion of the video content and the second portion of the video content in the first edit of the video content is located at a given audio event of the musical track at or near a mid-point between the first video event and the second video event;

determining, by the computing system, a change to the musical track; and

determining, by the computing system, second instructions defining a second edit of the video content that is synchronized with the changed musical track so that the one or more moments within the second edit of the video content corresponding to the one or more occurrences of the video events are aligned with one or more moments within the changed musical track corresponding to one or more of the audio event markers, wherein determination of the second instructions includes;

identifying the mid-point between the first video event and the second video event;

identifying an audio event of the changed musical track that is nearest to the mid-point; and

shifting the boundary between the first portion of the video content and the second portion of the video content in the second edit of the video content to be located at the audio event of the changed musical track that is nearest to the mid-point.

12. The method of claim 11, wherein the musical track has a first duration and the changed musical track has a second duration different from the first duration.

13. The method of claim 11, wherein the musical track has a first tempo and the changed musical track has a second tempo different from the first tempo.

14. The method of claim 11, wherein the musical track has a first number of the audio event markers and the changed musical track has a second number of the audio event makers, the first number being different from the second number.

15. The method of claim 11, wherein the boundary between the first portion of the video content and the second portion of the video content in the first edit of the video content defines a first duration of the first portion and a second duration of the second portion.

16. The method of claim 11, wherein the audio event markers are ranked.

17. The method of claim 16, wherein the audio event markers are aligned to the video events based on the ranking.

18. The method of claim 16, wherein the audio event markers are aligned to the video events based on a reverse of the ranking.

19. The method of claim 11, wherein one or more of the audio event markers indicate one of more of a beat, a tempo, a rhythm, an instrument, a volume, a vocal, a frequency a style, a start, and/or an end of the audio track at corresponding moments within the audio track.

20. The method of claim 11, wherein the given audio event of the musical track at or near the mid-point of the first video event and the second video event includes a beat nearest to the mid-point.