System and methods for encoding live multimedia content with synchronized audio data

• US 9,281,011 B2
• Filed: 09/27/2012
• Issued: 03/08/2016
• Est. Priority Date: 06/13/2012
• Status: Active Grant

• Current Assignee: Sonic IP Incorporated (Endeavor Group Holdings Inc.)
• Original Assignee: Sonic IP Incorporated (Endeavor Group Holdings Inc.)
• Inventors: Erofeev, Kirill, Petrova, Galina, Sahno, Dmitry
• Primary Examiner(s): Anyikire, Chikaodili E
• Assistant Examiner(s): KALAPODAS, DRAMOS
• Application Number:
  US13/629,306
• Publication Number:
  US 20130336412A1
• Time in Patent Office: 1,258 Days
• US Class Current: 1/1
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: H04N 21/2187 Live feed
• CPC Class Codes: H04N 21/242 Synchronization processes, e.g. processing of PCR [Program Clock References] arrangements for synchronising broadcast or distribution via plural systems in broadcast distribution systems H04H20/18

Abstract

Systems and methods for encoding live multimedia content with audio data synchronized with other streams of data, including video data, in accordance with embodiments of the invention are disclosed. In one embodiment of the invention, an encoding system includes multimedia content storage configured to store live multimedia content including audio data and video data, a processor, and a multimedia encoder, wherein the multimedia encoder configures the processor to receive live multimedia content, generate a timeline using the video data, compute a first time window, align the audio data to the video data using the audio data samples and the timeline, measure a synchronization value of the aligned audio data to the video data using the timeline, realign the audio data to the video data when the synchronization value exceeds a threshold value using the timeline, and multiplex the audio data and video data into a container file.

Claims

1. An encoding system, comprising:

live multimedia content storage configured to store live multimedia content, where the live multimedia content comprises audio data and video data, where the audio data comprises a plurality of audio data samples where each of the plurality of audio samples has an audio data sample duration and the video data comprises a plurality of video frames;

a processor; and

a multimedia encoder;

wherein the multimedia encoder directs the processor to;

receive live multimedia content;

generate a timeline using the video data of the received live multimedia content, where the timeline contains a plurality of timestamps, where at least one timestamp in the plurality of timestamps is determined using at least one video frame in the plurality of video frames in the video data of the live media content;

compute a first time window having a duration corresponding to the difference in time between a first one of the plurality of timestamps in the timeline and a second one of the plurality of timestamps in the timeline;

align audio data of the received live multimedia content to the video data of the received live multimedia content using a plurality of audio data samples of the received live multimedia content and the timeline by assigning a number of the plurality of audio data samples to the first time window based upon a number of the plurality of audio data samples that can occur within the first time window based upon the audio data sample durations of each of the plurality of audio data samples in the received live multimedia content and the duration of the first time window;

measure a synchronization value of the aligned audio data to the video data using the timeline wherein the synchronization value is measured by subtracting the audio data sample duration of the at least one of the plurality of audio data samples from the duration of the first window;

realign the audio data to the video data when the synchronization value exceeds a threshold value using the timeline; and

multiplex the audio data and video data into a container file.

2. The encoding system of claim 1, wherein the audio data is aligned to the video data with no gaps or overlays.

3. The encoding system of claim 1, wherein the audio data sample duration is a fixed duration.

4. The encoding system of claim 1, wherein the audio data sample duration is a variable duration.

5. The encoding system of claim 1, wherein the threshold value is pre-determined.

6. The encoding system of claim 1, wherein the threshold value is determined dynamically.

7. The encoding system of claim 1, wherein the audio data is realigned by moving at least one of the plurality of audio data samples assigned to the first time window to a second time window.

8. The encoding system of claim 1, wherein:

at least one video frame in the plurality of video frames comprises a video frame timestamp; and

at least one timestamp in the plurality of timestamps of the timeline is determined using the video frame timestamp.

9. The encoding system of claim 1, wherein:

at least one video frame in the plurality of video frames comprises a video frame duration; and

at least one timestamp in the plurality of timestamps of the timeline is determined using the video frame duration.

10. A method for encoding live multimedia content, comprising:

receiving live multimedia content using an encoding system, where the live multimedia content comprises audio data and video data, the audio data comprises a plurality of audio data samples wherein each of the plurality of audio data samples has an audio data sample duration, and the video data comprises a plurality of video frames;

generating a timeline using the video data of the received live multimedia content and the encoding system, where the timeline contains a plurality of timestamps, where at least one timestamp in the plurality of timestamps is determined using at least one video frame in the plurality of video frames from the video data of the received live multimedia content;

computing a first time window using the encoding system having a duration corresponding to the difference in time between a first timestamp in the timeline and a second timestamp in the timeline;

aligning the audio data of the received live multimedia content to the video data of the received live multimedia content using the encoding system, the plurality of audio data samples from the audio data of the received live multimedia content and the timeline by assigning a number of the plurality of audio data samples to the first time window based upon the audio data sample durations of each of the plurality of audio data samples in the received live multimedia content and the duration of the first time window;

measuring a synchronization value of the aligned audio data to the video data using the timeline and the encoding system wherein the synchronization value is measured by subtracting the audio data sample duration of the at least one of the plurality of audio data samples from the duration of the first window;

realigning the audio data to the video data when the synchronization value exceeds a threshold value using the timeline and the encoding system; and

multiplexing the audio data and video data into a container file using the encoding system.

11. The method of claim 10, wherein the audio data is aligned to the video data with no gaps or overlays.

12. The method of claim 10, wherein the audio data sample duration is a fixed duration.

13. The method of claim 10, wherein the audio data sample duration is a variable duration.

14. The method of claim 10, wherein the threshold value is pre-determined.

15. The method of claim 10, wherein the threshold value is determined dynamically.

16. The method of claim 10, wherein realigning the audio data comprises moving at least one audio data sample assigned to the first time window to a second time window using the encoding system.

17. The method of claim 10, wherein:

at least one video frame in the plurality of video frames comprises a video frame timestamp; and

determining at least one timestamp in the plurality of timestamps of the timeline utilizes the video frame timestamp and the encoding system.

18. The method of claim 10, wherein:

at least one video frame in the plurality of video frames comprises a video frame duration; and

determining at least one timestamp in the plurality of timestamps of the timeline utilizes the video frame duration and the encoding system.

19. A non-transitory machine readable medium containing processor instructions, where execution of the instructions by a processor causes the processor to perform a process comprising:

receiving live multimedia content, where the live multimedia content comprises audio data and video data, the audio data comprises a plurality of audio data samples wherein each of the plurality of audio data samples has an audio data sample duration, and the video data comprises a plurality of video frames;

generating a timeline using the video data of the received live multimedia content, where the timeline contains a plurality of timestamps, where at least one timestamp in the plurality of timestamps is determined using at least one video frame in the plurality of video frames from the video data of the received live multimedia content;

computing a first time window, where the first time window duration corresponding to the difference in time between a first timestamp in the timeline and a second timestamp in the timeline;

aligning the audio data of the received live multimedia content to the video data of the received live multimedia content using the plurality of audio data samples from the audio data of the received live multimedia content and the timeline by assigning a number of the plurality of audio data samples to the first time window based upon the audio data sample durations of each of the plurality of audio data samples in the received live multimedia content and the duration of the first time window;

measuring a synchronization value of the aligned audio data to the video data using the timeline wherein the synchronization value is measured by subtracting the audio data sample duration of the at least one of the plurality of audio data samples from the duration of the first window;

realigning the audio data to the video data when the synchronization value exceeds a threshold value using the timeline; and

multiplexing the audio data and video data into a container file using the encoding system.