System and Methods for Encoding Live Multimedia Content with Synchronized Audio Data

• US 20130336412A1
• Filed: 09/27/2012
• Published: 12/19/2013
• Est. Priority Date: 06/13/2012
• Status: Active Grant

• Current Assignee: Sonic IP Incorporated (Endeavor Group Holdings Inc.)
• Original Assignee: DivX, LLC (f/k/a DivX CF Holdings LLC) (FIG LLC (d/b/a Fortress Investment Group LLC))
• Inventors: Erofeev, Kirill, Petrova, Galina, Sahno, Dmitry
• Granted Patent: US 9,281,011 B2
• Time in Patent Office: Days
• US Class Current: 375/240.28
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: H04N 21/2187 Live feed
• CPC Class Codes: H04N 21/242 Synchronization processes, e.g. processing of PCR [Program Clock References] arrangements for synchronising broadcast or distribution via plural systems in broadcast distribution systems H04H20/18

Abstract

Systems and methods for encoding live multimedia content with audio data synchronized with other streams of data, including video data, in accordance with embodiments of the invention are disclosed. In one embodiment of the invention, an encoding system includes multimedia content storage configured to store live multimedia content including audio data and video data, a processor, and a multimedia encoder, wherein the multimedia encoder configures the processor to receive live multimedia content, generate a timeline using the video data, compute a first time window, align the audio data to the video data using the audio data samples and the timeline, measure a synchronization value of the aligned audio data to the video data using the timeline, realign the audio data to the video data when the synchronization value exceeds a threshold value using the timeline, and multiplex the audio data and video data into a container file.

Claims

1. An encoding system, comprising:

live multimedia content storage configured to store live multimedia content, where the live multimedia content comprises audio data and video data, where the audio data comprises a plurality of audio data samples having an audio data sample duration and the video data comprises a plurality of video frames;

a processor; and

a multimedia encoder;

wherein the multimedia encoder configures the processor to;

receive live multimedia content;

generate a timeline using the video data, where the timeline contains a plurality of timestamps, where at least one timestamp in the plurality of timestamps is determined using at least one video frame in the plurality of video frames;

compute a first time window, where the first time window comprises a first time window duration corresponding to the difference in time between a first timestamp in the timeline and a second timestamp in the timeline;

align the audio data to the video data using the audio data samples and the timeline by assigning at least one audio data sample to the first time window based upon the number of audio data sample durations that can occur within the first time window duration;

measure a synchronization value of the aligned audio data to the video data using the timeline;

realign the audio data to the video data when the synchronization value exceeds a threshold value using the timeline; and

multiplex the audio data and video data into a container file.

2. The encoding system of claim 1, wherein the audio data is aligned to the video data with no gaps or overlays.

3. The encoding system of claim 1, wherein the audio data sample duration is a fixed duration.

4. The encoding system of claim 1, wherein the audio data sample duration is a variable duration.

5. The encoding system of claim 1, wherein the synchronization value is measured by subtracting the duration of the at least one audio data sample from the first time window duration.

6. The encoding system of claim 1, wherein the threshold value is pre-determined.

7. The encoding system of claim 1, wherein the threshold value is determined dynamically.

8. The encoding system of claim 1, wherein the audio data is realigned by moving at least one audio data sample assigned to the first time window to a second time window.

9. The encoding system of claim 1, wherein:

at least one video frame in the plurality of video frames comprises a video frame timestamp; and

at least one timestamp in the plurality of timestamps is determined using the video frame timestamp.

10. The encoding system of claim 1, wherein:

at least one video frame in the plurality of video frames comprises a video frame duration; and

at least one timestamp in the timeline is determined using the video frame duration.

11. A method for encoding live multimedia content, comprising:

receiving live multimedia content using an encoding system, where the live multimedia content comprises audio data and video data, the audio data comprises a plurality of audio data samples having an audio data sample duration, and the video data comprises a plurality of video frames;

generating a timeline using the video data and the encoding system, where the timeline contains a plurality of timestamps, where at least one timestamp in the plurality of timestamps is determined using at least one video frame in the plurality of video frames;

computing a first time window using the encoding system, where the first time window comprises a first time window duration corresponding to the difference in time between a first timestamp in the timeline and a second timestamp in the timeline;

aligning the audio data to the video data using the encoding system, the audio data samples and the timeline by assigning at least one audio data sample to the first time window based upon the number of audio data sample durations that can occur within the first time window duration;

measuring a synchronization value of the aligned audio data to the video data using the timeline and the encoding system;

realigning the audio data to the video data when the synchronization value exceeds a threshold value using the timeline and the encoding system; and

multiplexing the audio data and video data into a container file using the encoding system.

12. The method of claim 11, wherein the audio data is aligned to the video data with no gaps or overlays.

13. The method of claim 11, wherein the audio data sample duration is a fixed duration.

14. The method of claim 11, wherein the audio data sample duration is a variable duration.

15. The method of claim 11, wherein measuring the synchronization value comprises subtracting the duration of the at least one audio data sample from the first time window duration using the encoding system.

16. The method of claim 11, wherein the threshold value is pre-determined.

17. The method of claim 11, wherein the threshold value is determined dynamically.

18. The method of claim 11, wherein realigning the audio data comprises moving at least one audio data sample assigned to the first time window to a second time window using the encoding system.

19. The method of claim 11, wherein:

at least one video frame in the plurality of video frames comprises a video frame timestamp; and

determining at least one timestamp in the plurality of timestamps utilizes the video frame timestamp and the encoding system.

20. The method of claim 11, wherein:

at least one video frame in the plurality of video frames comprises a video frame duration; and

determining at least one timestamp in the timeline utilizes the video frame duration and the encoding system.

21. A machine readable medium containing processor instructions, where execution of the instructions by a processor causes the processor to perform a process comprising:

receiving live multimedia content, where the live multimedia content comprises audio data and video data, the audio data comprises a plurality of audio data samples having an audio data sample duration, and the video data comprises a plurality of video frames;

generating a timeline using the video data, where the timeline contains a plurality of timestamps, where at least one timestamp in the plurality of timestamps is determined using at least one video frame in the plurality of video frames;

computing a first time window, where the first time window comprises a first time window duration corresponding to the difference in time between a first timestamp in the timeline and a second timestamp in the timeline;

aligning the audio data to the video data, the audio data samples and the timeline by assigning at least one audio data sample to the first time window based up-on the number of audio data sample durations that can occur within the first time window duration;

measuring a synchronization value of the aligned audio data to the video data using the timeline;

realigning the audio data to the video data when the synchronization value exceeds a threshold value using the timeline; and

multiplexing the audio data and video data into a container file using the encoding system.