CAPTURING MEDIA IN SYNCHRONIZED FASHION

• US 20080304573A1
• Filed: 10/16/2007
• Published: 12/11/2008
• Est. Priority Date: 06/10/2007
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Bushell, John Samuel, Chapman, Gregory R., MOSS, Nicolas, Ford, Bradley
• Granted Patent: US 8,576,922 B2
• Time in Patent Office: Days
• US Class Current: 375/240.28
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: H04N 21/2368 Multiplexing of audio and video streams
• CPC Class Codes: H04N 21/4305 Synchronising client clock from received content stream, e.g. locking decoder clock with encoder clock, extraction of the PCR packets arrangements for synchronising receiver with transmitter by comparing receiver clock with transmitter clock H04L7/0012; arrangements for synchronising receiver with transmitter wherein the receiver takes measures against momentary loss of synchronisation H04L7/0083
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/4341 Demultiplexing of audio and video streams
• CPC Class Codes: H04N 21/4622 Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56
• CPC Class Codes: H04N 9/802 involving processing of the sound signal H04N9/806, H04N9/835 take precedence

Abstract

Techniques for synchronizing audio and video content for presentation to a user at a same rate are provided. Streams of content from two or more sources of media, each media source having an associated clock, are synchronized by a synchronizing component and processor with respect to a master clock. As well, techniques are provided for ensuring that output devices are synchronized at preview startup. That is, such techniques ensure that the output devices start playing the media at the same time as well as at the same rate.

Claims

1. A computer-implemented method comprising:

receiving a first media stream from a first media source of two or more media sources, wherein the first media source is associated with a first clock,

receiving a second media stream from a second media source of the two or more media sources, wherein the second media source is associated with a second clock,

determining a first measurement by measuring the first clock relative to a master clock and processing the first media stream based on the first measurement to result in a synchronized first media stream;

determining a second measurement by measuring the second clock relative to the master clock and processing the second media stream based on the second measurement to result in a synchronized second media stream; and

performing any of;

writing the synchronized first media stream and the synchronized second media stream to one or more files;

determining a third measurement by measuring a third clock, which is associated with a first output device of one or more output devices, relative to the master clock and processing the synchronized first media stream based on the third measurement to result in a second synchronized first media stream; and

determining a fourth measurement by measuring a fourth clock, which is associated with a second output device of the one or more output devices, relative to the master clock and processing the synchronized second media stream based on the fourth measurement to result in a second synchronized second media stream.

2. The method of claim 1, further comprising the step of:

assigning any of the first, second, third, or fourth clocks as the master clock.

3. The method of claim 1, wherein the first media stream and the second media stream are any of:

video, audio, and audio-video muxed.

4. The method of claim 1, wherein the second media stream is video and wherein processing the second input stream comprises any of:

adding one or more video frames to the second media stream;

duplicating an existing frame in the second media stream;

deleting one or more video frames from the second media stream;

lengthening observed duration of a video frame of the second media stream; and

shortening observed duration of a video frame of the second media stream.

5. The method of claim 1, wherein the first media stream is audio and wherein processing the first media stream comprises:

re-sampling the first media stream based on the master clock.

6. The method of claim 1, wherein:

the first media stream is audio;

the second media stream is video; and

the second media source is a video device and applies any of the following media classes;

I-frame;

P-frame; and

B-frame;

or

one of the one or more output devices is a video device and applies any of the following media classes;

I-frame;

P-frame; and

B-frame.

7. The method of claim 1, wherein each of the synchronized first media stream and the synchronized second media stream comprises a start point and a rate at which the synchronized first media stream and the synchronized second media stream are delivered.

8. The method of claim 1, wherein each of the second synchronized first media stream and the second synchronized second media stream comprises a start point and a rate at which the second synchronized first media stream and the second synchronized second media stream are delivered.

9. The method of claim 8, wherein determining the start point and the rate further comprises using a state machine and six states as follows:

reset;

priming video output synchronizer units;

priming video output units;

priming audio output units;

starting to run; and

running.

10. The method of claim 1, further comprising a parameter that specifies a maximum amount of latency between when the first output device is sent the second synchronized first media stream and when the first output device presents the second synchronized first media stream to a user, or between when the second output device is sent the second synchronized second media stream and when the second output device presents the second synchronized second media stream to a user.

11. A computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the steps of:

receiving a first media stream from a first media source of two or more media sources, wherein the first media source is associated with a first clock,

receiving a second media stream from a second media source of the two or more media sources, wherein the second media source is associated with a second clock,

determining a first measurement by measuring the first clock relative to a master clock and processing the first media stream based on the first measurement to result in a synchronized first media stream;

determining a second measurement by measuring the second clock relative to the master clock and processing the second media stream based on the second measurement to result in a synchronized second media stream; and

performing any of;

writing the synchronized first media stream and the synchronized second media stream to one or more files;

determining a third measurement by measuring a third clock, which is associated with a first output device of one or more output devices, relative to the master clock and processing the synchronized first media stream based on the third measurement to result in a second synchronized first media stream; and

determining a fourth measurement by measuring a fourth clock, which is associated with a second output device of the one or more output devices, relative to the master clock and processing the synchronized second media stream based on the fourth measurement to result in a second synchronized second media stream.

12. The computer-readable storage medium of claim 11, wherein storing one or more sequences of instructions which, when executed by one or more processors, further causes the one or more processors to perform the step of:

assigning any of the first, second, third, or fourth clocks as the master clock.

13. The computer-readable storage medium of claim 11, wherein the first media stream and the second media stream are any of:

video, audio, and audio-video muxed.

14. The computer-readable storage medium of claim 11, wherein the second media stream is video and wherein processing the second input stream comprises any of:

adding one or more video frames to the second media stream;

duplicating an existing frame in the second media stream;

deleting one or more video frames from the second media stream;

lengthening observed duration of a video frame of the second media stream; and

shortening observed duration of a video frame of the second media stream.

15. The computer-readable storage medium of claim 11, wherein the first media stream is audio and wherein processing the first media stream comprises:

re-sampling the first media stream based on the master clock.

16. The computer-readable storage medium of claim 11, wherein:

the first media stream is audio;

the second media stream is video; and

the second media source is a video device and applies any of the following media classes;

I-frame;

P-frame; and

B-frame;

or

one of the one or more output devices is a video device and applies any of the following media classes;

I-frame;

P-frame; and

B-frame.

17. The computer-readable storage medium of claim 11, wherein each of the synchronized first media stream and the synchronized second media stream comprises a start point and a rate at which the synchronized first media stream and the synchronized second media stream are delivered.

18. The computer-readable storage medium of claim 11, wherein each of the second synchronized first media stream and the second synchronized second media stream comprises a start point and a rate at which the second synchronized first media stream and the second synchronized second media stream are delivered.

19. The computer-readable storage medium of claim 18, wherein determining the start point and the rate further comprises using a state machine and six states as follows:

reset;

priming video output synchronizer units;

priming video output units;

priming audio output units;

starting to run; and

running.

20. The computer-readable storage medium of claim 11, further comprising a parameter that specifies a maximum amount of latency between when the first output device is sent the second synchronized first media stream and when the first output device presents the second synchronized first media stream to a user, or between when the second output device is sent the second synchronized second media stream and when the second output device presents the second synchronized second media stream to a user.

21. An apparatus, comprising:

a module for receiving a first media stream from a first media source of two or more media sources, wherein the first media source is associated with a first clock,

a module for receiving a second media stream from a second media source of the two or more media sources, wherein the second media source is associated with a second clock,

a module for determining a first measurement by measuring the first clock relative to a master clock and processing the first media stream based on the first measurement to result in a synchronized first media stream;

a module for determining a second measurement by measuring the second clock relative to the master clock and processing the second media stream based on the second measurement to result in a synchronized second media stream; and

a module for performing any of;

writing the synchronized first media stream and the synchronized second media stream to one or more files;

determining a third measurement by measuring a third clock, which is associated with a first output device of one or more output devices, relative to the master clock and processing the synchronized first media stream based on the third measurement to result in a second synchronized first media stream; and

determining a fourth measurement by measuring a fourth clock, which is associated with a second output device of the one or more output devices, relative to the master clock and processing the synchronized second media stream based on the fourth measurement to result in a second synchronized second media stream.

22. The apparatus of claim 21, further comprising:

a module for assigning any of the first, second, third, or fourth clocks as the master clock.

23. The apparatus of claim 21, wherein the first media stream and the second media stream are any of:

video, audio, and audio-video muxed.

24. The apparatus of claim 21, wherein the second media stream is video and wherein processing the second input stream comprises any of:

adding one or more video frames to the second media stream;

duplicating an existing frame in the second media stream;

deleting one or more video frames from the second media stream;

lengthening observed duration of a video frame of the second media stream; and

shortening observed duration of a video frame of the second media stream.

25. The apparatus of claim 21, wherein the first media stream is audio and wherein processing the first media stream comprises:

re-sampling the first media stream based on the master clock.

26. The apparatus of claim 21, wherein:

the first media stream is audio;

the second media stream is video; and

the second media source is a video device and applies any of the following media classes;

I-frame;

P-frame; and

B-frame;

or

one of the one or more output devices is a video device and applies any of the following media classes;

I-frame;

P-frame; and

B-frame.