Methods and systems for synchronizing data streams

• US 20030135822A1
• Filed: 01/15/2002
• Published: 07/17/2003
• Est. Priority Date: 01/15/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Evans, Glenn F.
• Granted Patent: US 6,930,620 B2
• Time in Patent Office: Days
• US Class Current: 715/500.1
• CPC Class Codes: H04N 21/2368 Multiplexing of audio and video streams
• CPC Class Codes: H04N 21/4143 embedded in a Personal Computer [PC]
• CPC Class Codes: H04N 21/4305 Synchronising client clock from received content stream, e.g. locking decoder clock with encoder clock, extraction of the PCR packets arrangements for synchronising receiver with transmitter by comparing receiver clock with transmitter clock H04L7/0012; arrangements for synchronising receiver with transmitter wherein the receiver takes measures against momentary loss of synchronisation H04L7/0083
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/4341 Demultiplexing of audio and video streams
• CPC Class Codes: H04N 21/4344 Remultiplexing of multiplex streams, e.g. by modifying time stamps or remapping the packet identifiers

Abstract

Methods and systems are provided for synchronizing various time-stamped data streams. The data streams can be synchronized to another data stream or to a point of reference such as a reference clock. In one embodiment, synchronization processing takes place in association with a filter graph comprising multiple filters. The filter graph is configured to process multiple timestamped data streams for rendering the data streams in accordance with data stream timestamps. A synchronization module is provided and is associated with the filter graph queries individual filters of the filter graph to ascertain input timestamp-to-output timestamp mappings. The module computes adjustments that are to be made to output timestamps in order to synchronize the data streams, and then instructs queried filters to adjust their output timestamps in accordance with its adjustment computations.

Claims

1. A method comprising:

providing at least one time-stamped data stream that is to be rendered in accordance with data stream timestamps; and

synchronizing said at least one time-stamped data stream relative to a reference by adjusting one or more timestamps of said at least one time-stamped data stream.

2. The method of claim 1, wherein the reference comprises another time-stamped data stream.

3. The method of claim 1, wherein the reference comprises a reference clock.

4. The method of claim 1, wherein the act of synchronizing comprises doing so periodically.

5. The method of claim 1, wherein the act of synchronizing comprises determining whether said at least one time-stamped data stream is within a defined tolerance and, if not, performing said act of synchronizing.

6. The method of claim 1, wherein said at least one time-stamped data stream comprises a video stream.

7. The method of claim 1, wherein said at least one time-stamped data stream comprises an audio stream.

8. The method of claim 1, wherein said at least one time-stamped data stream comprises a video stream and an audio stream.

9. A method comprising:

providing a filter graph comprising multiple filters, the filter graph being configured to process multiple timestamped data streams for rendering the data streams in accordance with data stream timestamps; and

synchronizing said at least one time-stamped data stream relative to a reference by adjusting one or more timestamps of said at least one time-stamped data stream.

10. The method of claim 9, wherein the act of synchronizing comprises instructing one or more of the filters to adjust at least one timestamp associated with at least one data sample of one or more of the data streams.

11. The method of claim 10, wherein the act of instructing comprises instructing one or more decoder filters to adjust at least one timestamp associated with at least one data sample.

12. The method of claim 9, wherein the reference comprises another time-stamped data stream.

13. The method of claim 9, wherein the reference comprises a reference clock.

14. The method of claim 9, wherein the act of synchronizing comprises doing so periodically.

15. The method of claim 9, wherein the act of synchronizing comprises determining whether said at least one time-stamped data stream is within a defined tolerance and, if not, performing said act of synchronizing.

16. One or more computer-readable media comprising instructions which, when executed by one or more processors, cause the one or more processors to implement the method of claim 9.

17. A method comprising:

providing a filter graph comprising multiple filters, the filter graph being configured to process multiple timestamped data streams for rendering the data streams in accordance with data stream timestamps;

providing a synchronization module associated with the filter graph, the synchronization module being configured to;

query individual filters of the filter graph to ascertain input timestamp-to-output timestamp mappings,

compute adjustments that are to be made to output timestamps in order to synchronize the data streams, and

instruct individual queried filters to adjust their output timestamps in accordance with computed adjustments;

querying individual filters of the filter graph, using the synchronization module, to ascertain input timestamp-to-output timestamp mappings;

computing, based on the ascertained input timestamp-to-output timestamp mappings, adjustments that are to be made to output timestamps in order to synchronize the data streams; and

synchronizing the data streams by instructing one or more of the queried filters to adjust their output timestamps in accordance with the computed adjustments.

18. The method of claim 17, wherein the act of synchronizing comprises doing so periodically.

19. The method of claim 17, wherein the act of synchronizing comprises determining whether the multiple time-stamped data streams are within a defined tolerance and, if not, performing said act of synchronizing.

20. The method of claim 17, wherein the act of querying comprises querying individual filters for a current input timestamp and a current output timestamp.

21. The method of claim 20, wherein the act of computing comprises:

extrapolating at least one curve characterizing the timestamp mappings for at least one of the timestamp mappings to a defined point corresponding to a common input timestamp; and

computing said adjustments based on the extrapolated line(s).

22. The method of claim 21, wherein the act of extrapolating comprises performing a linear extrapolation.

23. The method of claim 21, wherein the defined point comprises the largest value of an input timestamp returned by the act of querying.

24. The method of claim 21, wherein the act of computing comprises computing a skip value by taking the difference between the largest output timestamp value at the common input timestamp and the output timestamp value for said at least one curve at the common input timestamp.

25. The method of claim 24, wherein the act of instructing comprises instructing one or more filters to jump their output timestamp values by an associated skip value at the common input timestamp.

26. An architecture comprising:

a filter graph comprising multiple filters, the filter graph being configured to process multiple timestamped data streams for rendering the data streams in accordance with data stream timestamps; and

a synchronization module associated with the filter graph, the synchronization module being configured to;

query individual filters of the filter graph to ascertain input timestamp-to-output timestamp mappings,

compute adjustments that are to be made to output timestamps in order to synchronize the data streams, and

instruct the queried filters to adjust their output timestamps in accordance with its adjustment computations.

27. The architecture of claim 26, wherein the synchronization module is configured to periodically instruct one or more of the queried filters to adjust their output timestamps.

28. The architecture of claim 26, wherein the synchronization module is configured to determine whether the multiple time-stamped data streams are within a defined tolerance and, if not, at least instruct one or more of the queried filters to adjust their output timestamps.

29. The architecture of claim 26, wherein the synchronization module is configured to query individual filters for a current input timestamp and a current output timestamp.

30. The architecture of claim 29, wherein the synchronization module is configured to compute adjustments by:

extrapolating at least one curve characterizing the timestamp mappings for at least one of the timestamp mappings to a defined point corresponding to a common input timestamp; and

computing said adjustments based on the extrapolated line(s).

31. The architecture of claim 30, wherein the synchronization module is configured to linearly extrapolate said at least one curve.

32. The architecture of claim 30, wherein the defined point comprises the largest value of an input timestamp returned by querying the one or more filters.

33. The architecture of claim 30, wherein the synchronization module is configured to compute adjustments by taking the difference between the largest output timestamp value at the common input timestamp and the output timestamp value for said at least one curve at the common input timestamp.

34. The architecture of claim 33, wherein the synchronization module is configured to instruct one or more filters to jump their output timestamp values by an associated skip value at the common input timestamp.

35. One or more computer-readable media having computer-readable instructions thereon which, when executed by one or more processors, cause the one or more processors to:

query one or more filters configured to process timestamped data streams for each filter'"'"'s input timestamp-to-output timestamp mapping;

receive responses from the queried filters;

extrapolate one or more lines characterizing an individual mapping to a selected input timestamp value;

calculate a skip value for one or more of the lines, a skip value representing a value by which the output timestamps for a given filter are to be corrected; and

provide instructions to one or more of the filters to adjust their output timestamps in accordance with an associated skip value.

36. The computer-readable media of claim 35, wherein the instructions cause the one or more processors to query the one or more filters for each filter'"'"'s data stream playback rate.

37. The computer-readable media of claim 35, wherein the instructions cause the one or more processors to query one or more decoders filters.

38. The computer-readable media of claim 35, wherein the instructions cause the one or more processors to linearly extrapolate the one or more lines.

39. The computer-readable media of claim 35, wherein the selected input timestamp value comprises a current input timestamp value for one of the filters.

40. The computer-readable media of claim 35, wherein the selected input timestamp value comprises a future input timestamp value.

41. One or more computer-readable media having computer-readable instructions thereon which, when executed by one or more processors, cause the one or more processors to:

define a skip value tolerance that can be used to ascertain when synchronization processing of multiple data streams should take place;

ascertain whether any input timestamp-to-output timestamp mappings associated with any of the multiple data streams meet or exceed the skip value tolerance; and

synchronize the multiple data streams if any input timestamp-to-output timestamp mappings meet or exceed the skip value tolerance.

42. The computer-readable media of claim 41, wherein the instructions cause the one or more processors to synchronize to a common data stream.

43. The computer-readable media of claim 41, wherein the instructions cause the one or more processors to synchronize to a reference clock.

44. The computer-readable media of claim 41, wherein one of the multiple data streams comprises a video stream.

45. The computer-readable media of claim 41, wherein one of the multiple data streams comprises an audio stream.

46. The computer-readable media of claim 41, wherein one of the multiple data streams comprises a video stream and another of the multiple data streams comprises a video stream.