Method and computer program product for synchronizing the processing of multiple data streams and matching disparate processing rates using a standardized clock mechanism

• US 5,815,689 A
• Filed: 04/04/1997
• Issued: 09/29/1998
• Est. Priority Date: 04/04/1997
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Woodruff, Bryan A., Shaw, George H. J., O'Rourke, Thomas J.
• Primary Examiner(s): Heckler, Thomas M.
• Application Number:
  US08/826,560
• Time in Patent Office: 543 Days
• Field of Search: 395/551, 395/552, 345/302
• US Class Current: 713/400
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: H04N 21/4307 Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen

Abstract

A method and computer program product for synchronizing processing between two or more data streams (e.g., video and sound input) and for rate matching between two different hardware clocks that may drift with respect to one another (e.g., an originating clock represented in a timestamped data stream versus a clock actually rendering the data) in a system of interconnected software drivers running in kernel mode. The present invention overcomes the coordination complexity and inaccuracies in the prior art by providing a clocking mechanism in a system wherein multiple drivers having input and output connection pin instances are chained together. The clocking mechanism synchronizes between data streams by providing a master clock on an input pin instance of a driver that is used to synchronize with other input pin instances on other drivers and "slave" clocks. Synchronization is achieved through event notification or stream position queries so that corresponding frames of data in separate streams are rendered together (e.g., video frames with corresponding sound track). Rate matching is achieved through monitoring a physical clock progression in comparison with a series of data stream timestamps thereby allowing adjustments to match the different clock rates. A common physical clock (e.g., PC clock) can be used as a reference for a component to translate a particular clock time to a time shared by all components with a minimum of error.

Claims

1. A method for interconnecting software drivers to allow efficient kernel mode processing of data and having a standardized way of providing timing information for synchronizing different data streams or rate matching different hardware clocks, the method comprising the steps of:

opening one or more kernel mode drivers;

forming one or more connection pin instances for connecting the drivers, each connection pin instance hierarchically related to one of said one or more drivers and used for data transmission between said one or more drivers;

creating one or more clock mechanisms for rate matching and stream synchronization, each clock hierarchically related to one of said one or more connection pin instances and providing a data stream time and a physical time based on an underlying hardware oscillator; and

interconnecting said one or more connection pin instances so as to provide a continuous data flow path through the one or more drivers residing in kernel mode, said clocks making timing information available for rate matching between different clock oscillations and for synchronizing different data streams.

2. A method as recited in claim 1 wherein each connection pin instance is represented by a file object and the hierarchal relationship is created by specifying the related driver, the driver referenced as a file object of an I/O device available on the system, during connection pin instance file object creation and each clock mechanism is represented by a file object and the hierarchal relationship with a connection pin instance is created by specifying the related connection pin instance file object as a parent during the clock file object creation.

3. A method as recited in claim 1 further comprising the step of querying each of said connection pin instances on each of said one or more drivers to determine the availability of clock mechanisms in order to determine whether to instantiate a clock mechanism for synchronization or rate matching on a particular connection pin instance prior to interconnecting said one or more connection pin instances.

4. A method as recited in claim 1 wherein at least one of said one or more connection pin instances on said one or more drivers support at least one predefined set of properties, set of methods, and set of events to indicate to a third party component the availability of a clock mechanism on the at least one connection pin instance and for allowing a third party component to form said clock mechanism on respective connection pin instances and the clocks support at least one predefined set of properties, set of methods, and set of events to control said respective clocks as determined by the third party component.

5. A computer program product comprising:

a computer usable medium having computer readable program code means embodied in said medium for providing a standardized interconnection mechanism to other components in kernel mode, said computer readable program code means comprising;

program code means for forming connection pin instances, said connection pin instances used for transmitting data to or from other components in kernel mode and capable of interconnection with other connection pin instances located on other components; and

program code means for forming clock mechanisms on some of said connection pin instances, said clock mechanisms providing a positional time value that reflects a current position in a stream of data having time interval information associated therewith.

6. A computer program product as recited in claim 5 wherein the clock mechanism further provides a correlated time value that comprises a reference time value based on a common hardware oscillator available to multiple components and said positional time value in an atomic operation so as to allow other components to perform time translations using said common hardware oscillator as a reference.

7. A computer program product as recited in claim 5 wherein the clock mechanism further provides a correlated time value that comprises a reference time value based on a common hardware oscillator available to multiple components and designated time value in an atomic operation so as to allow other components to perform time translations using said common hardware oscillator as a reference.

8. A computer program product as recited in claim 5 wherein the clock mechanism further provides a physical time value that is based on a hardware oscillator.

9. A computer program product as recited in claim 8 wherein the clock mechanism further provides a correlated physical time value that comprises a reference time value based on a common hardware oscillator available to multiple components and said physical time value in an atomic operation so as to allow other components to perform time translations using said common hardware oscillator as a reference.

10. A computer program product as recited in claim 5 wherein said positional time value is accessed by other components as part of a property set implemented in said program code means.

11. A computer program product as recited in claim 5 wherein the clock mechanism further provides notifications of relevant events to other components.

12. A computer program product as recited in claim 5 wherein the clock mechanism further provides notifications based upon said positional time of relevant events to other components so as to provide useful synchronization points.

13. A method for synchronizing multiple streams of data comprising the steps of:

creating a master clock reference based on a reference data stream of data samples, each data sample having time interval information indicating position within the reference data stream, the master clock reference created from the time interval information;

processing one or more other data streams based on the master clock reference so that the other data streams are synchronized with the processing of the reference data stream.

14. A method as recited in claim 13 wherein the master clock reference is accessed as part of a property set.

15. A method as recited in claim 13 wherein the master clock reference is part of a selectively insatiable clock mechanism that is created in response to a request by a third party agent.

16. A method as recited in claim 13 wherein the time interval information is provided by timestamp information on the media samples.

17. A method as recited in claim 13 wherein the time interval information is provided media stream convention.

18. A method for rate matching a data stream processed through a processing component to a hardware processor comprising the steps of:

providing, to the processing component, a physical time reference based on the hardware oscillations of a hardware processor; and

adjusting the rate of processing at the processing component of a data stream to match the hardware processor processing rate based on the physical time reference, the data stream comprised of samples having time interval information associated therewith.

19. A method as recited in claim 18 wherein the physical time reference is accessed as part of a property set.

20. A method as recited in claim 18 wherein the physical time reference is part of a selectively insatiable clock mechanism that is created in response to a request by a third party agent.

21. A method as recited in claim 18 wherein the time interval information is provided by timestamp information on the data samples.

22. A method as recited in claim 18 wherein the time interval information is provided data stream convention.

23. A method for synchronizing multiple media streams comprising the steps of:

creating a master clock reference based on a reference media stream of media samples, each sample having timestamp information indicating position within the reference media stream, the master clock reference created from the timestamp information;

processing one or more other media streams based on the master clock reference so that the other media streams are synchronized with the processing of the reference media stream.

24. A method as recited in claim 23 wherein the master clock reference is accessed as part of a property set.

25. A method as recited in claim 23 wherein the master clock reference is part of a selectively insatiable clock mechanism that is created in response to a request by a third party agent.

26. A method for rate matching a media stream processed through a processing component to a hardware renderer comprising the steps of:

providing, to the processing component, a physical time reference based on the hardware oscillations of a hardware renderer; and

adjusting the rate of processing at the processing component of a media stream to match the hardware renderer processing rate based on the physical time reference, the media stream comprised of media samples having time interval information associated therewith.

27. A method as recited in claim 26 wherein the physical time reference is accessed as part of a property set.

28. A method as recited in claim 26 wherein the physical time reference is part of a selectively insatiable clock mechanism that is created in response to a request by a third party agent.

29. A method as recited in claim 26 wherein the time interval information is provided by timestamp information on the media samples.

30. A method as recited in claim 26 wherein the time interval information is provided media stream convention.

31. A method for translating a positional time value in a first component based on a positional time value in a second component and utilizing a common hardware oscillator comprising the steps of:

receiving, at a first component from a second component, a correlated time value in a single operation, the correlated time value comprising a positional time value based on the position within a data stream having time interval information associated therewith and a common time value based a common hardware oscillator;

querying, by the first component, the current value of the common hardware oscillator; and

translating, by the first component, the positional time value in the first component based on the current value of the common hardware oscillator and the correlated time value received from the second component.

32. A method for translating a designated time value in a first component based on a designated time value in a second component and utilizing a common hardware oscillator comprising the steps of:

receiving, at a first component from a second component, a correlated time value in a single operation, the correlated time value comprising a designated time value and a common time value based a common hardware oscillator;

querying, by the first component, the current value of the common hardware oscillator; and

translating, by the first component, the designated time value in the first component based on the current value of the common hardware oscillator and the correlated time value received from the second component.

33. A method as recited in claim 32 wherein the designated time value is a positional time value based on the position within a data stream having time interval information associated therewith.

34. A method as recited in claim 32 wherein the designated time value is a physical time value based upon a hardware oscillator.

35. A kernel mode data processing system allowing synchronous processing of two data streams comprising:

a first data source;

a first plurality of kernel mode data processing components including an originating component and a terminating component, the originating component reading data samples of a first data stream originating from the first data source, and at least one of the processing components having a master clock component associated therewith;

a second data source;

a second plurality of kernel mode data processing components including an originating component and a terminating component, the originating component reading data samples of a second data stream originating from the second data source, and at least one of the processing components synchronizing processing based on a master clock component associated with a processing component found in the first plurality of processing components; and

kernel mode component interconnections between the data processing components of respective first and second plurality of kernel mode data processing components to route the processing of respective data samples from the respective originating component to the respective terminating component.

36. A kernel mode media rendering system allowing synchronous processing of two media streams comprising:

a first media source;

a first plurality of kernel mode media processing components including an originating component and a terminating component;

the originating component reading media samples of a first media stream from the first media source;

the terminating component rendering said first media stream;

each media processing component having connection pin instances for passing first media samples between media processing components; and

at least one pin instance having a master clock component associated therewith;

a second media source;

a second plurality of kernel mode media processing components including an originating component and a terminating component;

the originating component reading media samples of a second media stream from the second media source;

the terminating component rendering said second media stream;

each media processing component having connection pin instances for passing second media samples between media processing components; and

at least one pin instance synchronizing processing of the second media stream with processing of the first media stream by using the master clock component associated with one of the pin instances of a processing component found in the first plurality of processing components; and

kernel mode component interconnections between the respective media processing components found in respective first and second plurality of kernel mode processing components created using the respective connection pin instances to route processing control of the respective media samples from the respective originating component to the respective terminating component.