Synchronized fault tolerant clocks for multiprocessor systems

• US 5,146,585 A
• Filed: 11/05/1990
• Issued: 09/08/1992
• Est. Priority Date: 10/25/1988
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Smith, Thomas B. III
• Primary Examiner(s): Fleming, Michael R.
• Assistant Examiner(s): Ray, Gopal C.
• Application Number:
  US07/610,336
• Time in Patent Office: 673 Days
• Field of Search: 364/200 MS File, 364/900 MS File, 371/61, 307/269, 375/107, 375/108, 395/550, 395/650, 370/100.1, 340/825.14
• US Class Current: 713/400
• CPC Class Codes: G06F 1/14 Time supervision arrangements, e.g. real time clock
• CPC Class Codes: G06F 11/1604 where the fault affects the clock signals of a processing unit and the redundancy is at or within the level of clock signal generation hardware
• CPC Class Codes: G06F 11/20 using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements

Abstract

A system for providing fault-tolerant synchronized operation of the Time of Day (TOD) clocks of the respective data processors in a multiprocessor complex. Basically, the system is comprised of a duplex implementation having redundant TOD clock sources, and a plurality of TOD slaves which provide the TOD clocks in the associated processors. A register/counter in each TOD clock source is incremented by a high frequency signal to achieve the required TOD value resolution, and the latter signal is divided down to provide a lower reference frequency signal for synchronization of the clock sources. Each TOD slave includes terminals for receiving a pair of reference frequency signals and for trouble-free switching between the signals, as required. Alternatively, a quad implementation of clock sources which is substantially free of single points of failure of the synchronization mechanism is described. Frequency steering of the clock sources provides increased accuracy and conformity to real time when desired.

Claims

1. A fault-tolerant synchronization system for TOD clocks in a multiprocessor complex comprising:

at least a pair of time of day (TOD) clock sources, each of said clock sources including;

means for generating an internal clock signal having a nominal frequency,

TOD register/counter means for holding a data value and being responsive to said internal clock signal for incrementing the data value held thereby at a rate determined by said internal clock signal and including frequency divider means responsive to the internal clock signal for providing a periodic reference signal having a selected period, wherein the data value held by said register/counter represents a TOD clock value,

means coupled to said register/counter means for transmitting said reference signal to the other of said TOD clock sources,

means for receiving said reference signal from said other of said clock sources, and

means, responsive to said received reference signal for synchronization said clock source to the other of said clock sources, to within a predetermined skew, wherein the period of said reference signal is selected to be greater than said predetermined skew.

2. A fault-tolerant synchronization system as defined in claim 1 wherein the means for synchronizing said clock source to the other of said clock sources includes:

means in each of said TOD clock sources for comparing the respective phases of the transmitted and received reference signals and for generating an error signal having a value proportional to any phase misalignment represented by a difference between the respective transmitted and received reference signals, and

means for applying said error signal to said generating means to change the frequency of the internal clock signal and of the reference signal derived therefrom in a sense which tends to cause said error signal to approach zero.

3. A fault-tolerant synchronization system as defined in claim 2 further characterized in that the nominal frequency of said internal clock signal is chosen to realize a preselected resolution of the TOD clock value stored in said register/counter;

wherein said preselected resolution is less than said skew.

4. A fault-tolerant synchronization system as defined in claim 1 further including;

a plurality of TOD slave clock signal generators associated respectively with the processors of said multiprocessor complex,

means coupled to said pair of TOD clock sources for transmitting said respective reference signals to each of said TOD slave clock signal generators,

each of said TOD slave clock signal generators having a pair of decoder/receivers for receiving respectively said reference frequency signals from said pair of TOD clock sources,

means for selecting the reference signal provided by one of said pair of TOD clock sources, and

means responsive to the selected reference signal for providing a TOD clock value for use by the processor associated with the TOD slave clock signal generator.

5. A fault-tolerant synchronization system as defined in claim 4 wherein:

said means for transmitting said reference signal in each of said pair of TOD clock sources includes means for transmitting a value, provided by the register counter means, representing the TOD clock value to each of said TOD slave clock signal generators, and

said pair of TOD slave clock signal generators are ordinarily numbered first and second, said first and second TOD slave clock signal generators including first and second data registers coupled respectively to the decoder/receivers of said first and second TOD slave clock signal generators for storing the value representing the TOD value transmitted by the selected one of said pair of TOD clock sources.

6. A fault-tolerant synchronization system as defined in claim 5 wherein said means for providing a TOD clock value for use by its associated processor includes:

an oscillator disposed in a phase-locked loop, and

slave TOD register/counter means coupled to said oscillator for holding a value representing said TOD clock value, said oscillator incrementing the value held by said slave TOD register/counter means at a predetermined rate,

wherein said signal select means is coupled to said oscillator to drive said oscillator with said selected one of said pair of reference signals transmitted by said pair of TOD clock sources to said decoder/receiver of each of said TOD slave clock signal generators.

7. A fault-tolerant synchronization system as defined in claim 6 characterized in that the processor associated with a given one of said TOD slave clock generators effects initialization of the TOD register/counter means of the one TOD slave clock generator with the received TOD value stored in the data register corresponding to the selected reference signal, said processor reading said received TOD value into said TOD register/counter means of said one TOD slave clock generator and inhibiting the incrementation of the value held thereby until the occurrence of a syncpoint signal from the TOD clock signal generator corresponding to said selected reference signal.

8. A fault-tolerant synchronization system as defined in claim 4 characterized in that said means for selecting one of said reference signals includes,

error detection means operatively coupled to both of said reference signals for detecting inconsistent and erroneous states of said reference signals, and

signal select means coupled to said error detection means and responsive thereto for selectively coupling said means for providing said TOD clock, to one of said reference signals,

wherein the detection of a failure in the reference signal coupled to one of said TOD slave clock signal generators at any given time by said error detection means causes said signal select means to switch said means for providing said TOD clock to the other of said reference signals, thereby preventing loss of synchronization in said multiprocessor complex.

9. A clock signal synchronization system including frequency steering means comprising:

precise time reference means, for producing a precise oscillatory signal,

first and second clock signal generating means for generating respective first and second clock signals, said first and second clock signal generating means including means coupled to said second and first clock signal generating means, respectively, for changing the respective first and second clock signals in phase in response to respective changes in phase of the respective second and first clock signals to maintain mutual phase coherence between the respective first and second clock signals provided thereby, and

clock signal adjusting means, coupled to said precise oscillatory signal and to one of said first and second clock signal generating means, for adjusting the clock signal provided thereby to be in substantial phase coherence with said precise oscillatory signal, using a magnitude limited control signal, without disturbing the phase coherence between said first and second clock signals.

10. A clock signal synchronization system as defined in claim 9 wherein:

each of said first and second clock signal generating means includes means for generating an internal phase error signal representing a difference in phase between the respective first and second clock signals, the respective phase error signals of said first and second clock signal generating means being internally coupled to said first and second clock signal generating means, respectively, to maintain mutual phase coherence between the respective first and second clock signals, and

said clock signal adjusting means includes;

compute trim correction means having a pair of input terminals and an output terminal,

means for coupling said precise oscillatory signal and clock signal provided by the one of said first and second clock generating means to said pair of input terminals of said compute trim correction means, said compute trim correction means comparing the signals coupled to its input terminals and generating a steering signal on its output terminal proportional to the difference in the relative phases of said signals coupled to its input terminals,

sum means having a pair of input terminals and an output terminal, and

authority filter means interposed between the output terminal of said compute trim correction means and one of said pair of input terminals of said sum means, said authority filter means limiting said steering signal in magnitude so as to preclude a break in the phase coherence between said first and second clock signals,

wherein the internal phase error signal from the one of said first and second TOD clock sources is applied to the other input terminal of said sum means, and

the output terminal of said sum means is coupled to the one of said first and second clock signal generating means to adjust the frequency of the clock signal provided thereby to tend to conform in frequency to the precise oscillatory signal.

11. A fault-tolerant synchronization system for time of day (TOD) clocks in a multiprocessor complex comprising:

first and second pairs of TOD clock sources disposed in a quad configuration which is substantially immune to all single point failures, each of said TOD clock sources including,

means disposed in a phase-locked loop for generating an internal clock signal having a predetermined frequency,

TOD register/counter means for holding a data value and being responsive to said internal clock signal for incrementing the data value held thereby at a rate determined by said internal clock signal and including frequency divider means responsive to the internal clock signal for providing a periodic reference signal having a selected period, wherein said data value held by said register/counter represents a TOD clock value,

means coupled to said register/counter means for transmitting said reference signal to each of the other three TOD clock sources, and

means for receiving reference signals respectively from said other three clock sources,

means for applying the received reference signals to said generating means whereby the respective phases of said received reference signals are compared with the phase of said transmitted reference signal to generate three respective phase difference signals,

voting means for selecting one of the three phase difference signals, and

means for applying said selected phase difference signal to said generating means to bring said transmitted reference signal substantially into alignment with said transmitted reference signal.

12. A fault-tolerant synchronization system as defined in claim 11 including:

a plurality of TOD slave clock signal generators associated respectively with the processors of said multiprocessor complex,

a selected pair of said first and second pairs of TOD clock sources transmitting respective reference signals to each of said TOD slave clock signal generators,

each of the TOD slave clock signal generators including means for receiving the reference signals from said selected pair of TOD clock sources and means for selecting one of said received reference signals, and

each of the TOD slave clock signal generators further including means responsive to the selected reference signal for providing a TOD clock signal for use by its associated processor.

13. A fault-tolerant synchronization system as defined in claim 12 characterized in that said means for selecting one of said reference signals includes,

error detection means operatively coupled to both of said reference signals from said selected pair of TOD clock sources for determining the status thereof, and

signal select means coupled to said error detection means and responsive thereto for selectively coupling said means for providing said TOD clock to one of said reference signals,

wherein the detection of a failure in the reference signal coupled to a TOD slave at any given time by said error detection means causes said signal select means to switch said means for providing said TOD clock to the other of said reference signals, thereby preventing loss of synchronization in said multiprocessor complex.

14. A fault-tolerant synchronization system for TOD clocks in a multiprocessor complex comprising:

at least a pair of time of day (TOD) clock sources, each of said clock sources including;

means for generating an internal clock signal having a nominal frequency,

TOD register/counter means for holding a data value and being responsive to said internal clock signal for incrementing the data value held thereby at a rate determined by said internal clock signal and including frequency divider means responsive to the internal clock signal for providing a periodic reference signal having a selected period, wherein the data value held by said register/counter represents a TOD clock value,

means for generating a syncpoint signal indicating an instant in time at which a synchronization event is to occur,

means coupled to said register/counter means for transmitting said reference signal, said syncpoint signal and a value representing the TOD clock value to the other of said TOD clock sources,

means for receiving said reference signal from said other of said clock sources, and

means, responsive to said received reference signal for synchronizing said clock source to the other of said clock sources, to within a predetermined skew, wherein the period of said reference signal is selected to be greater than said predetermined skew.

15. A fault-tolerant synchronization system as defined in claim 14 wherein:

said means for transmitting said reference signal, said syncpoint signal and the value representing said TOD clock value includes encoder/transmitter means for modulating an oscillatory signal to obtain a signal conveying said reference signal, the value representing said TOD clock value and said syncpoint signal, and

said means for receiving said reference signal, said syncpoint signal and the value representing said TOD clock value includes;

decoder/receiver means for demodulating the modulated oscillatory signal provided by said encoder/transmitter means, and

a holding register coupled to said decoder/receiver means for storing said received TOD clock value.

16. A fault-tolerant synchronization system as defined in claim 15 wherein:

said pair of TOD clock sources are ordinarily numbered first and second, and said fault-tolerant synchronization system further includes first and second control logic means coupled to the respective register/counter means in each of said first and second TOD clock sources for effecting the initialization thereof,

the TOD values transmitted by said first and second TOD clock sources represent TOD clock values at the next occurrence of the syncpoint signal, and

said first control logic means includes means for changing the value held in said first register/counter means to be said received TOD clock value and means for holding said received TOD clock value in said first register/counter means by inhibiting the incrementing of the value held by said first register/counter means until the syncpoint signal is received from the second TOD clock source.

17. A fault-tolerant synchronization system as defined in claim 16 wherein the respective TOD clock value data held by the respective register/counters of said first and second TOD clock sources are facsimiles of a standard format TOD clock value utilized by the processors in said multiprocessor complex.

18. A fault-tolerant synchronization system as defined in claim 17 characterized in that said generating means comprises a voltage controlled oscillator disposed in a phase-locked loop.

19. A fault-tolerant synchronization system as defined in claim 18 further including error detection means for monitoring the transmitted and received reference signals in one of said first and second TOD clock sources to detect inconsistent and erroneous states of said transmitted and received reference signals.

20. A fault-tolerant synchronization system as defined in claim 19 wherein:

said error detection means further includes means for providing various status flags representing the states of the transmitted and received reference frequency signals, and

said encoder/transmitter means includes means for encoding and transmitting said flags with the syncpoint signal and the value representing said TOD clock value from said one of said first and second TOD clock sources.

21. Apparatus for synchronizing two TOD clocks of predetermined clock resolution when the maximum skew between said TOD clocks is greater than said clock resolution, comprising:

means responsive to one of said TOD clocks for generating a periodic reference signal synchronized with said one TOD clock, said reference signal having a period greater than said maximum skew;

means for transmitting said reference signal from said one TOD clock to the other of said TOD clocks; and

means at said other TOD clock for receiving said reference signal and for phase locking said other TOD clock to said reference signal,

whereby said two TOD clocks are synchronized with each other but offset from each other by said skew.