Byzantine fault-tolerant SCADA system

• US 8,880,198 B2
• Filed: 06/05/2012
• Issued: 11/04/2014
• Est. Priority Date: 08/23/2011
• Status: Expired due to Fees

• Current Assignee: Siemens AG
• Original Assignee: Siemens AG
• Inventors: Goose, Stuart, Kirsch, Jonathan
• Primary Examiner(s): Padmanabhan, Kavita
• Assistant Examiner(s): LEE, DOUGLAS S
• Application Number:
  US13/488,977
• Publication Number:
  US 20130053986A1
• Time in Patent Office: 882 Days
• Field of Search: 700/9, 700/17, 700/18, 700/79, 700/82, 709/250, 707/661, 717/151
• US Class Current: 700/9
• CPC Class Codes: G05B 19/4183 characterised by data acquisition, e.g. workpiece identification
• CPC Class Codes: G05B 2219/32404 Scada supervisory control and data acquisition
• CPC Class Codes: G05B 2219/32406 Distributed scada
• CPC Class Codes: Y02P 90/02 Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]

Abstract

A system for automatically monitoring and controlling an infrastructure or process includes a plurality of remote clients installed along various portions of an industrial infrastructure or an infrastructure performing a process. Each of the remote clients collects data. A plurality of server replicas is in communication with the plurality of remote clients. The server replicas receive the collected data from the remote clients and process the received data. The plurality of remote clients and the plurality of server replicas communicate across an electronic network. The plurality of server replicas includes a state machine replication system that is tolerant of a failure of one or more of the server replicas.

Claims

1. A system for automatically monitoring and controlling an infrastructure or process, comprising:

a plurality of remote clients installed along various portions of an industrial infrastructure or an infrastructure performing a process, each of said remote clients collecting data pertaining to the infrastructure or process;

a plurality of server replicas in communication with said plurality of remote clients, receiving said collected data from said remote clients and processing said received data; and

an electronic network across which the plurality of remote clients and the plurality of server replicas communicate,

wherein said plurality of server replicas comprises a state machine replication system that is tolerant of a failure of one or more of said server replicas, and

wherein in the state machine replication system, each of the plurality of server replicas receive identical incoming message events and timeouts and process the incoming message events and timeouts in an identical order.

2. The system of claim 1, wherein the plurality of remote clients comprises a plurality of Remote Terminal Units (RTUs) or a plurality of Programmable Logic Controllers (PLCs).

3. The system of claim 1, wherein the plurality of server replicas comprises a plurality of Supervisory Control and Data Acquisition (SCADA) replica servers.

4. The system of claim 1, wherein said state machine replication system is tolerant of up to a predetermined number of Byzantine failures of said server replicas.

5. The system of claim 1, wherein each of the plurality of server replicas is configured to dynamically agree on an expiration of a plurality of logical timeouts.

6. The system of claim 5, wherein said agreement on the expiration of the plurality of timeouts includes exchanging a constant number of messages for any number of logical timeouts, within said plurality of timeouts, greater than or equal to one, wherein the size of said messages does not depend upon the number of logical timeouts.

7. The system of claim 1, wherein each of said plurality of server replicas is configured to send messages to one of said remote clients of said plurality of remote clients using a first protocol and is configured to receive messages from one of said remote clients of said plurality of remote clients using a second protocol that is different from said first protocol.

8. The system of claim 7, wherein said first protocol is a unidirectional logical channel for reliably sending a message from multiple sources to a single destination and for acknowledging, to one or more of the multiple sources, that the message has been successfully received.

9. The system of claim 8, wherein the first protocol is implemented via a selective repeat sliding window protocol with cumulative acknowledgments and negative acknowledgements where a server replica of the plurality of server replicas sends data messages to a client and the client sends data acknowledgments and negative acknowledgements.

10. The system of claim 8, wherein the single destination'"'"'s acknowledgements of the receipt of the messages from the replica servers are processed by the replica servers without first ordering them.

11. The system of claim 7, wherein said first protocol provides for a server replica to vote to close a connection between the server replicas and a client as a local decision and said connection is closed when a predetermined number of replicas vote to close the connection.

12. The system of claim 7, wherein said second protocol is a unidirectional logical channel in which cumulative acknowledgements are used to advance a sliding window to a particular value when said cumulative acknowledgments acknowledging at least up to said particular value are received from a sufficient number of server replicas.

13. A method for automatically monitoring and controlling an infrastructure or process, comprising:

collecting data pertaining to a function of an infrastructure or process at one or more clients;

receiving said collected data at each of a plurality of server replicas, said plurality of server replicas comprising a fault-tolerant state machine replication system; and

using said received data to monitor or control said infrastructure or process,

wherein in the fault-tolerant state machine replication system, each of the plurality of server replicas receive identical incoming message events and timeouts and process the incoming message events and timeouts in an identical order.

14. The method of claim 13, wherein the one or more clients comprises a plurality of Remote Terminal Units (RTUs) or a plurality of Programmable Logic Controllers (PLCs) and the plurality of server replicas comprises a plurality of Supervisory Control and Data Acquisition (SCADA) replica servers.

15. The method of claim 13, wherein the fault-tolerant state machine replication system is a Byzantine fault-tolerant state machine replication system.

16. The method of claim 13, wherein each of the plurality of server replicas dynamically agree on an expiration of a plurality of logical timeouts, and

wherein the agreement on the expiration of the plurality of timeouts includes exchanging a constant number of messages for any number of logical timeouts, within said plurality of timeouts, greater than or equal to one, wherein the size of said messages does not depend upon the number of logical timeouts.

17. The method of claim 13, wherein one or more of said plurality of server replicas sends messages to one of said one or more clients using a first protocol and receives messages from one of said one or more clients using a second protocol that is different from said first protocol.

18. The method of claim 17, wherein said first protocol is a unidirectional logical channel for reliably sending a message from multiple sources to a single destination and for acknowledging, to one or more of the multiple sources, that the message has been successfully received.

19. The method of claim 17, wherein said second protocol is a unidirectional logical channel in which cumulative acknowledgements are used to advance a sliding window to a particular value when said cumulative acknowledgments acknowledging at least up to said particular value are received from a sufficient number of server replicas.

20. A state machine replication system, comprising:

a plurality of server replicas collecting data pertaining to an industrial process;

at least one remote client in communication with said plurality of server replicas; and

an electronic network across which the plurality of replicas and the remote client communicate,

wherein said plurality of server replicas comprises a state machine replication system that is tolerant of a failure of one or more of said server replicas, and wherein said plurality of replicas is configured to dynamically agree on an expiration of a plurality of logical timeouts, and

wherein in the state machine replication system, each of the plurality of server replicas receive identical incoming message events and timeouts and process the incoming message events and timeouts in an identical order.

21. The state machine replication system of claim 20, wherein said agreement on the expiration of the plurality of timeouts includes exchanging a constant number of messages for any number of logical timeouts, within said plurality of timeouts, greater than or equal to one, wherein the size of said messages does not depend upon the number of logical timeouts.

22. A state machine replication system, comprising:

a plurality of server replicas collecting data pertaining to an industrial process;

at least one remote client in communication with said plurality of server replicas; and

an electronic network across which the plurality of replicas and the remote client communicate,

wherein said plurality of server replicas comprises a state machine replication system that is tolerant of a failure of one or more of said server replicas,

wherein each of said plurality of server replicas is configured to send messages to said remote client using a first protocol and is configured to receive messages said remote client using a second protocol that is different from said first protocol,

wherein said first protocol is a unidirectional logical channel for sending a message from multiple sources to a single destination and for acknowledging, to each of the multiple sources, that the message has been successfully received, and

wherein in the state machine replication system, each of the plurality of server replicas receive identical incoming message events and timeouts and process the incoming message events and timeouts in an identical order.

23. The state machine replication system of claim 22, wherein said second protocol is a unidirectional logical channel in which cumulative acknowledgements are used to advance a sliding window to a particular value when said cumulative acknowledgments acknowledging at least up to said particular value are received from a sufficient number of server replicas.

24. The state machine replication system of claim 22, wherein each of said plurality of server replicas is configured to advance a corresponding local window upon receiving a data packet with a sequence number higher than an end of said local window.

25. The state machine replication system of claim 22, wherein each of said plurality of server replicas is configured to detect when one or more of said at least one remote clients is experiencing a Byzantine failure when said server replica receives a data packet with sequence number greater than a last sequence number collectively acknowledged by the plurality of replica servers by a value at least equal to a window size of said server replica.