Today we had a perfect example of distributed control working.
The way our DCS is set up is with individual plants on their own local networks and these are connected to a central network that runs to the control room where the operators and the advanced control schemes that work between the plans sit. Wherever possible, the control is setup as close to the plant as possible. Ideally, all components of a control loop (Instrument, Controller and Output) will be configured on the one device. When this is not possible, the control loops should transfer the data across their own network. The last option is to transfer the data over the central network. Everything in this setup is redundant. There are two cables for each network, two controller boxes, two power supplies for everything.
In today’s case, a power failure managed to take out both the interface boxes that connected one of the plant networks to the central network. It should have only taken one out, but for some reason the power supplies were not as independent as they were supposed to be. As a result, the plant network was completely severed from the central network. The operator lost all visibility and control of his plant and was flooded with a bunch of system alarms.
But the plant did continue to operate, to the last known position. If a controller was in auto, attempting to maintain a setpoint, it would continue to operate as before. After a few minutes, one of the network interface boxes was restarted and all the data returned. Most of the alarms cleared. The plant was still operating without issue.
Obviously there is lots to investigate here, mainly why did we lose both interfaces? but the fundamental philosophy of the design has been proven. It was situations like this that the designers of the original system had envisioned. Failure of both power supplies should not have happened, but the robust nature of the design meant it was only a minor event, a loss of view and advanced control, rather complete loss of control and uncontrolled shutdown.
Just because something shouldn’t happen, it is still useful mitigate it through robust designs.Go Top