Friday, December 21, 2012

Pay Now Or Later! Life Cycle Cost of Automation System neglected

People responsible for huge automation projects can focus on the cost for installing and commissioning a system OR on the cost for operating and maintaining a system. A reasonable approach would be to figure out what the System Life Cycle Cost are likely.

Yesterday I read in a technical magazine about a very bad example of focusing first on minimum costs for installing and commissioning and neglecting System Life Cycle Cost. In this case the whole automation system is completely refurbished a few years after the system was put in operation. The refurbishment has cost some 15.000.000 Euro. Unbelievable.

The 2 times 34.6 km “Lötschberg Alpine Base Tunnel” (Switzerland) was build between 1999 and 2007. The project’s cost were some 4.000.000.000 Euro. The crucial priority for building the tunnel was meeting the calculated costs and deadlines for opening the tunnel for operation. The project was finished on time and the costs were in the limits set! Perfect! … compared to many huge projects …

But! A running system has to run for years! Very often little efforts are spent to assure that the system remains “clean” and maintainable and expandable even after many years.

In the case of the Lötschberg Tunnel the operation costs were far to high due to the fact that there were very little efforts made during planning and engineering phase to allow a smooth information flow between the many devices and systems. There were many islands of information.

The technical infrastructures had been tendered and realized as separate systems for: Fire protection, Ventilation/ A/C, Lighting control, Escape and evacuation, Cross tube doors, Power supply, Water supply, … Many gateways, protocol inverters, and and had to be installed to let components communicate and share information. The 24*7 operation of the tunnel required personnel on site to run behind alarms: during the first year of operation the many systems produced between 1.000 and 5.000 Alarms per day (!!!). Even after some improvement two experts had to process some 30 alarms per day … causing operating costs (including the people to look after the alarms) of some 4.000.000 Euro per year!

This was far to high!

What to do now? It was decided soon (in 2009) to refurbish the complete automation and SCADA system build mainly by a SINGLE vendor’s solution. Cost for refurbishment: some 15.000.000 Euro. The new system is scheduled to take over the control of the tunnel mid 2013. The operation and maintenance cost are expected be reduced from 4.000.000 Euro to 1.500.000 Euro per year. WOW!

It was reported also that due to the overtime of the service and maintenance personnel many of these people left the company. I guess they were frustrated … or?

Don’t focus on message encoding of one or the other protocol. Always focus on the SYSTEM and Life Cycle Cost.

Missing capabilities to smoothly share information for the some 100.000 signals of the tunnel system have let the costs of operating and maintaining the system sky-rocketed to 4.000.000 Euro per year!

IEC 61850 is intended to provide a smooth and secure information sharing solution – independent of a SINGLE small, medium or big vendor!

Lesson learned: Open (vendor independent) information sharing could have a crucial impact of the Life Cycle Cost.

Do you care about Life Cycle Cost? Yes!? If the answer is Yes, then IEC 60870-5-104, IEC 61850, IEC 61400-25 and DNP3 are recommended options for the future needs of energy system information sharing.

Access a report from the main contractor (vendor) of the refurbished system for the Loetschberg tunnel project [pdf, en].

Another report published in Dec 2012 [pdf, de]

1 comment:

Rodney Hughes said...

Let me get this right…
They spent 4Billion to build the tunnels
2 years after completion they spent a further 15Million to replace the SCADA and Controls with a single vendor system.
Is this really a sensible solution?
The tunnel controls now rest with the longevity of a single vendor’s support.
The Tunnel will last for tens of decades.
Will this single vendor still be there always? They might go broke.
Will the single vendor always have the functions available that are needed as new controls and functions are required in the future for enhanced security and safety
Has the single vendor solution delivered the optimum performance of each function or just because it all has to appear on one screen for the operator are there deliberate or as yet to be discovered compromises in functional performance?
Is the system duplicated with common failure mode scenarios?
What happens in 5-10 years’ time when equipment from the single vendor needs to be replaced at end of life cycle – is it available as exact box replacement or will there be a requirement to interface their newer boxes to the old system with additional re-engineering of what was working “perfectly well” just a few days earlier
Single vendor scares me a little …