Background:
A large electricity transmission utility in Australia operates over 100 high voltage substations throughout the state. These substations monitor and control the electricity flowing in the high voltage transmission lines, carrying power throughout the state.
The substations contain a lot of technical equipment, which performs tasks such as:
- converting the voltage up or down
- protecting the network against fault conditions
- measuring the electrical current and voltage (and other parameters)
- switching the power between the different transmission lines and transformers that they connect to
- ancillary tasks such as weather monitoring, video surveillance, battery backup, etc.
To keep control of all this equipment and allow remote access to the whole network of substations, sophisticated computer systems are required in each substation. These substation industrial servers run software that controls and remotely monitors the equipment in the electricity transmission network.
Since 2012 ESIS has been engaged via the electricity utility’s IT provider, to design and build industrial servers to meet the stringent needs of the utility.
Project Requirements
The substation industrial servers needed to fulfil a very specific set of requirements:
- Maximum reliability for industrial applications
- 4U Rack mount chassis with short rack depth limited to 420mm
- Powered by 48V DC
- Intel Server Motherboard with VMWare ESXi certification
- Xeon E3 CPU
- 32GB ECC RAM
- Server grade hardware RAID card (VMWare certified)
- 3x server grade 2TB hard drives in hot swappable bays, configured as RAID 5
- 4-6 extra Gigabit Ethernet ports via server grade network cards (VMWare certified)
- Full IPMI (lights out) remote server management support via dedicated network port
And later in the project, another requirement was added:
- Long product lifecycle for ongoing procurement without significant changes
This combination of requirements is quite unusual, and it is not possible to find an off the shelf server to meet all these needs. Most standard rackmount cases do not meet the depth constraint while also providing hot swap drive bays, plus the 48VDC power requirement adds to the difficulty. Hence it is necessary to carefully design and build a custom industrial server.
Industrial Server Solution
Given the unusual combination of requirements, ESIS engineers came up with a unique server design combining components from several different manufacturers.
Components used:
- APGHI-404 rack mount server chassis, with a depth of 408mm
- 600W 48VDC server grade power supply from a reputed industrial grade manufacturer
- Intel serverboard, Xeon E3 CPU and 32GB ECC RAM as requested by the electricity utility * See notes below about how the story of this board evolved over the following years!
- IPMI (or integrated lights out) module with dedicated LAN port, to provide network-based “hardware” remote control and management of the PC
- LSI 4-port hardware RAID card
- Western Digital enterprise grade hard drives
- Enterprise grade hot swap drive caddy, 3 bay
- Intel server grade multi-port network cards
- Extra cooling fans to increase airflow and specifically target the RAID card which runs hot
This system was built, tested and verified by Esis. The system met all the requirements, was compatible with the substation software, and worked reliably. Since 2012 these systems have been supplied to the customer on an ongoing basis, to fit out the substations.
Lifecycle Issues
Two years after Esis started supplying this server, the Intel motherboard that the utility had selected, was discontinued by Intel, superseded by the next generation product. So Esis sourced an equivalent motherboard from Supermicro, with the same chipset and specifications. This satisfied the requirements and allowed production to continue for the time being.
However some months later the original Xeon CPU was also discontinued. This presented a problem for the electricity utility: changing to a newer motherboard & CPU platform would require their entire documentation and software testing to be done again, which would take significant time and resources. Hence, in 2015 the customer approached Esis to find a better solution: this time around, with a much longer lifecycle.
New long-lifecycle industrial server
For this new version of the substation server, the requirements specification was updated by the customer, and Esis engineers researched the market again. This time Esis selected an industrial grade server motherboard made by Advantech, along with an “embedded” Intel Xeon CPU with a long lifecycle. The RAID and network cards were also updated to the latest models, as the older models were likely to be superseded.
The new platform, based on an Advantech industrial server motherboard and Intel Xeon E3V3 CPU, meets all the requirements, including VMWare certification. The server was tested by the utility and it worked well with their software. Some of the early servers encountered installation issues with VMWare. These issues were diagnosed by Esis and Advantech to be due to an old BIOS version. Hence all the systems’ BIOS were upgraded to the newer version, and the problems were solved.
The new motherboard and CPU have a long product lifecycle of at least 5 to 7 years. This system, specifically developed for this electricity utility, has now been in production by Esis for the past 3 years and is being successfully used to upgrade substations throughout the state.
2018 and beyond
As of 2018, the electricity utility is also starting to plan logistics for the end-of-life of this motherboard and CPU, which could be in the next 2 years. This underlines the importance of having a stable platform with a long product lifecycle: the electricity utility wants to keep rolling out the exact same platform throughout the substation upgrade program, which runs over many years.
In this kind of industrial application, it is much better to use an existing, tried and tested platform, for more than 5 years — rather than to have to design, build, test, document and troubleshoot a new platform every 2 years which is the life cycle of standard IT equipment.
Hence, when designing IT related equipment for industrial use cases like electricity networks, it is important to always consider the lifecycle of the products, and design a system that not only meets the technical requirements, but also maximises its availability over time.
Do you need an industrial PC or server built to meet your needs? Contact us today for a quote.
