* Posts by Russtavo

2 publicly visible posts • joined 12 Dec 2017

Pickaxe chops cable, KOs UKFast data centre

Russtavo

Update from UKFast

Please find below an update on yesterday's issue from our Critical Power Director, explaining our return to mains power:

We have been running successfully on our back up generators since 10.48am 12th December 2017.

We have 12,250 litres of fuel which equates to a run time of 49 hours, with tankers on standby who are able to deliver with immediate effect.

At this time we await confirmation of a time to re-energise the power network to MaNOC 5,6 & 7. The cable has been fixed by Electricity North West. We understand this will more than likely be later today.

To return to mains power we will be doing this in a controlled manner. The UKFast data centre is currently locked onto generator power so when ENW switch the power back on we will prove it’s working perfectly before starting the process to switch over back to mains. This mitigates any risk of the power coming on intermittently or incorrectly.

Once ENW confirm to UKFast the electric supply is energised we will then check the electricity supply is present at our transformer and check the electricity for phase rotation to ensure the supply is electrically correct. Even though ENW should have proved this themselves we will double check it before proceeding to the next step.

Once we are happy that the supply is stable and correct we will activate our automatic power change-over system. The automatic change-over system will also monitor the power for 5 minutes and then initiate the automated return to mains power. This is called a proving period.

At this point the generator's bus-coupler main breaker will open, removing generator power from the system. The UPS which is constantly in operation will automatically support the technical load for around 10 seconds while the mains electricity power circuit breaker is closed and reconnected to the power systems.

The UPS battery system also supports the mains change-over for a further 2 minutes as the UPS slowly transfers from battery power to mains power. This is called a "walk in" procedure and removes the risk of the UPS seeing huge power demands and creates a smooth transition. There is no break in power to the technical load during the walk in period.

During the walk in period the air-conditioning will stop for a moment and restart and also perform a walk in procedure to ensure no stress is placed on the power network. This takes around 1-2 minutes as the CRAC units stop and restart.

During this period the generators continue to run should they be required. Once the transfer is complete the generators run on for a further 3 minutes then shut down and go back into standby mode.

Return to mains power is then complete should there be any issue during the transfer period we can automatically switch back to generator power.

On site managing the process we have the UKFast Electrical Team and Ingram Generator Service Partners.

It's not uncommon to lose power in the UK. In the last 12 months we have performed this exercise twice after losing power to the area. Both times the systems have switched over perfectly. This is one of the reasons we run an N 1 environment.

We prove the start signal on a weekly basis which fires up the generators and the UPS tests itself every day at 8am. We are the only data centre that we are aware of to hold the NICEIC accreditation meaning we are a fully licensed electrical contractor and can manage and maintain our data centres without the need for external contractors.

Unfortunately we do not have an exact time when the supply will be reconnected, however we are on standby and working closely with ENW who indicate it will probably be late afternoon or early evening.

Yours sincerely

Miles Allen Critical Power Director

Russtavo

Just got this email from ukfast

Please find below an interim report following the power issue affecting our MaNOC5 facility data centres on Tuesday 12th December 2017.

Please be aware that this is an interim report from the information we currently have available, we are waiting on further information from our Generator suppliers which will add to the final report when it's available.

At 10:28 GMT on Tuesday 12th December 2017, our MANOC 5, 6 and 7 facilities were impacted by an instability on the incoming mains power as a result of a civil contractor passing a spike through the main feed. This was not work being carried out on behalf of UKFast or within our site but at another location 0.75km away on the path to our onsite transformer.

The UPS system supported the load for its designed time and the generators started; however, due to the physical damage to the power cable, service to the site was unstable and intermittent. As a result, the generator set failed to synchronise and take over service.

UKFast engineers on site were alerted to the generators being unable to take over the load , and that manual intervention was required. During this time the UPS batteries depleted past the designed runtime resulting in total power loss to equipment on site at 10:40 GMT.

The manual synchronisation was completed at 10:48 GMT and the onsite generator set took over power, enabling us to start bringing power back on for client services.

Our engineers have worked throughout the day to restore individual services and continue to do so for those clients who remain affected by the power issue. The process of bringing back all services in a facility requires more than just powering on equipment and coupled with the resulting failures in physical devices that have required replacement, means this is a lengthy process.

Once we have resumed full service, we will be investigating what we can do to prevent this from happening again for both the incoming power issue and also the time taken to restore service for some of our clien ts.

We will update this report as we get more information from our generator supplier and also from our teams to discuss the delays in resolving this issue.

Kind regards,

Charlotte

Charlotte Bentley-Crane

Account Management Director

UKFast