Posts by SteveMilner

Flood, fire at BT Paddington node causes widespread problems

Sage Pay Still have problems with HBOS * Cardnet

Incident Report from SagePay:

At 9:50am this morning we had problems connecting to the authorising banks.

We investigated and subsequently moved our authorisations to our DR site which was processing at 9:59

After a brief period of processing the authorisations slowed, this was a result of many people moving to backup configurations to route round an issue that transpired to be a fire at a main telecoms/datacoms exchange in docklands.

Within minutes the bottle neck had eased and we were processing transactions for all banks except HBOS and Lloyds TSB. We switched the destination NUA’s to the backups and there was still nothing being processed by those banks

We made the appropriate calls and even though our backup X25 system worked perfectly it transpires that the bureau responsible for processing HBOS and Lloyds transactions did not have resilience outside of that exchange.

We have been working with the bureau and alternative to X25 in order to resolve the issue.

As of 15:15 we now have a workaround in place, however with the X25 network working at a much reduced capacity we are experiencing delays in processing transactions

Inderect Effects

Really.

Try using your credit card online then.

S.

More Inderect Effects

Expect other card services to be impacted.

This from a SagePay communication email:

Hello,

We'd like to update you on an external issue which is effecting nationwide telecomms/datacomms.

British Telecom has confirmed a fire at their Exchange in the Paddington area of London. This has impacted a huge number of UK businesses, including Sage Pay and the major banks that route transactions through the Exchange. Sage Pay has taken action on your behalf by quickly moving transaction processing to our backup data centre. However, the issue has affected connectivity to some banks who rely on the exchange and we are currently unable to process via those banks at this time.

Please note, as banks and payment providers look for alternative transaction routing solutions away from the Paddington Exchange, this could create some general capacity issues, resulting in transaction 'time-outs'.

SagePay

This, according to SagePay website:

http://www.sagepay.com/system_monitor.asp

31 March 2010 12:02:45 - Please be advised Lloyds Cardnet and Halifax Bank of Scotland customers are currently unable to process transactions via the Sage Pay gateway.

This is due to a BT communications network failure, and not an issue on the processing platforms.

We are currently awaiting an update from the respective parties.