Intelligent Finance upgrade downs web service

Intelligent Finance (IF) hit a system upgrade catastrophe on Sunday which left the internet banking service severely disrupted for nearly two days, much to the chagrin of its users. Thousands of people who attempted to access their account via IF's website were greeted with a static message that declared "we are currently …

[Simon Painter]

"I don't think this is any way to run a bank, being unavailable for two days. Totally hopeless."

"I don't think this is any way to run a bank, being unavailable for two days. Totally hopeless."

What, pray tell, are the weekends? Don't the majority of banks close for two days EVERY WEEK?

I do understand the whiney pathetic point of view of the customers who were unable to check their balance online for a whole two days however some slack must be cut for a company which a) has a problem with an upgrade - let's face it most of us have and b) has the stones to admit they fucked up. I suspect that your quote comes from yet another blame culture junkie who thinks he has a claim for damages.

[Dave]

The Good Old Days of Banking

Imagine the bank being closed for two whole days... Who else remembers when you had to turn up at a bank branch to do business, they opened about 9:30 or 10am and closed at 3:30 and were not available on weekends or bank holidays?

[Dillon Pyron]

Re: The Good Old Days of Banking

Yes, Dave. And it wasn't all that long ago. Heck, I remember when we (US) got two mail deliveries a day, M-F.

And what if a holiday fell on a Monday or Friday? Three days without banks, what a horror!

And ATM's. What did we do without ATM's? Other than write checks and get the money from the bank M-F, 9 to 4 (at least those were the hours when I first started using banks. But at least they had drive throughs by then).

[Daniel Ballado-Torres]

Re: The Good Old Days

"they opened about 9:30 or 10am and closed at 3:30 and were not available on weekends or bank holidays?"

That is still the case over here... some branches open on Saturdays for those who can't do their stuff on weekdays. Except now you have online banking!

Ironically, I work *at* a bank, and service availability for the online banking systems is a top priority, especially on pay-day. Many businesses have an automated process for salary payment that depend on the online services. So even 1 hour of downed service might equal *millions* lost in transactions

[David Eddleman]

Re: Simon Painter

There's a caveat here, however. Mainly, they rolled out a system without *fully testing it*: a major NO-NO in any systems development. You *always* roll it out on an isolated network for testing before implementing it on the mainstream.

[Chris Young]

Title

Caveat or Cobblers?

Who's to say the test roll out would haev exposed a problem that only became apparent under heavy load?

It needs *Users* to really screw things up :-) Lots of 'em in this case.

Where was BOFH when he was needed, I wonder.

[Pascal Monett]

Two days is a problem ?

I dare say that a two day outage on online service is most certainly NOT a problem. If you have something so urgent that 48 hours is an unacceptable delay, then visit a local branch and talk to the teller - that's what he is for.

If you are too lazy to get off your fat rear and move a thousand yards, then don't whine about two days without service. And if your retort is that the local branch is far, then my answer is that if your banking service is sooo important to you, then move closer to the local branch.

Pascal.

[Anonymous Coward]

IF only they could run a bank...

IF's online banking service has been flakey for weeks, repeated 'system errors' when trying to make payments and transfers, randomly getting kicked out and having to log in again, it never used to be that way.

The problem with IF is that they've lost their competitive advantage - every bank and his dog now offers a half-competent online banking system, and without that they're just another rip-off arm of the Halifax with uncompetitive rates and poor customer service...

[Anonymous Coward]

Totally hopeless

Bear in mind that telephone and internet services were not available until late on Tuesday - ie Wednesday in banking terms. This makes one weekend and two working days where accounts were inaccessible and charges were ramping up for anyone with say an unauthorised overdraft or credit card due for payment. It's an internet-and-telephone-only bank, so it's not as if you can pop into a branch. It's all very well and laudable that IF admitted that they screwed up, but that shouldn't detract from the fact that if internet and telephone are the SOLE ways of dealing with customers then these should be kept running at the published times.

[Jason Swain]

Testing and rollout

I'm sure it was tested, but possibly not load tested enough, or perhaps there were problems with the production architecture and the test enviroment did not fully model. It looks like this was a hardware upgrade as well as a software upgrade.

During the development of IF there was extensive performance testing done. They have a stateless server architecture with 8 or more application servers running independently (you can read about it in a book called Enterprise SOA if you are interested, there's a case study on IF).

It is possible to upgrade one server independently of the others, but only if network or database schema changes are not involved. During a software upgrade is is almost always possible to roll back the upgrade, hardware upgrades do not always have that option. I bet there are some very tired people in Edinburgh now. Probably in meetings that they'd rather not be in too.

IF's track record has been very good until now though.

[Anonymous Coward]

And they're not the only ones...

Protx - who provide "secure online credit card and debit card payment solutions for thousands of online and mail order businesses across the UK" are currently offline after an upgrade gone wrong...

http://www.protx.com/monitorvsp.asp

Meaning that their "thousands of online and mail order businesses" cannot process any payments. Whoops.

[Anonymous Coward]

Extensive performance testing????

During the development of IF there was extensive performance testing done. They have a stateless server architecture with 8 or more application servers running independently....

I'm sorry, we must be talking about a different bank. The biggest lesson to learn from IF is how NOT to organise a major IT project. Any 'extensive' performance testing at the outset obviously came to the conclusion that performance wasn't a priority - why else was a major performance enhancement project initiated in 2002. The Project Management at IF in those early halcyon days of six figure contrator salaries was diabolical, resulting in a software system that was tightly coupled and near impossible to refactor, as well as a database schema that is frankly an embarrassment. Then in 2003 a major refactoring project is initialised. The result? Nine months late delivery, twice the estimated budget, and a whole new architectural tier inserted into the system, making it more complex and slower.

And they do not have 8 application servers. There are three unix boxes running 20 odd instances of the mid-tier codebase, none of which are very well clustered.

Project management has improved at IF over the years, but they're going to run into further problems owing to their database setup. Watch this space.......