Wells Fargo? Well fscked at the moment: Data center up in smoke, bank website, app down

Wells Fargo customers have been unable to access their online bank accounts for more than seven hours today – after smoke knackered one of its data centers. Starting around 6am Pacific Time (2pm UTC), the American bank's online portal and its mobile application have been totally unusable. There are also reports of cards being …

[Eric Olson]

This isn't the first time...

I think there was a nationwide ATM outage in Feb and Apr of 2011, also caused by a single point-of-failure. Speculation was a bad software rollout, but I haven't been able to find anything more than the bank saying it was an "internal system."

But hey... it's not like being able to query a bank account is important for customers, right?

[TheVogon]

Re: This isn't the first time...

Probably because US datacentres are seemingly run on a shoe string and have sprinkler systems as the cheapest form of fire suppression. So they have to kill to power if a fire is detected. Hence why no one uses sprinklers in DCs on most of the planet.

[TheVogon]

Re: This isn't the first time...

Did Wells Fargo by any chance do a TSB and outsourced support to a third world country?

[Anonymous Coward]

Yep, just checked my accounts, and I am kind of borked....

I can see my balances, but I can't move anything between accounts. Fortunately, I don't have any checks clearing at the moment.

Cue the jokes about how "I can't see the difference in Wells Fargo's usual customer service!", but actually they have been pretty rock solid through the many years they have held my money, except that they really pissed me off with their F'd up online bill paying service when the rolled it out years ago.

I have to do some grocery shopping, paid for via my ATM card, in a few hours. We'll see if I have to pay the store with another card.

[arctic_haze]

Re: Yep, just checked my accounts, and I am kind of borked....

I used to be a Wells-Fargo client long time ago and I can confirm that their website was rock solid and generally much more user friendly than the websites f banks I now use. Which does not change the fact that all company (banks / utilities etc.) website become more and more cluttered and unusable with time.

[Oengus]

Job opening

I see the position of Business Continuity Manager coming available soon.

People wonder why I always have cash on me. With events like this happening more and more often or even when recently I was in a shopping centre that had a power failure earlier but their EFTPOS systems were off-line I was still able to purchase my weekly food (or more importantly booze)... The queues at the checkout were non-existent because they couldn't process card payments.

[Fatman]

Re: Job opening

<quote>I see the position of Business Continuity Manager coming available soon.</quote>

You can probably add Executive VP of Operations to that list. (One would expect that PHB to supervise the BCM.)

Icon, because it is appropriate.

I wonder, did they hire anyone from the UK banking industry???

[Mr Benny]

Re: Job opening

I doubt it. Management are masters in pointing the finger downwards. Most likely some poor grunts who were simply following moronic orders will be told to clear their desks while the actual MBA gimps responsible for the mess will get a bonus for "solving" the problem they created.

[Phil Endecott]

“It is baffling that a single incident has set off such a chain reaction that it knocked out the entire internet-facing infrastructure of the bank.”

To be read with a sarcastic tone of voice, right?

[Anonymous Coward]

Wells Fargone

^ Nobody thought of this?

[LateAgain]

Smoke detection should just be a warning

Every engineer knows that it's when the equipment runs out of smoke that it stops working.

[Mike 16]

Re: Smoke detection should just be a warning

They should have had spares on hand:

http://ibm-1401.info/#NewWorkR

( In the toolbox next to the left-handed monkey wrench and the snipe-nets)

[jake]

Re: Smoke detection should just be a warning

You must be the one who stole my metric die grinder.

[ma1010]

The BOFH Strikes Again

We're told a fire suppression system was activated...When the fire department got there, though, they found no evidence of a blaze.

Perhaps their BOFH used the Halon system to get rid of some annoying users or managers? Bad on allowing a total shutdown and cascade, though. But I'm sure service will be restored after some quality pub time.

[swschrad]

Re: The BOFH Strikes Again

Halon has not been manufactured for years, greenhouse gas, you know. Nitrogen and CO2 are in vogue. all of which will still have to be ventilated before the pocket-protector brigade can go in and try to bring the center up one power cord at a time.

[Oengus]

Re: The BOFH Strikes Again

Halon may not have been manufactured for years but there was no requirement to remove or replace the installed systems. Also there are other systems using other "not nice" gasses that replace Halon. While they are supposed to be "safe" I am not sticking around if they go off; also the force that they come out with can send loose objects flying around the room.

[phuzz]

Re: The BOFH Strikes Again

I had to go in for 'health and safety training' at one of our data centres, and while talking us through what to do if the fire alarm goes off (turns out you should leave the building, good thing they told me that eh?), they told us that while you'd survive being in the room when the fire suppression went off (it uses some fancy new, non-deadly, chemical), you'd probably have both your ear drums ruptured by the air pressure suddenly increasing.

[Anonymous Coward]

Re: The BOFH Strikes Again

normally you have a button to delay release of the gas if you're in the DC and its set off

[Amos1]

Re: The BOFH Strikes Again

We had an entire data center "go quiet" once for a semi-related reason. The EPO button (emergency power off or Big Red Button) can be wired one of two ways, normally closed or normally open.

Normally closed is similar to a light switch that is always on and flipping the switch kills the power. That's how this one was wired. Even though everything had preventive maintenance twice a year, "everything" did not include that 20 year-old push button and its wiring screws slowly loosened up.

Then one day someone came into the data center and when the door closed behind them, the door next to the Big Red Button, it got very quiet.

The electricians said someone had pushed the button and the person who walked in was worried he was going to lose his job, But the security cameras showed he was nowhere near it.

The EPO button is now wired as normally open...

[TheVogon]

Re: The BOFH Strikes Again

A large investment bank I worked at had that happen a number of years ago by a Nigerian security guard That apparently could not read English. Pressed the EPO button next to the door.

[phuzz]

Re: The BOFH Strikes Again

The 'delay release' button is right next to the exit door, why stand there pressing the button when you can just leave? (unless you open the door and the corridor is on fire I suppose)

[Mr Benny]

Re: The BOFH Strikes Again

You wouldn't survive, you'd suffocate. All these system displace oxygen which is the whole point of them.

[jake]

Re: The BOFH Strikes Again

Ah yes, the "displaces oxygen so you die of suffocation" myth. I suggest you eyeball a Halon 1301 fact sheet, Mr Benny. It's actually quite safe for humans ... at the concentrations used for fire suppression, no special precautions are required for evacuation.

[Mr Benny]

Re: The BOFH Strikes Again

It is? I'd suggest you find a better fact sheet pal, perhaps one thats not written by the manufacturers.

https://en.wikipedia.org/wiki/Bromotrifluoromethane

"Human exposure to Halon 1301 can be toxic, affecting the central nervous system and other bodily functions."

"It is considered good practice to avoid all unnecessary exposure to Halon 1301"

[jake]

Re: The BOFH Strikes Again

Oh, goody. Wiki. The thing that anybody with an agenda can edit, and anybody can quote without understanding what they are quoting, such as how many ppm it takes to cause the effects reported.

A fact sheet not written by the manufacturers? How about the National Institute of Health?

https://toxnet.nlm.nih.gov/cgi-bin/sis/search2/f?./temp/~KAhpfw:1:human

tl;dr version: "Halon 1301 is a medically safe agent for use in fire fighting in confined spaces.[Harrison JN et al; J Soc Occup Med 32 (1): 37-43 (1982)"

--or--

https://www.ncbi.nlm.nih.gov/pubmed/4072911 which describes "An accidental discharge of a Halon 1301 total flooding fire extinguishing system." Seems that in this case, no significant illness or injury due to the Halon exposure was found.

I wouldn't spec it if it were dangerous.

[Mr Benny]

Re: The BOFH Strikes Again

"The thing that anybody with an agenda can edit, and anybody can quote without understanding what they are quoting,"

I doubt some random guy off the street wrote that wiki article. But you want something more official?

Try this:

https://www.ncbi.nlm.nih.gov/pubmed/1345555

"An accidental discharge of a Halon 1301 system is reported... .... workers present during the halon discharge reported significantly more lightheadedness, headache, voice change, cough, and a fast heartbeat than did those who worked the later shift."

"I wouldn't spec it if it were dangerous."

Sure you would, you probably only care about cost.

[jake]

Re: The BOFH Strikes Again

Whatever, dude. Believe what you like, no skin off my teeth.

[jake]

Re: The BOFH Strikes Again

Halon is no longer manufactured, true. But it IS still used. Use your .fav search engine to look up "recycled halon" and the rules & regs relating to it.

I have had a recycled bromotrifluoromethane protected data center go "live" this year, in ultra-green California. (That's Halon 1301, if you care).

[Anonymous Coward]

Re: The BOFH Strikes Again

@jake

and California has about the most strict environmental laws in NA, unless I'm mistaken. I'm generally impressed with impressed when I hear about things that California is doing in the news.

[jake]

Re: The BOFH Strikes Again

I wouldn't call them strict, I'd call them draconian.

[TechDrone]

Re: The BOFH Strikes Again

If you're in an office and somebody sets off a dry powder extinguisher you don't want to hang around. Although most of the powder will go where it's pointed you also get a fine dust that spreads everywhere and you'll inevitably inhale some. For the next several hours you don't want to be too far from a toilet...

[Anonymous Coward]

Re: The BOFH Strikes Again

I heard a story recently about a man who set off a powder extinguisher in his kitchen when something minor caught fire.

On Christmas day. Just as his wife was getting the dinner out of the oven.

All totally ruined obviously.

Bet he’s still in the dog house.

[J. Cook]

Re: The BOFH Strikes Again

You also don't want to use a ten pound extinguisher to put out a very minor fire in a tiny closet. Ask me how I know... *coughs*

[Anonymous Coward]

Re: The BOFH Strikes Again

“Perhaps their BOFH used the Halon system to get rid of some annoying users or managers? Bad on allowing a total shutdown and cascade, though.”

Possibly just particularly challenging contract negotiations? When the BOFH provided his or her new rate, it was considered too high and they were asked to demonstrate their value to the business?

A head count reduction through a middle management data centre visit and showing the business the damage that can be done when due care isn’t taken may have helped both parties reach an understanding.

Although 5AM suggests some of the managers may not have knowingly consented to the site visit...

[jake]

Wells Fargo is still a thing?

Who knew.

[Mike 16]

Re: Wells Fargo is still a thing?

Yup. Much like Bank of America, AT&T, and Atari. What's left of them is a name and some "goodwill" (in accountant-speak) after being acquired some years ago (In Atari's case, at least a half-dozen times).

Common decency forbids my naming several (formerly) U.S. and British auto and motorcycle brands...

(And don't get me started on "Craft beers" owned by mega-breweries)

[Evil_Goblin]

Re: Wells Fargo is still a thing?

Very much so given the size and expense of the new Wells Fargo building they have just opened in London...

[Anonymous Coward]

Bollocks!

Saying a prayer for my wife who works at one of their call centers - she has been front line for a lot of stuff over the past few years, and this means I had better get some sleep by the time she gets home, so I can wake up and help - my spouse is pretty strong, but when the bank has a bad day, everyone where she works has a worse one.

[jake]

Re: Bollocks!

Hint: Meet her at the door with a glass of her .fav plonk. For extra points, have a bath run and supper in the oven.

[YourNameHere]

Friday story

Sounds like a story for your Friday story where you report on those things that you did by accident...

[Throatwarbler Mangrove]

Re: Friday story

. . . and then someone calls "fake!" because reasons.

[redpawn]

Single point of failure

Well... They got it down to one, can't get much better than that.

[Kevin McMurtrie]

Hopefully everybody is still getting their 0.01% annual yield during the outage. That can add up to tens of cents over time.

[Anonymous Coward]

0.01%

Well, it's Wells Fargo, so at least you're getting 0.01% on eight different accounts! (that was their previous scandal... pushing a million services on everyone)

[W.S.Gosset]

Hardcore admins

> smoke was detected following routine maintenance

Say what you like about them, that's some SERIOUS maintenance. I've never had a script set fire to a machine -- I must clearly be a lightweight.

[Oengus]

Re: Hardcore admins

Just yesterday I had a laptop charger cable (C5) short and take out the circuit breaker and black out the whole section of the office. There was a flash, bang and a puff of smoke. I wasn't even doing maintenance. The laptop was charging on a desk beside me. The cable had been inspected and passed a few months ago. If it had been in the server room it could have triggered the smoke detection system. There is still a mark on the desk where the cable was sitting.

[Korev]

Re: Hardcore admins

We had a GPU used for MD calculations overheat and catch fire!

[Anonymous Coward]

Where's there's smoke there's fire.... What's the rest of this story??

[Anonymous Coward]

modern VESDA system will detect before you even get visible smoke, very sensitive so you need to be careful what you take in the DC and have fly paper traps by the doors

[John Brown (no body)]

"modern VESDA system will detect before you even get visible smoke, very sensitive so you need to be careful what you take in the DC and have fly paper traps by the doors"

Modern? I had a tour of Kidde Fire 30 years ago and they were claiming that level of "fire" detection even then, ie, detecting the vapours pre-smoke or fire. Having said that, I'd imagine that level of detection is a moving target as newer or different materials are being used.

[Nate Amsden]

working fine for me?

I have been a wells fargo customer since the 90s.

I don't use the online stuff often maybe once a week or something. But I just logged in, no issue(8:40 PM pacific time). Someone mentioned they couldn't move funds between accounts(2 checking accounts though the transfer doesn't use wells fargo it uses Zelle or something?? but it's fully integrated into the UI). I just did no issue. I don't really use the online banking for much more than viewing the balance and seeing the transactions.

From those aspects everything seems perfectly normal. I even clicked on a check I wrote and the image for the check came up immediately.

No errors anywhere.

I do not, and have never used their mobile app maybe that is more impacted I don't know.

[Anonymous Coward]

Re: working fine for me?

I have been a Wells customer since the 1970s, and have had great difficulty getting into their website these past two days, including this morning. There are reports today of payroll deposit delays, thankfully I get paid bi-weekly so they have a week to get their act together before I completely lose it.

I may need to rethink where I bank.

[chivo243]

Wells hastogo

An ex-colleague was a contractor for WF back in the 90's and early 00's. Oh, the stories he would tell. Kludge this, fudge that. I will never put one cent in their incapable hands...

[Down not across]

Re: Wells hastogo

Kludge this, fudge that. I will never put one cent in their incapable hands...

What makes you think it is any better at any other institution?

[89724102172714182892114I7551670349743096734346773478647892349863592355648544996312855148587659264921]

"fscked"

Why don't we FCUK anymore?

Trademarks are out of control

[John Brown (no body)]

"Why don't we FCUK anymore?"

Because it's an IT site, not a fashion site.

[M.V. Lipvig]

I like fscked better than fcuk anyway. It implies "two" things.

[tony2heads]

bad update maybe??

The one with the 'halt and catch fire' instruction

[Joe Harrison]

Datacenter fire just a cover story?

A couple of nights ago half a dozen military helicopters carried out some kind of operation at a complex of buildings which included a big Wells Fargo office. This was alleged to be training exercise but it looks pretty hairy and dangerous if it was.

https://youtu.be/slCN7oz-420

[Mike 16]

Re: Datacenter fire just a cover story?

I was wondering if someone had recently seen Ocean's 11 (The Rat Pack Version) and thought it could be profitably updated.

(TL;DR Latent hack to casino security is triggered by a widespread power failure. OK, blowing up a major power transmission tower is more dramatic than striking a match under the smoke detector, or maybe wearing particularly natsy cologne.)

Maybe we should check if Benny Hill is somehow still active in heists (Does WFB have Italian branches?)

[John Brown (no body)]

Re: Datacenter fire just a cover story?

"https://youtu.be/slCN7oz-420"

Looking at that, I'd say no way is that an exercise. If they need to train for that sort of situation, there are plenty of large military bases with street layouts and narrow landing zones they could practice in without doing it in a real city. There's too much potential for things to go horribly wrong, not to mention the possible damage from the helicopter down wash in such an enclosed space. You were right to use "alleged\" :-)

[jake]

Re: Datacenter fire just a cover story?

Of course it's a fucking exercise. Eyeball 1:14, see how the streets are conveniently barricaded at the intersection? See the city cops getting into position to take pictures right around 2:56, same intersection? Also note that none of the folks loading up are covering their rear ... one of the "rear guard" actually looks to be out for an evening stroll.

Sorry to bust a good conspiracy, but use your eyes.

[John Brown (no body)]

Re: Datacenter fire just a cover story?

"Also note that none of the folks loading up are covering their rear ... one of the "rear guard" actually looks to be out for an evening stroll."

You think they might have been under attack while leaving and so need "rear guard" cover in a US city? Whether it was for real or an exercise, they'd not be leaving an incident in a US city without being absolutely sure there was no threat behind them. It may well have been an exercise, but none of you reasons for it being so are confirmation. If it was an exercise, it was an incredibly irresponsible one.

[imanidiot]

Re: Datacenter fire just a cover story?

Man, they call that dangerous and hairy nowadays?? They'd be able to fly 2 abreast in that street! Plenty of space, roads closed off. Calm clear weather. <sarcasm> Yeah, really dangerous... </sarcasm>

[fredesmite]

A simple software upgrade to the UPS controllers !!!

Don't worry

It's just a 1 line bug fix

[Ken Mitchell]

Wells Fraudo

Couldn't happen to a better bank than Wells Fraudo. And I was NOWHERE NEAR the place!

[Kanhef]

Of course there were backups

They were in the next server rack over.

[fredesmite]

EVERYONE KNOWS WHEN THE SMOKE LEAKS OUT

of those little black chips they stop working !

smoke makes them work !

[DerekCurrie]

Why use Wells Fargo?

After a multitude of failings, both deliberate and inexplicable...

After gross customer abuse...

After contributing to the 'Great Recession' through malfeasance...

After requiring a US taxpayer paid bailout after malfeasance...

Do failures attract?