Scotiabank slammed for 'muppet-grade security' after internal source code and credentials spill onto open internet

Scotiabank leaked online a trove of its internal source code, as well as some of its private login keys to backend systems, The Register can reveal. Over the past 24 hours, the Canadian financial giant has torn down GitHub repositories, inadvertently left open to the public, that contained this sensitive information, after The …

[Daedalus]

Mean Mr. Mustard

"Public" GitHub means one thing to me: it didn't cost them anything. Had they paid the rent, they would have had a private repository from the beginning. Either the bean counters refused to OK the fees, or somebody set this up intending to take it private Real Soon Now.

[Sgt_Oddball]

Re: Mean Mr. Mustard

They couldn't even be bothered to scratch up the cash for a private server and run something like GitLab instead.

At least the should be able to white list access then as a most basic noddy step.

[cornetman]

Re: Mean Mr. Mustard

I am a soon-to-be ex-customer and I have to say this doesn't shock me in the least.

[Anonymous Coward]

Re: Mean Mr. Mustard

A little late there; but then again, maybe you can find a secure home with say, Capitol One.

[cornetman]

Re: Mean Mr. Mustard

People are telling RBC or TD are the best bet at the moment.

[Robert Moore]

Re: Mean Mr. Mustard

People are telling RBC or TD are the best bet at the moment.

TD tried to charge me an account closing fee when I decided to leave them after they refused to help me with a company I could not stop from removing money from my account. Much profanity was heard by everyone in the bank, near the bank, or within about 0.5Km of the bank. I then went home transfered my remaining funds onto a third party credit card, and never used them again. When they sent me mail, I wrote "Deceased" on it and stuffed it back in the mailbox.<BR>

For whatever it bight be worth, I am now with RBC for many years, and have been very happy with them.

[Anonymous Coward]

Re: Mean Mr. Mustard

Are you a soon-t-be ex because of this or similar cockups, or is it coincidence?

[JimboSmith]

Re: Mean Mr. Mustard

I find the term Muppet grade highly offensive to Muppets.

I find the banks attitude to security worrying in the extreme.

There's something quite impressive about just how crap they are. Maybe it should qualify for a Guinness World Record?

[Anonymous Coward]

Re: Mean Mr. Mustard

IME the cobbler's children go barefoot.

[Mr.Nobody]

Re: Mean Mr. Mustard

Sadly, this is the same story everyone is dealing with in the cloudy world and developers.

Anyone who has let devs run free with a credit card or an account in the cloud winds up with stories like this. I am not suggesting that security and systems engineers don't make similar mistakes, but developers just don't think about, or have a lot of experience with locking down these environments, but the bean counters and CIOs that want to be "cloud enabled" and "flexible" and want to "innovate" just keep allowing this to happen.

We had a bunch of open S3 containers with data on it we didn't want in the wild. We got the email from AWS telling us about it, and it took days not only find out who was responsible for it, but who had the creds to do anything about it. It was a sole developer.

[MAF]

Re: Mean Mr. Mustard

Worse yet, one of the positive things about Microsoft buying Github was that they changed the free tier to include unlimited private repos. So not an excuse even for cheap-skates...

[iron]

Re: Mean Mr. Mustard

Private repos have been free on GitHub for quite some time and setting a repo private is very easy. Imo either Scotia employ brain dead devs who shouldn't be allowed near code or one of them set these repos public on purpose. Perhaps a disgruntled and now former employee?

[Blank Reg]

Re: Mean Mr. Mustard

The real question is why does any large corporation store any proprietary code on an external repository. I understand using something like Github if you're an individual or just a small company with little infrastructure, but surely a bank can afford to host their own repositories internally.

[fajensen]

Re: Mean Mr. Mustard

Because they would only stuff that up also, with the entire blame falling squarely on them?

[Anonymous Coward]

Re: Mean Mr. Mustard

Bank source code should not be kept in any kind of cloud storage in the first place. It should be on internal systems on site and on failover systems at a backup site only with maybe 3rd party escrow with a trusted source. Even private cloud storage can be viewed by the admins running it especially with amateur hour sites such as github.

This is what happens when you have idiots straight out of university running important systems.

[Anonymous Coward]

Re: Mean Mr. Mustard

Viewed? Even being able to view is dangerous... but what about editing?

[John 104]

Morons

Why would you put credentials for anything anywhere online in the first place? Private or public, its a bad idea.

[Cuddles]

Re: Morons

Convenience. The perennial enemy of security.

[Tom Paine]

Re: Morons

Convenience, and not knowing any better.

[Anonymous Coward]

Re: Morons

There is no convenience in compliance, an auditor once said.

[macjules]

Re: Morons

Because those morons (and I'm looking at YOU TCS) do not know what security is, do not care that they do not know and even if they did would not know what to do about it.

"Other People's Money" and all that ..

Oh, and by the way has anyone looked at https://www.scotiabank.com/global/en/global-site.html and checked the response headers? Just saying, like.

[AdamWill]

called it

Oh man, I was counting the days until this happened ever since reading this story:

https://www.theglobeandmail.com/business/article-brian-porter-is-making-big-changes-at-scotiabank-but-will-investors/

as I wrote in the comments there:

"As someone who works in IT and was previously considering switching to Scotia...the above has convinced me definitely not to do that, ever, or to invest in them."

[Anonymous Coward]

Re: called it

paywall :-(

[TFL]

Re: called it

Only if you have JavaScript enabled...

[AdamWill]

Re: called it

Ah, sorry, I thought you got one or two free from the Globe before they cut you off. Here's the relevant quote:

"Early on, Mr. Porter took senior bankers and board members to Silicon Valley for a lesson in pace.

“He wanted that mindset to be understood,” Mr. Zerbs says. “A mediocre outcome is clearly not okay any more. Like, we have to be the best … and also make mistakes along the way.”

Much of the heavy lifting on technology occurs behind the scenes, as the bank strips costs out of its legacy infrastructure through automation, cloud computing and advances in artificial intelligence. But to incubate the necessary sense of urgency, Scotiabank set up "digital factories” in five countries – Canada, Mexico, Peru, Chile and Colombia. The labs undertake rapid-fire projects aimed at solving bank-wide pain points for customers, sometimes rewriting computer code that can be deployed in the span of a few days."

[Rich 11]

Re: called it

sometimes rewriting computer code that can be deployed in the span of a few days

...QA optional.

[Jimmy2Cows]

Re: called it

Optional would suggest someone had considered QA a possibility.

This is more like typical banking 'Agile'; release it and let customers do the QA.

[Rich 11]

Re: called it

Beta permanence.

[sal II]

Re: called it

Aaah DevOps... <3

[Blank Reg]

Re: called it

I can remember when banks would take 12-18 months validating new versions of an OS before they would deploy it in production and require a guarantee of at least 10 years of support for that particular version. Now they want to push out code in a few days? What could possibly go wrong?

[Bronek Kozicki]

Re: called it

lynx rules

[IGotOut]

You see Dev's

This is why your IT guys and gals are "being obstructive" when they make sure you follow procedures. But hey don't bother with them, just use 'the cloud", far easier.

[iron]

Re: You see Dev's

You see Ops it's a shame the devs have to point out you can't handle a simple apostrophe. It makes you look stupid.

Here's a hint, nothing belongs to the devs in your sentence so you don't need it.

I'll leave your incorrect use of commas for another day.

[Law]

Re: You see Dev's

"This is why your IT guys and gals are "being obstructive" when they make sure you follow procedures. But hey don't bother with them, just use 'the cloud", far easier."

Whatever... where I work it's management and IT driving us into the cloud, because it'll solve all those pesky issues of IT having to maintain servers or deliver a usable internal network for our build infrastructure to run on. But hey, why not blame all the developers...

[Anonymous Coward]

Re: You see Dev's

But hey, why not blame all the developers...

Why not both? Just rebrand the team to 'DevOps' and let rip!

[JaseCoulls]

Sorry I'm late to the party...

If anyone has any questions, I'll try to answer.

[Anonymous Coward]

Re: Sorry I'm late to the party...

Why Donald Trump. And how the hell Brexit?

[Giovani Tapini]

Re: Sorry I'm late to the party...

Where did you find them? and why did you bring them to me?

[Kane]

Re: Sorry I'm late to the party...

Tea or Coffee?

[Tom Paine]

Re: Sorry I'm late to the party...

The story says you previously discovered / disclosed some Scotia fail in 2017. Is it purely coincidence that you stumbled over stuff from the same bank twice, or do you keep a specific lookout because their record suggests you know there'll be another security snafu along before long?

[JaseCoulls]

Re: Sorry I'm late to the party...

I keep a special eye on this bank. Originally, as a customer, I was concerned that they were putting me in danger. In 2016 I told friends and family to stop using their mobile products after I spotted the insult screen aimed at Kony Inc, hidden in the Android app (they only started Android java obfuscation in summer 2019). That told me right there that anyone could pull off an inside job, because if you can add an entire screen to the app and the bank doesn’t catch it, adding two lines to siphon bank credentials from the login process was going to be a walk in the park. That got me hooked on watching the stupid things they do. Until they disbanded the CCIRC, I would report the big things like that to them, practicing my observation skills along the way. Later, in 2018, Scotiabank and I got into a tussle where they showed a side of customer service that was reprehensible. Since then, I’ve upped the ante (including automation to keep tabs on stuff), and aim to document as much as possible to show orgs like OSFI and the Privacy Commission of Canada that these people don’t know what they’re doing.

[Tom Paine]

Re: Sorry I'm late to the party...

Thanks for the excellent answer, very interesting stuff. I worked in the (enormous) security team at a well-known international megabank where money was no object, for a while. Any one of those incidents would have caused massive ructions, sudden resignations in management, reorgs, total revamps of SDL and testing processes, etc. Current berth (not a bank) has a bit more of a Scotia attitude (it seems) in that they're throwing money at half-baked attempts at agile and what they think security is* whilst carrying a gigantic tech and security debt load. In a way, it's just as interesting an experience... (After all, it's not _my_ firm. If that what they want to do...)

*(Expensive blinkenlights and minimum wage analysts in a very, very cheap offshore location, mostly)

[Anonymous Cowtard]

Why is orange jam called marmalade?

eh?

[BebopWeBop]

Re: Why is orange jam called marmalade?

cos it contains peel

[The Oncoming Scorn]

Re: Why is orange jam called marmalade?

I prefer shredless, is it still Marmalade then?

[BebopWeBop]

Re: Sorry I'm late to the party...

importantly, before I get into life, the universe and everything, where's the free beer?

[Anonymous Coward]

Hi grandma, may I use your fax machine?

I have bad experiences with them and technology. Did you have to notify them via signed fax, courier, or registered mail?

I would never attempt to configure critical services myself but, if I did, I would try hitting them with my cellphone to see if anything was world visible before loading up the data. (no, not your flip phone)

[Michael H.F. Wilkinson]

Flabbergasting level of incompetence

They might want to make a new Ig Nobel award category for this.

Or perhaps we should have a Dr. Bunsen Honeydew Prize for Muppet-Grade Security Flaws

[Tom Paine]

Re: Flabbergasting level of incompetence

Or a Professor Denzil Dexter Award for fail in the name of innovation?

[Not Elvis]

Re: Flabbergasting level of incompetence

Perhaps a financial Darwin award is in order here?

[Robert Moore]

Re: Flabbergasting level of incompetence

we should have a Dr. Bunsen Honeydew Prize for Muppet-Grade Security Flaws

This totally NEEDS so become a thing.

[Tom Paine]

Re: Flabbergasting level of incompetence

https://pwnies.com/

[Aristotles slow and dimwitted horse]

Whilst...

Yes, whilst this is totally idiotic and obviously Scotiabanks problem to resolve, I do wonder if this is a fuckup caused by in-house resources or by the cheap and inexperienced offshore labour in India being so remote from the business that they don't properly understand the implications, or perhaps give a f**k.

Either way, you do have to smirk (a bit)...

[JaseCoulls]

Re: Whilst...

It's definitely gotten worse since many more devs were brought on in Chile, Peru, and Mexico.

[Groaning Ninny]

Totally off topic, another Canadian bank (ANC) keeps sending account info to one of my gmail addresses. Clearly there's no verification going on there. Additionally, I've tried to get them to stop, but any email I send seems to go unnoticed.

Sadly, the account owner keeps going overdrawn, and it's not a lovely thing to see.

[Anonymous Coward]

Have you tried dropping the account owner a line? They have motivation to go properly ballistic and they are "authorised" to whinge about it.

[Prst. V.Jeltz]

I doubt the email contains enough info to identify the owner .

[Steve Graham]

I did some development work for a UK bank back in the nineties. The restrictions, rules and regulations were strict, and were strictly enforced. After all, it was people's money that was at stake.

Clearly. times have changed.

[andyL71]

Because, Agile.

[Kubla Cant]

Because, Agile

Crap QA has nothing to do with agile.

There's nothing in agile development that says you have to be careless or deploy untested code. In a well-run project, nothing gets into development without a clear set of acceptance criteria that are subsequently used to validate the work.

I don't recall any of the waterfall projects I used to work on having superior QA. The way waterfall works tends to mean that QA is crammed into whatever time is left over before the delivery deadline, and the components being tested are more complex.

[BebopWeBop]

Yeh, true, but then ISO9000 didn't say anything about quality - yet, people used it as shorthand for such.

[Anonymous Coward]

@Kubla Cant - Yeah, right!

Security ? We've heard of it.

[cdegroot]

Indeed. Well before the term was coined, I worked on projects in the pharmaceutical industry, heavily regulated code (Good Clinical/Laboratory/Manufacturing Practice, FDA inspections, that kind of fun), and we could still be light on process and heavy on rapid learning and quick delivery; the acceptance criteria just specified lots of testing, including automation, and including printing out test reports that the project lead had to sign and file. I had similar experience building software for some banks.

However, now that "agile software development" has turned into "Agile for Enterprise" and all sorts of nonsense (SAFe anyone?) - that sort of stuff just makes project managers fight different political battles and QA is still out the door. Quality is a mindset, not a process.

[Blank Reg]

It sounds like they have embraced the "fail fast, fail often" mindset to the fullest.

[jsioui]

I worked for Scotia in security back in the 90's. All was good until IBM took over. It's all about the money (pun inteneded or not) which reminds me, have to sell some stock.

[DontFeedTheTrolls]

Perhaps banks should be required to have "Accident" boards on their front webpage a bit like constructions sites on their fences.

It has been 3 days since our last security breach

[Anonymous Coward]

Former Scotiabanker

"By the time you read this, the GitHub repositories, which presumably were accidentally misconfigured by Scotiabank's techies, should be hidden or removed."

As a former developer at Scotiabank, I can tell you they were not "accidentially misconfigured"; the proper adverb is "incompetently". There are good coders at Scotiabank. Too bad their entire day is spent trying to stem the tide of the legions of horrid typers (they aren't coders because a coder makes code, not just keystrokers).

[Anon Coward (there are nutters out there - I've worked with them)]

huh?

what is all this GitHub billix about?

I'm seeing it all over the shop.

[Anonymous Coward]

Security mindset from way back

Back in the 80s, I accidentally left my Scotiabank credit card behind at a Business Depot (but did not realise it until a week later). The first fraudulent purchase was at a gas-station just down the road. Then came a string of fraudulent purchases at one store over the course of a few weeks. Scotiabank's grunt-and-poke security talked to me, asking if a relative or friend has visited before the theft. Security angle: my first name is a very common Anglo-Saxon name, the fradulent purchaser was a female, and I learnt that the person signing for the purchase need not be the card owner. (And no, they never bothered to visit the store in question.)

[mstreet]

What amazes me...

Is that more than 12 hours after this story was posted, not a single Canadian news outlet seems to have picked up the story.

Apparently our half-wit PM wearing black face at a party 20 years ago, is far more important...

[The Oncoming Scorn]

Re: What amazes me...

Now you are just being kind about sock boy.

Roll on October hopefully he can do a bollywood dance out of the door & I hope it hits his ass hard on the way out.

[JSIM]

Re: What amazes me...

Yes, they're focusing on the breaking brownface news because they know people like you will lap it up. At least something on a slow news day and in a quiet election run-up, guaranteed to send Trudeau's haters into joyous convulsions. This minuscule "scandal", will be forgotten in a few days, and the 3 anti-Trudeau comentards here and their ilk will have left their usual dinners of nothing and nothing with an extra helping of nothing. Nothing to do except troll around IT forums spreading hate.

[Alistair]

Excuse me ScotiaBank, would you mind informing us who it is you've outsourced your IT to lately?

/walks down the hall whistling dixie.

[Anonymous Coward]

Can you make it up?

You must have some good stories to tell of all the great projects you've been working on. ;)

[Barry Rueger]

Surprised? Hardly.

It's only been a year since Scotia Bank finally relented and made passwords case sensitive and allowed special characters.

Lately though they forced mobile customers to "upgrade" to a new improved app that lacks several heavily used features, and which is inarguably harder to use for everyday tasks.

For instance, instead of tapping a button to confirm an action you now need to drag a slider across the screen.

[jsioui]

What the...

I would ensure that the Cheif Risk Officer, CISO (who I knew well when I was there), CTO and the CEO are all held accountable for the dreadful manner in which they handle information assets. I'm almost ashamed to say I worked for Scotiabank. Makes you leary of how they handle other facets of the business. On another note their stock is up today. Incredible.

[LeahroyNake]

Muppet grade security

Almost lost my drink out of my nose.

Best headline ever lol

[queueback]

Not just incompetence, also a culture of unethical conduct in Scotiabank

The story and comments are rightfully talking about incompetence. Keep in mind the Scotiabank’s security department, which is called Information Security and Control inside the bank, embodies underhand and unethical conduct. There is a branch in the security depart called Information Security Advisory Services that is supposed to do security risk assessments of all Scotiabank technology products and services as a gatekeeper function. Unfortunately their Security Advisory is marred not just by poor competence, but also because of falsified assessments and risk profiling.

This Security Advisory group is well groomed to make any serious security gaps and risk impact findings, in their so-called assessment reports, as low impact or low risk (no matter what the real risk is). If you read security risk assessment documents of Scotiabank, most, if not all, threats from their services are classified as low or inconsequential impact.

So, the incident or breach like this one should not come as a surprise. If you don’t believe me ask Scotiabank CISO to make available reports called TRA (threat and risk assessment) for their customer, Internet facing services.

[Lion]

A security breach, even when tens of millions have been exposed to possible exploitation, seems to not move the shock meter a single click any more. The known damage and the potential damage is treated as a mere notification. We have been numbed of our senses.

Canadians think highly of their banks, so I am not expecting front page headlines giving Scotiabank the WTF treatment or a run on the bank by its business clients and private account holders. The corporate suits are counting on this and they will spew out the usual PR to cover it. This complacency, if not checked now, will end in a world of hurt for the bank's customers. If the leaked data has been collected, it will be appraised and sold to miscreants, i.e. organised crime and rogue states. The grim reality is that the bank will probably escape with a small fine by the regulators. They can also keep lawsuits in the courts for decades. They accept responsibility for causing the leak, but they are never held absolutely accountable.

[raving angry loony]

The same people who own shares in the banks also own and control 99% of our "free" press in this country. So no, I don't expect ANY public notice to happen in said publications. As it wasn't client information that was leaked, which would be covered by legislation, "muppet grade" security (and that's an insult to muppets) is not covered by legislation. No crime, no news, no mention. Move along, nothing to see here, everything is fine.