Ta-ta, security: Bungling Tata devs leaked banks' code on public GitHub repo, says IT bloke Staff at Indian outsourcing biz Tata Consultancy Service uploaded a huge trove of financial institutions' source code and internal documents to a public GitHub repository, an IT expert has claimed. Jason Coulls, CTO of food safety testing company Tellspec and a former banking software developer, said he stumbled upon the …
There appear to be a lot of comments like this around here recently. People should remember that problems with offshoring and/or outsourcing are usually caused by penny-pinching PHBs signing up to crappy agreements. People's location and nationality are not good indicators of competence.
Korev said " People's location and nationality are not good indicators of competence."
No, but repeated screw-ups, terrible customer service, staff who cannot understand the difference between a non-technical user and a desktop support tech with 10+ years of experience ("no, switching it off and on FOR THE 4TH TIME in 1 call will not help - the hard drive is making horrible grinding noises and is knackered - I just need a fault reference number to give to the client!"), relentlessly sticking to a script when the user is able to give a perfectly good explanation of what the problem is, etc etc etc... *these* are good indicators of competence, and I've experienced these - and more! - since a bunch of Cowboys (allegedly) Supporting Computers 'onsourced' our hell desk to another continent.
Spot on, however I wonder if El Reg actually understood the significance of the following:
The documents related to programming work Tata was carrying out for six big Canadian banks, two well-known American financial organizations, a multinational Japanese bank, and a multibillion dollar financial software company.
Silly me, but shouldn't there be a glass wall between different clients?
Meaning if I were one of those clients, not only would I give Tata the boot, but also would be getting the team of in house counsel lined up to sue the carp [sic] out of them for violating and sort of MSA and NDA.
At the same time, this wasn't the work of a single bad programmer, but a concerted effort to share code.
This also shows not only the lack of professionalism but also calls to question their skills as developers. Which implies Tata is over charging their customers by providing under skilled employees.
Sorry for the flame, but its more about the situation than anything else.
... is probably a beginner developer who believes everything today has to be stored on GitHub, he read it on the Internet...
Anyway whoever still use file names like document_releaseX.Y really needs a version control system and a document management solution - not a public GitHub repo, maybe...
I doubt it was just a beginner developer. This was probably the entire team collaborating on a set of documents.
And I bet no bank put "Do not upload everything to github" in the contract. If someone did, Tata would probably have gone to Sourceforge instead.
Just because it's older doesn't mean it doesn't work. I have seen instances of modern document management systems losing history, it can happen. The old ways may be top heavy but it was harder to lose history. Several times when old documents were needed this was because people were too lazy to manage the old versions out; inefficient but it was a life saver.
My guess is that it was someone who was applying for another job and told by someone, somewhere (these days its "everyone, everywhere") that they needed their work on github so potential employers could see their work.
Oddly enough, most people who demand a Github repository have no idea what they are looking at anyway.
We have exactly the same problem with our London devs. In fact it's worse with our London devs - the Indians say "sorry it won't happen again" the London devs say "developing in public is a human right, you're trying to suppress my creative spirit so we're going to carry on doing it and we'll probably leak confidential information again, it's just a risk we have to take."
Far too often the whole offshoring idea is to pay people peanuts to fatten the execs bonuses and stock options. Otherwise often you would have no need to outsource or offshore.
Then there are the cases when you need more competent people than those locally available, or you need a broader presence around the world, but I guess that's a smaller percentage.
"Except your London devs will be able to code. Unlike your Indian ones."
Not always, the Indians have their faults and idiosyncrasies but so do we and everyone makes mistakes. BA maintain it was a Brit that did it because outsourcing was not the cause. Technical superiority and excellence is required to beat the opposition. It's the same as sport, the best team usually wins.
If anyone thinks voting Brexit or rejecting outsourcing will save them from a lower cost base then who is deluded? If we insist on raising our living standards by buying cheaper from elsewhere then why should our jobs be any different? If more technicians read economics they would understand why this is a losing battle. As my instructor told us " Get good or get gone!"
PS I have worked with TCS and seen colleagues outsourced to them.
The BA situation isn't really about who pulled the plug out. It's more about when the plug was pulled out the system stopped. I find it staggering in this day and age that a single data centre can be responsible for a large companies critical systems. The developers are to blame in my opinion, as they should know you don't build systems this way.
This kind of perceived technical superiority...
It's not misplaced superiority, but weary irritation. Every big bank I know of who's outsourced development to the Indian code-shops has needed to create a (very expensive) team, often made up with some of the people whose elimination was supposed to be part of the cost saving, to 'review' (i.e. hack until it compiles) the fruits of our Indian consultancies. Until you've seen it, you might not believe just how bad the output is, and how little theses people know or want to learn.
In the same way that if you're a good financier, why would you work for the Inland Revenue; if you're a good developer, you'd never ever work for a code shop.
"We have exactly the same problem with our London devs. In fact it's worse with our London devs"
Not sure where you get that one from. We offshore a lot of development work to around 3 separate countries with code review and management being handled in London to very strict guidelines with full automated unit, security and BDD testing. Missing a carriage return in PHP can cause a module to be rejected, as an example. To my knowledge every agency I encounter in London now have very similar standards set up in their dev environments.
I do notice that Scotiabank's public-facing website does not exactly conform to any known standards though. Must be a completely new system.
Reminds me of many years ago when our company introduced the Crosby "Quality" regime.
At first people were raising reports for existing problems outside their control. Then they started to complain that they never saw any resulting "Corrective Action".
Management solved that by a decree that anyone who raised a problem - became the "owner" to progress it. Overnight the number of problems being reported plummeted - as everyone took the Pink Spaceship approach to an S.E.P.
http://coulls.blogspot.co.uk/2017/05/how-do-you-fix-mobile-banking-in-canada.html
Is it true that having urls embedded in the code of the application but not obfuscated in any way is really a major security issue? I'm guessing the queries sent to the server side of the application constitute the threat.
Icon: Clueless end user who does not make use of any form of online banking out of a natural tendency to prefer face2face transactions.
> Is it true that having urls embedded in the code of the application but not obfuscated in any way is really a major security issue?
No. Not sure why he has a problem with it. Although having them easily accessible definitely makes things easier for attackers, the whole point of good security practice is that if you have your shit together then people hitting your API end points isn't a problem.
Security by obscurity isn't really a best practice approach. ;)
"Clueless end user who does not make use of any form of online banking out of a natural tendency to prefer face2face transactions."
The banks in our large-ish UK town have adopted machines to handle most counter transactions for retail customers - with a floor walker to show people how to use them. Any "commercial" customer's other transactions are serviced by a single teller.
Yup, the lad with the iPad and the chip'n'pin thingy. He always points out that I can get identical services from home. I always point out that by coming in and queuing up and accessing the system via a member of bank staff, the liability for any subsequent security issues is clear.
Coat: annual gas check done, off out to dodge the smart meter salesman
I had to take a relatively large (just over £1k) sum of money for the business to our local HSBC bank to deposit it. It was the proceeds of a charity raffle and I went with a colleague for security who then stood outside the bank smoking. When I walked into the branch there was the floorwalker and several empty but open counters. I suspect that not looking my finest clothing wise because I was to be chasing cables in the basement later he decided to pounce on me. He indicated that I could use the machines to make my deposit and said all I needed was my bank card. Confused when I said I didn't have one he told me he would order me one immediately. I explained that I didn't bank with them the business I worked for did and I was there to make a deposit with the paying in book. Only then was I allowed near the counters and only after I said I needed to actually have the book stamped - company policy.
The girl who counted the cash by hand told me they saw less and less people because everyone was pushed towards the machines. When I asked why she did it by hand and then scales (my deposit was all notes) she said they were too tight fisted to buy electronic counters. As I'm leaving the floorwalker with no one else to attack asks me who I bank with and then tries to tempt me away from Lloyds. He failed miserably though when I explained that I had private banking with Lloyds for which they don't charge me. "We can't match that" he grumped "They must really like you".
When asked in a Lloyds branch why I didn't do something on-line that I had come into the branch to do I said because I'm supposedly one of your better customers. I also explained that I trusted the private banking call centre in Leeds more than I did internet banking.
I agree. If I don't like it I will keep move banks till one of them decides having a human in place is a good plan. NatWest just closed my local branch, so I closed my account. I know they are likely to win in the long run and we will then have to pay through the nose, until then I am a customer and the customer is always king.
The Solution Architecture, and most of those other documents are 0.x....
Having worked with outsource providers before, we always approved and finalised documentation prior to development - otherwise you get the inevitable change requests from scope creep and change. Quality was always hit and miss so scoping the work package correctly with the outsource partner was always required.
You do realize that just parking client's material off the client site could be illegal?
If any of the Financial institutions considers the code to be trade secrets... you have a crime of theft.
At a minimum, you most definitely have multiple breaches of the contracts per institution.
This could cost Tata a boatload of money.
Disclaimer: I am a TCS employee.
It appears that the filenames were more like the names of the customers the presentations were made for, rather than *data* pertaining to the customers themselves. I therefore suspect a lot of the content may have been the same (i.e., present to customer A, modify slightly, rename, present to customer B).
It's still a pretty stupid thing to do, but thank God it wasn't stupidER, I guess!
Unfortunately, I was one of the people shouting from the rooftops (a few years ago) that we need unfettered github access, so I'm getting a wee bit of -- good-natured, don't worry! -- ribbing for this!
But then, I couldn't do without this access. I estimate that a good percentage of the commits for gitolite are made at work and I push them from my work laptop, simply due to how I divide my time.
Sitaram
PS: gitolite is a fairly popular access control system for git that is used by Fedora, kernel.org, Gentoo, and several other open source projects, and probably thousands of others
And yes, I intentionally mentioned it, in a shameless and blatant attempt to suggest that if you've heard of it, or even better, used it, then *you* at least won't generalise about TCS :-)
Doesn't matter.
You create a presentation for a client, its usually a work product for which the client owns. (YMMV depending on the contract ...)
I knew for years that American Express was using MapR. Due to my contract(s) I was unable to say anything publicly about it.
Posting anonymously as I used to work on the Scotiabank account through IBM and have seen things (the horror, the horror...)
First of all, there is a quality issue with bringing outsourced teams from another place (currently, it's India) be that through IBM Canada or directly by the bank as part of diversity hiring needs, etc.
This is not a racist statement - it's fact.
Outsourced teams will and do cut corners to save money - and having worked in India for many years, there is a malaise there - a level of mid-management there that act like feudal lords and treat the programmers / architects / tech resources as serfs. Accordingly, turnover is higher than the US/Canada (can't speak of the UK as I haven't worked there) which results in candidates who aren't well trained because they've been hired in a hurried fashion to be seat-fillers or because they haven't been brought upto speed yet.
As some of you already may have experienced, hiring on the basis of diversity does not get the best candidate on a technical level. Nepotism does thrive - human nature, tribalism, etc.
Work ethics based on place of origin also come into play - some follow the West Point 'duty, honor, country' creed, others not so much.
Just like any other nationality, I've worked with some excellent folks from India and some people who were so bad, I wanted to slit my wrists to avoid the sheer stupidity.
Someone used the right acronym to describe that mid-management bunch - PHB's - can't agree more.
Back to the topic at hand, I'm rather concerned with what this bunch of devs did - and more so at their not getting it as a big deal.
Even worse is the head in the sand attitude of the big banks.
CIBC & Scotia are both terrible when it comes to things like this. And guess what, they're the ones who outsource a lot more by comparison.
Outsourced teams will and do cut corners to save money - and having worked in India for many years, there is a malaise there - a level of mid-management there that act like feudal lords and treat the programmers / architects / tech resources as serfs.
That's a cultural thing. I've seen it from on-shore Indian managers. While this isn't across the board, its a high percentage.
Go figure!
(Scotiabank's physical security was just as bad. Decades ago, I noticed my VISA card missing after a purchase at Business Depot. The first fraudulent purchase was at a gas station next door to Depot. Then came a dozen fraudulent purchases at the same clothing store 20 km away (by a female and my male name is on the card). Scotiabank security kept pestering me about my sister-in-law, who lived 150 km away. (Sigh)
That has nothing to do with it.
The US already had a large infrastructure in place and they looked at the cost of moving to chip and pin versus the losses (theft) that they had at the time. The cost of moving was higher. So they didn't move.
When there was more data theft and fraud such that the cost of moving was cheaper, the US moved.
That's pretty much the gist of it.
The various PHBs have not figured out (probably never will) that most of an organization's IT functions should remain in house, done by direct, permanent, employees. This is the only scenario that allows for the most control of the information, data, and code. This is especially critical for privacy issues. The third party company may or may not have all the proper controls required by various national data privacy laws. Medical data laws are similar but not identical to financial data laws. I am reasonably familiar with one (HIPPA) and only have a vague awareness of financial but I work in the medical industry.
Employees of the third party company first loyalty is the company paying them not the client, as it should be.
"HIPPA"
It may be hip, or even hippy, but what the heck is it?
Maybe you are familiar with that acronym HIPPA - whatever it is, but certainly not with the Health Insurance Portability and Accountability Act 1996 (aka HIPAA)... So much for claims of exertise by familiarity...
I am familiar with HIPPA and several different countries banking laws.
Its interesting to point out that none of the files shown on the screen shot included code, but were presentations made to different clients.
So while no data was compromised, (No shock there.) there was client specific information. The presentations are work products and should be owned by the client where TATA doesn't have the right to reuse them. (Note: This could be added to the contract, IBM does this all the time with their Type III contracts.)
I have a friend who tossed Mu Sigma from the account because he couldn't trust them to not reuse proprietary algorithms at other clients. The whole point is that many Enterprises are starting to learn that moving offshore may not be the best news. However... it will take many years to right this. Note that if there is another disruptive technology.. it could increase the speed of moving away from offshoring.
Of course if there is a case for a massive lawsuit against a company because there was a massive error done by the offshore team.. It could happen faster.
I am reminded of a software development group that I worked with the USA - they were shut down and outsourced to India - the indian people assured that they were up to the task (this was O/S level and Device driver stuff). There are a few online forums for that platform - all of a sudden, they start getting new members with Half Indian names (like Charles kutrapali), and are asking the most basic questions, as well as some very specific ones. An Associate had a major bug report outstanding and one of the questions was the same as appeared from the new Indian Forum member - So, his query was responded to in very specific language, and lo-and-behold - that was almost word for word, what my Associate received as information on his bug fix. That group is now back in the hands of a US Team of SW Engineers.
"Have noticed the same thing when searching MS for help. Canned answers are totally useless."
Agree, MS Help is useless. MS hired tech writers to compose Help files for the software. However, since the software and its associated help files get released simultaneously, the tech writers are working from software specs, not from final version software, nor would it be efficient to let mere writers ask questions of the software producers. See where this is going? This disaster is compounded by other factors. Of course, once you have a help entry established for a question, bean counters say it's hardly economical to hire another team to do the same work again. That's why Help for 21st century Windows can look like it was written (in ignorance) for Windows 95. Not a candy mint? Please consult your system administrator.
PS I have never worked for MS. The above is from inference.
> ... nor would it be efficient to let mere writers ask questions of the software producers.
Efficiency be damned. Make sure the writers can ask the dev's, and get a useful result, otherwise the written docs will be shit. Leading to unnecessary support expenses, bad reputation, etc.
Keeping software devs away from the doc production is rarely a good idea.
That happened to someone on a Solaris / Sun Microsystems / Oracle forum I read and post at a few years ago.
Forum member's job, and actual whole department was outsourced to India. He was looking for work and had more time to spend on the forums and suddenly new members joined. They first started out with Solaris basics and then it moved onto more detailed questions and finally the new members quoted directly from the documentation our older member wrote himself (that was proprietary) and he realized he had been helping the people who replaced him.
So, he had a bit of fun with them "answering their questions the best he could".
I am not a proper programmer. Any code I provide, real programmers will think: "Is that some sort of awkward pseudo code?" Anyway, I scanned the linked article down to the red ink before going to bed. In the sweet befuddlement of waking up the next day I was dreaming of Hester History. Deep Hestory. I remembered from the days when you had to fit the OS, the program(me) code, and the program's own space into 64K RAM, that you'd never have a list of resources all starting with "https://www.somebank.ca/...". No, you'd define the quoted part as $BaseAddr and make up each resource name as $BaseAddr & "whatever" on the fly, this economizing (ResourceListLength% - 1) * (BaseAddrLengthInBytes%) bytes of precious RAM. Well, almost. Less pretentious than "asymptotically". The year was 1979.
I also remembered the Deep Hester of 194? (before my time, honest) when they were able to decode the Enigma messages but not in the 24-hour expiry of each Enigma code--until somebody (and I like to think it was one of the chess players because this is the kind of thing that a chess player should notice, rather than one of the math geniuses who get the lion's share of the credit) noticed that one military clerk liked to begin each message with "Hail Hester" (Godwin forfend what he really wrote) and using that repeated pattern they were able to decode the messages in less than 24 hours and bingo, Coventry was saved from total destruction. Oops.
So using $BaseAddr not only makes the compiled code slimmer, it also eliminates the repeated-handle fulcrum into de-obfuscating the code-that-you-want-to-hide. Returning to the linked article, I noticed that the author Coulls made that same points, inter alia, though in very different words. Whether this actually makes a tinker's cuss of difference to cybersecurity I know not, but one gets a warm fuzzy feeling to use Hestory to pretend to delve into some of the Great Problems facing Mankind in these Troubled Times.</billhocks>
It has proven that offshoring hell of issues like quality, commitments and availability of dependent resources however the management in the big companies don't give a sh!t. This also has severely impacted local job market with less number of jobs and decreasing pay rates.
What I learned from this article, it that off-shore development firms like the one from India that is mentioned, is where you will find the lax security and cultural differences.
There is no surprise at all about that statement.
If there is any cultural difference between Canada and US, it would be all the socialist/communist laws in Canada that make developing our own code a cost prohibitive activity.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
