"The supermarket employed a team of security specialists in Bangalore tasked with monitoring its computers around the clock."
... and there is, ladies and gents, what you get for "outsourcing" something as critical as IT Security.
Staff at US chain Target reportedly failed to stop the theft of 40 million credit card records despite an escalating series of alarms from the company's computer security systems. Bloomberg Businessweek claims that security technology from FireEye detected the malware-powered hack – but Target staff failed to act on the alerts …
I was just about to post that myself.
I wonder if it would have made any difference if their security team had paid attention to the alerts because it sounds like senior management have a head in sand policy. I hope some government regulator gives them such a good reaming over this that the CEO can't sit down for a month.
Do you have any idea how many other fortune 500 companies outsource their IT security operations and they are operating just fine ?
While you just simply posted some dumb*ss comment about outsourcing, you didnt get the critical fact that, the alert was already sent from the Banglore location, and was just simply ignored by the "big heads" in the US HQ.
No wonder, the IT security was outsourced ...
Except that if you read the linked article you'll find that it was the Target directly employed security team that dropped the ball. The outsourced service was on the ball and sent the alerts to the Target Team who promptly ignored them.
Given that the malicious payload is alleged to have had a filename similar to a Dell management component, it's entirely possible the directly employed Target Team, overflowing with your attitude, went, "Idiots have no idea what they're monitoring. That's one of our management components, whitelist it."
Given my time in the trenches, I'm not sure an insourced monitoring team would have gotten through any better than the outsourced team.
Given my time in the trenches, I'm not sure an insourced monitoring team would have gotten through any better than the outsourced team.
An insourced team in the same building as the senior people does at least have the ability to go bang on desks in person and look the management in the eye. However,they'd probably be stuck in some other office, and wouldn't have that advantage.
Insource or outsource, if the same people who designed and built it administrated it and monitored it for intrusions, I think it would have been caught and fixed much sooner.
Essentially, you give one organization responsibility to perform a function and leave the details to them. It doesn't matter if they use a DBMS or an office building full of elves and filing cabinets. As long as they keep it running to spec, who cares?
This was a retailer during the holiday rush. This typically involves a change freeze that starts a number of months earlier. You're not going to see much of anything new going on in IT at a retailer once summer ends.
Mucking around with anything will likely require more authority than your typical boffin has.
Just imagine going to your PHB and announcing to him that you've got to go all Andromeda Strain with your point of sale systems right before Xmas.
"...a change freeze that starts a number of months earlier."
Quite. And management often believe that if they apply strict change management, nothing bad can ever happen.
When facing a new style of attack using DNS on our obsolete firewalls, the big boss asked who had authorised the change (in the type of attacks). She seemed confused when told that the hackers had not submitted a change request. The upside was that our requests to replace all the obsolete kit and have properly designed, configured and managed network finally made sense to management.
"The supermarket employed a team of security specialists in Bangalore tasked with monitoring its computers around the clock."
First, beaten to the punch.
Second, that what you get for outsourcing....
Third, damagement strikes AGAIN!!!! Didn't these dumb fucks get the message from the RBS fuckup - "Don't outsource critical functions to third parties" But, bonuses all around for ...
It's a Good Thing I don't shop there!!!!
The Bangalore monitoring team sounds like pretty standard industry practice. Likely, Target contracted for a Security Operations Center to be set up and staffed offshore (India in this case). The Reuters story reported that the Bangalore team forwarded the alerts directly to Target headquarters (no doubt compliant with their SLA). The personnel in Minneapolis were the ones who overlooked the significance of these particular alerts (for the reason stated in the Reuters and The Registers reports).
"The Reuters story reported that the Bangalore team forwarded the alerts directly to Target headquarters. The personnel in Minneapolis were the ones who overlooked the significance."...
...Understated but key point, . The outsourcing wasn't the issue here, but it did bring about a breakdown in communication, presumably because there was no one to follow it up back home... or the message was misunderstood.
This case should be studied ala Air Crash Investigation, with other corporates taking note. Wow, hacking has become so pervasive, all the doomsayers were right, and things aren't improving either! At the rate we're going, where will we be in 10 years? 20 years? The NSA and 5-Eyes should be focusing their clout tracking down these guys, not dragnet spying on the masses!
Not listening? Or not understanding. I've had phone conversations with support based in Idian for quite a few products over the last 20 years. I've found that some of them, while claiming to speak English, have never seen a US (or UK) pronouciation guide. Their grammar might be absolutely correct, but understanding the actual words is sometimes impossible.
I had a call yesterday with tech support for a very good, useful product and I'm going to have to wait for the email report because I don't know what they guy was saying.
I used to have this problem with Cisco and RSA support until I realized that I could get some sleep while waiting for their Australian call centers to take over from India.
The reports were sent by email. Not sure if phone calls were made, I assume that happened as well. And there was at least a couple day lag between the first event and the second event. Reports are even if they had acted after the second event they would have stopped the exfiltration of the captured data.
I'd prefer insourced teams myself. But the kind of crap that happened at Target is one of the bullet points the outsourcers use against us.
Err no. Not this sort of circumstance. Whatever the level of English of the outsourced staff the context is one in which a very limited range of language would be required, very context specific. Which is also why air crew and air traffic control the world round can communicate. They might not be able to to discuss politics or films, but they can understand "There's a plane coming towards you!"
Unfortunately, it probably WON'T be the most correct person, who is probably far enough up the corporate ladder to be "protected" from such disciplinary measures. Some poor middle manager who did his job and passed the warnings up the food chain will wind up taking the fall "for inadequately identifying the potential danger" or similar BS. And the REAL problem will remain unsolved.
At least get your politics correct.
Crimea want to secede to Russia. It is what the majority of people living there want (or at least will be voting for). It is the rest of the Ukraine that is saying they can not. Think UK/Northern Ireland/Eire instead of Ukraine/Crimea/Russia.
It's not even "pro russian" versus "not pro russian". It's ETHNIC russian. It's about what country are your grandparents from. That's an issue entirely orthogonal from what particular political option you are for or against.
Even if you conflate the two, you still have the problem of a very large ethnic minority (or a group of them). It is likely not as simple as some people like to make it out.
> Crimea want to secede to Russia. It is what the majority of people living there want
Yeah. Like you're such an authority on what a bunch of people in a different country across the continent want.
The whole situation is fishy. Hooligans are running amok. The Russian army is running amok. Enough nonsense is going on that you can't trust a thing that's going on right now. It doesn't help that their parliment was overrun by hooligans.
Target isn't really a supermarket in any dialect, but it's also not really a department store, at least in British English. Department stores like John Lewis and Debenhams are relatively classy places. Target is a big shopping shed with a product range something like a combination of Tesco, Currys, Argos and Primark.
" Target isn't really a supermarket in any dialect, but it's also not really a department store, at least in British English."
The same is true in USian English. Target's ex-parent-company Dayton's started out with department stores. The Dayton's stores werw sold off, and after a series of acquisitions the Dayton's stores now say Macy's on them. At least in the US, Target is usually referred to as a "discount general merchandiser" or something equally awkward.
Really if you want to be totally accurate, it's were the slightly more affluent white trash go shopping.
Not really. If you want to buy things like cookware or soap or bed sheets or coffee makers in a lot of areas in the US your choices are Wal-Mart, Dollar General, or Target. There's simply nowhere else in town to get these sorts of things and Target is the best choice of the three by far. Unless, of course, you abandon brick and mortar stores entirely and order everything online.
At least in the US, Target is usually referred to as a "discount general merchandiser" or something equally awkward.
Agreed. "Discount retailer" is the term I think I see most often. Sometimes "upscale discount retailer" to distinguish it from Walmart and the like, as Target, er, targets a somewhat wealthier consumer. (Target's stores are less crowded and more lavishly decorated than Walmart's, and they carry more designer-branded merchandise.)
Of course, being Of A Certain Age, for me Target will always be "the store where Jennifer Connelly rode a rocking horse". Ah, youth.
Think you will find Herods were kings in ancient Judea (four of them) and don't have a department store in London!
There is however a Harrod's department store in the west end of London which sells some fine (if overpriced) merchandise.
Given the biblical nature of your name I am surprised at you confusing ancient Jerusalem and modern London.
Think of Target as equivalent to Walmart - except with a smaller selection of the same cheap merchandise, and most of it priced 30% to 60% higher than Walmart.
Actually Target generally carries slightly higher end brands than Wal-Mart. Which is not to say it's high quality stuff by any means -- it's not hard to get higher end than the rock bottom garbage Wal-Mart sells. A bit of anecdotal evidence from my experience: the last pair of work khakis I bought from Wal-Mart, at $25, lasted about 4 months before they deteriorated to the point that I was embarrassed to be seen in public in them. The last pair I bought at Target, for $40, have lasted me 2 years so far and show no signs of wearing out any time soon. Where there is direct crossover between the two the prices are comparable, at least for my local stores.
On the other hand if the security software was chucking out millions of false alerts it's not surprising if the one correct one was ignored.
An employee raising concerns which were apparently ignored sounds bad but if the employee in question was always spouting off about one thing or another or if they were in a blame culture which meant that people always "expressed concerns over security" in order to cover their backsides then I can see how it would be ignored.
Of course it could just be that they screwed up :-)
re: Cubical Drone
I think you're incorrect, but it depends on what you call a 'false alert'
The problems arise when you try to add security after the fact. If the security device/appliance/etc is in place before the applications are designed, written and deployed, there won't be any reason to dismiss alerts "because that's how its supposed to work."
If you use 16-digit numbers for transaction IDs, some of them are going to be flagged as credit card numbers, and you can't tell the device "don't scan the transactionID field' because then you have a nicely-defined hole that will never be inspected.
Agreed.
Try this at your next meeting:
"Please raise your hand if you have ever heard a car alarm. OK, now, keep your hand up if you have heard an alarm when a car was broken into or stolen, otherwise put them down."
They'll all have their hands down at this point.
"Please raise your hand if you have ever heard a car alarm. OK, now, keep your hand up if you have heard an alarm when a car was broken into or stolen, otherwise put them down."
They'll all have their hands down at this point.
I lived in Boston for a few years in the 1980s, and I frequently heard car alarms; and while many were either false positives (motion sensors set off by trucks driving past or the like) or not indicative of significant crimes (set off by college students rocking the cars, generally, which might technically constitute disturbing the peace or the like), a great many were quite correctly attempting to inform someone that the car was being broken into, either to steal the stereo or to steal the car itself.
Stereo thefts were so common that many people, myself included, had cars broken into even after the stereo had already been removed. I had the first stereo stolen out of my car just a few weeks after I bought it; when I replaced it, I didn't bother mounting the new one properly, just wedged it in with some spring clips so I could pull it out and hide it under a seat or lock it in the trunk when I parked. Didn't stop some idiot from breaking the window and getting nothing for his trouble, though.
I had a friend with a ragtop convertible who taped signs to the windows reading "doors are unlocked", in the hope that thieves would read them and try the doors rather than just slashing through the roof. (It worked in all but one case, if memory serves.)
So, yeah, my hand would stay up. I'm no fan of car alarms, and they seem to be pretty much completely ineffective at deterring crime, but that doesn't mean they're always false positives.
It really wouldn't surprise me if that was the case, some of our high quality, lowest bidder, outsourced to india, code throws errors non stop as part of its "normal" operation.
ie you could have up to 4 customers on an account, so lets just fire off 4 'get customer detail' requests, with 3 of them unpopulated for the 99.9% of cases that have just 1 customer. These all generate http 500 errors along with xml parsing errors due to shoddy error handling, for an unpopulated request.
You might be surprised to hear, we suppress alerts on http500, and XML parsing, errors as the devs consider it to be "working as designed" and won't fix it.
Given FireEye is reported to have a graded alert system, I expect there is built-in triage to filter out noise.
I won't rule out a Too Much Information problem, but to the extent it exists, it was likely a management failure on the Target side. Yeah, I've worked in such environments.
We brought it because it checked a box on an audit check-list - doesn't mean we costed using it, monitoring it, nor following up every false positive to the point where we trust it.
Same goes for AV, we expect it to do it's job - nobody monitors the logs or follows up every log incident!
We don't even have the competence to keep our systems patched an up to date - so can you really trust us with big-boys toys??
Let's get real with what is the norm in 99.9999% of companies out there.
Indeed. This seems to be the motivation for nearly all of our customers who use our SSL/TLS feature, judging by support calls. ("What's a certificate again? Why do I need to configure this stuff? Can't it just work automatically?")
Look, they said we needed security, so I clicked the box that said "Turn security on". Done!
If the article is correct, this was systemic failure. The whole security apparatus failed: Test and evaluation Audit/ Compliance failed to recognize the proper security controls were not implemented or working properly; CIRTs/ Security Controls failed by not strictly recognizing, developing, applying or enforcing the the appropriate SLA,
The Confidential, Availability, and integrity of Targets systems requires a systems approach; and the development / implementations of proper policies procedures and standards that I believe could have prevented this from happening.
Once upon a time I worked for the US DoD, which gradually established Standards for pretty much everything, accompanied by large volumes of Requirements (e. g., Security Technical Implementation Guides) for all sorts of things, along with similarly high volume compliance checklists. A SA or DBA would be expected to be familiar with thousands of pages of such material, and to maintain all systems, applications, and databases in compliance with the STIGs based on the checklists, a substantial part of which required manual checking. "STIGging" a new system image, application, or database typically added weeks to deployment time. Since the agency started out relatively noncompliant, and requirements were growing the workload just for compliance assurance was large and growing.
Meanwhile, the agency was being compelled to grow negatively at about 30% annually, the IT staff being expected to participate in that proportionate to its numbers. That, and the resulting workload increase for remaining employees, did wonders for morale, as did the agency director's statement at an I&T "all-hands" that "IT is not part of [deleted]'s primary mission."
I seem to recall seeing that Target's IT solution was a Microsoft poster child having cost savings as a key attribute. The on site IT staff might have had an imperfect understanding of the operation as a whole and its security configuration and management in particular (or maybe sourced that out as well).
But the I&T cost reductions flow pretty directly to the bottom line on the financial report.
I will say this much in their favor: I gave up on this kind of system a long time ago. Every one I've ever worked with will flood you with false positives to the point that you'd never know it if there was a real one, even on the least paranoid settings. An admin can't be expected to take an intrusion alert seriously when they've been getting 10,000 intrusion alerts every day for a month.
Yup. So many times.
"We got error on systemxxxx. We rebooted it and restarted services. Please do the needful."
By which time you can no longer triage/root-cause the error because they rebooted the damn thing.
I wonder if the same thing happened at Target, where the info they needed to diagnose things and figure out if it was a real problem was not available.
It's not the credit cards of Target Executives that were affected, it was just their customers. All that matters is that the customers have spent money with them and they would have stayed silent if they thought they could have gotten away with it without someone whistleblowing and telling the world at which time they would have been subject to angry customers that were lumbered with dealing with their credit information being ripped off and the court case that would have ensued that would have cost more than the paltry 12 months of lifelock of whatever credit fraud prevention they gave people.
BANGALPORE?!!!
Well, that explains it! Target got EXACTLY what they paid for.
That does it. I'm never shopping at Target again! Not I shopped there often. Their prices are too high, and this is hardly the first inkling I've gotten that they just don't give a ___. For instance, they are always automatically bagging purchases, not just in plastic bags without asking you or charging you for them, in violation of law, but they DOUBLE them! And the checkout clerks are very slow.
Honestly didn't occur to me they might have been imported from Bangalapore.
"FireEye's technology could have auto-nuked the Target malware but the functionality was disabled"
How did FireEye's technology detect malicious software being installed on POS devices. If they detected the initial breach within two weeks then how did the crooks manage to clone and distribute the cards in such a short period.
Well, from a person I know who worked at Target Corporate in MN, the vast majority of their IT infrastructure is outsourced overseas or contracted domestically. Very few people are staffed in-house, which in my professional opinion is a huge mistake for any corporation to make.
Whether the offshoring played a direct role here is a moot point, the bigger point is that not only did some dumb bass decide giving an HVAC company network access was a good thing, but that the infrastructure as a whole did not act on alerts. And really, I don't want to hear the argument of "but, but, but, it's the black season! It's like pulling the starting rotation just as you get to the world series because one tested for the flu!" Bull, full stop overnight lock down, deploy security teams to assess, neutralize, and secure the problem, and by opening East Coast time, the situation is resolved and people can go back to their consumerism.
This statement, "The issue is complicated by the prevalence of false alerts from security technologies," should be near the top of every story covering this security disaster. Security monitoring is not only "complicated" by false alerts, it is turned into a nearly impossible job. Automated action might well catch all malware, but will also shut down the company every day.