Pwned so many times - but saved by the incident response plan

Companies that are more proficient with technology are more likely to believe that their security is "very effective". Is this a form of contempt born of familiarity, or a true understanding of the risks? The bigger the company, the harder they fall, and no organisation – not even the US state department – has proven …

[ADC]

Design for failure

The best, most robust systems are those where the designers expect failures and add features to deal them when (not if) they occur. It does not matter how low is the failure rate, when systems run continuously then eventually those rare failures will happen. This applies in IT just as in electronics, industrial controls, or other disciplines.

[Anonymous Coward]

Re: Design for failure

Indeed, and when failure does occur it will still be unexpected. That is my humble experience.

Normally I start with a fairly paranoid/pessimistic outlook on how well things will work, try to break them, fix the routes I used to break them, and hope for the best when deployed. Then they get broken in ways I had not thought possible, so back to the start again...

[Terry 6]

Life skill

This has to be developed in every one of us, and every user, from 5 year olds upwards.

Anything that you can get to on a computer, someone else will be able to get to, maybe many someones.

And it can go wrong, or be lost.

Which means that anything potentially of any kind of value needs to be stored in more than one place.

And anything that can be used to harm you needs to be separated from anything that allows it to be used.

Not just your network, but your personal stuff.

Don't put your home address in your satnav, with a button labelled "Home".

You don't need it - anywhere close will do. But if it's there then the car thief will know where you live - and that you're not in because he's nicked your car.

The homework that your child has spent five hours on and is needed for her GCSEs mustn't just be stored on her memory stick - it will break or be lost, Make a local copy, and a cloud copy ( or two) so that it can be retrieved in school, as well as home.

That small business' database that lists all their clients should be backed-up into an encrypted folder and saved on and off-site, on DVDs maybe, or a portable drive and in a form that stops anyone identifying the program that opens it, if possible.

And so on

It's a skill for the 21stC.

[Anonymous Coward]

Re: Life skill

> Don't put your home address in your satnav, with a button labelled "Home".

> You don't need it - anywhere close will do. But if it's there then the car thief will know where you live - and that you're not in because he's nicked your car.

And for many people, he's got your house key on the same keyring as the car key he's lifted from you. So not only have you provided the transport, and directions to get there, but also the means of entry without arousing suspicion !

If someone hits "Home" on my satnav, and follows the route - they'll find themselves outside the local Police station :-)

But that's an aside. I'm fully with Trevor - you can't keep on top of all this, all you can do is be "reasonably" secure given the nature of what you are trying to protect and the resources you have available to you.

Assume that at some point you'll lose - and have a plan in place for how to deal with that.

In the first instance, as Trevor points out, you can limit the damage done. Prevent all non-essential outbound connectivity - so the malware can't phone home easily and/or can't send spam and/or can't make connections to attack another system* (whether your own or someone else's).

And have multiple backups so if you have to you can just nuke it and go back to a clean copy.

* Eg, normally there is no need for a web server to be able to make outbound connections other than a few pre-determined ones (such as to your backup appliance). So if it gets compromised, and the malware is configured to try and brute-force accounts on someone else's FTP server - the connections will just fail.

I consider this good netizenship - try and make sure that you aren't part of the problem when you do get hit - pity so few other practice it !

Of course, if it's "state level" actors, then they can bypass all your technical protections by just walking in the front door and "asking" for the information they want.

[Chris Miller]

It should start with a risk assessment.

What have you got to protect?

How valuable is it (to you or to someone else)?

Where are the threats coming from?

Then you're in a position to decide what to do about it and how much it's worth spending on protection.

If you're holding data that could attract the interest of state operators, as an SME you're basically stuffed (unless, perhaps, you can get assistance from your own state operators), but most SMEs won't have (maybe in the case Trevor quotes, your IP address differed from that of the intended target by a couple of transposed digits).

[Terry 6]

Re: It should start with a risk assessment.

I've trained in RA.

It's good, even essential. But it can blind you to risks that you never thought of.

So for making sure an office is safe it's the essential tool. But when the risks may be unpredictable it can be a trap.

[Anonymous Coward]

It's definitely not paranoia when we can all pull out log files full of people out to get us.

[Alister]

It's definitely not paranoia when we can all pull out log files full of people out to get us.

This, exactly.

Every day, in mail logs, web logs, FTP logs - in fact everything that listens to incoming connections, you can see the background level of malicious connection attempts. Most are at the silly script-kiddy level, but you'll probably get at least one serious attempt a day, from somewhere.

One of our Directors overheard a colleague and I discussing one such script kiddy attempt - we were taking the piss out of the fact he was trying to find aspx files on a Linux PHP server - and the Director was horrified, asking why we weren't doing something about it.

He had no understanding of just how many attacks go on, day in and day out, and yet he's the one who normally queries why we need to invest in expensive firewalls, IDP / IDS systems, etc.

As an SME Admin, I do my best to maintain a robust and secure environment, but I'm well aware that at some point, we are going to get pwned.

We've had one incident, where a junior developer put up a web form without sanitizing inputs, and it only took a day before someone had successfully re-written the content of the site's CMS.

In another incident the Web team wrote a comments page without a capcha on it, which allowed anyone to type in an email address, and some text (not checked) and press send, and it would email the address given - an automatic spam machine, which was discovered by a bot within hours.

All you can do is try, with the resources available, to keep on top of things, and accept that despite all your best efforts, you are going to be hacked at some point, and if they're good at it, you may not even realise it.

[Pookietoo]

You might write "overheard ... I discussing", but do you think perhaps "overheard ... me discussing" sounds a bit better?

[Alister]

Interesting pedantry, thank you.

Taken in the abstract, it is grammatically correct to say "a colleague and I discussing " and not "a colleague and me discussing" where "a colleague and I " are the subject of the verb "discuss"

In the context of the sentence as written, "a colleague and I" are the object of the verb "overhear", and therefore it should be "a colleague and me" as you say.

However, because there are the two verbs in the same sentence, it is unclear to me which is the correct usage, and to me it felt more natural to use "a colleague and I " as it would if I were to speak it out loud.

[Milo Tsukroff]

Bravo - well said

Well said - I thought it was just me that feels like I can't keep up. Well it's true then. Just do the best I can and reach for the phone to call an expert when trouble happens. Great article and another reason why I keep reaching across the pond to read The Register.

[Anonymous Coward]

Our company believes it's never been hacked.

There is no dedicated IT Security staff, and from personal experience I know that there are servers that have gone over two years without patches or updates.

But if you don't know it happened, I guess it hasn't....

[Anonymous Coward]

Excellent

Very well said.

[Unicornpiss]

Applause

Thank you for a heartfelt article that I can utterly relate to. There may not be another field in the world so much as IT where if you don't actively keep up, it will run you over and leave you squinting at the mud-caked license plate of the bus that hit you.

Many of us are so overworked and stressed anyway that things we'd all like to accomplish like an artisan putting the finishing touches on a Stradivarius, instead we end up doing like a coked-up plumber frantically patching leaks and plunging stopped up toilets. (I have many times described my job as similar to one of those entertainers that manage to keep a number of plates spinning on sticks by running frantically back and forth) There is never enough time for continuing education and researching problems.

IT is so often treated like the proverbial red-headed bastard stepchild. Even though most modern businesses cannot exist or at least thrive without a competent IT staff, IT often gets the least staffing, the worst budgets, and the least regard from business leaders. Yet when something major does go wrong, there's plenty of blame to go around. Most of us wouldn't build a building without a solid foundation, yet IT is allocated sandstone to work with.

There is probably no other field except Medicine where there is such a diverse array of specialized jobs, yet often we are expected to be Swiss Army knives where skills are concerned. And frequently we have to guard against attacks from outside, from within, and occasionally from management.

I'm sure we will all be Pwned at some point. (if we haven't already and just don't realize it) I think I started out with a point to make, but it's been a long day of madness and I'll just wrap up by saying thank you again for an excellent article that hits most of us where we live.

[cortland]

Re: Applause

IT shares with QC the lethal defect that it can't MAKE money, only SPEND money. Or course, management would call *air* part of the employee compensation package if they could get away with it -- and some day (in orbit) they will.

FWCUMI ( http://www.workplaceinsanity.com/2010/08/floggings-will-continue-until-morale.html)

[Medixstiff]

"Companies that are more proficient with technology are more likely to believe that their security is "very effective". Is this a form of contempt born of familiarity, or a true understanding of the risks?"

No it's stupidity, our rule of thumb is if someone wants in, they will, given enough time.

Which doesn't stop us from employing every trick in the book to protect ourselves from both internal and external threats.

[Henry Wertz 1]

"The shiny"

I've seen this certain type of IT person that I think the article is referencing.

They want to use "the shiny", whatever the latest and greatest technology is. Sometimes, this is perfectly appropriate -- if I were running heavy number crunching, whether I use CUDA or not I would at least want to look into it. Sometimes it's totally unnecessary or even harmful, using a technology or software that may not even be the most appropriate for what they are doing just because it's new and fun to play with.

Some of these guys also can go as far as developing this exaggerated view that whatever new technology they are looking into will automatically better... it (in there view) will use better, modern programming techniques and so avoid bugs, and even avoid security holes, compared to whatever older technologies or softwares they are comparing it to. Of course, this may occasionally be true, but usually it's not... if anything, the newer item will have more bugs simply due to the bugs not being found and patched out yet.

Finally.. of course, even if the security of the software is quite good, it's always good to layer on some *extra* security and logging. After all it only takes one security flaw for someone to thoroughly pwn your system. For instance, when I used some modules in a web project that were supposed to sanitize the inputs, I "pre-sanitized" the input anyway. So months later when an "oh no! This module's sanitation lets these couple items through!" I "laughed all the way to the bank" since the pre-sanitizing I did already filtered these items out.

[Anonymous Coward]

IT - work like a slave and plan like

Your entire resource pool is sitting under the proverbial coyote's anvil.

I've been doing this for 30 years now. From cabling things, to mainframes, to networking to storage and *nix coding to SA and others.

I've seen a crapton of things and forgotten more than I stand a chance of recalling. What scares the most crap out of me is that I've been outsourced to "MegaITCorps" twice in my life and goddamn it, the Really Big Guns in the industry typically have worse security than the little guys, not due to lack of knowledge, or due to bad SA's. Its due to trying too damn hard to do security, and making things so complex, long winded, and difficult to execute, that the SAs, programmers, architects etc develop a culture of working AROUND the security rules. Too many rules, too many security tools, and too many demands to meet schedules that were not realistic with functional requirements that were bolted on after the initial project plan.

KISS. Plan for the worst. Automate deployment. Automate basic SA use cfengine, puppet, chef, whatever, Global Policy, LDAP, centralize syslog and BUY a decent log miner. Kill the snowflakes before they land, because those jackasses are the reason systems will get compromised. Automate backups, and use *smart* rules about them - and then post those rules in big block capitals where the devs, the SA's and the users can all see them every day.

In truth, I don't think that putting your full set of security tools *on* the hosts you are trying to keep healthy is necessary, IDS can be done (given the funds I suppose) from the edge of the network and using a separate network for control. Yes, AV/AM software on windows systems is needed, and if your *nix hosts are doing SMB/CIFS to the windows world then appropriate tools are appropriate - I've seen a scanning set that kept the software on a single host and essentially copied the binaries over each time it ran scans on a node - but that was a fairly small network of nodes and the tool fit that circumstance. CODE REPOSITORIES !!! dammit - if you're developing stuff (any stuff) in house, use SOME sort of code repository that tracks who did what when and what code they changed. And lock it up tight, back it up daily and use it.

Part of the issue I've seen is that frequently one needs different tools on different platforms to do what is essentially the same job, which complicates matters, and prevents "single view" - it makes it quite hard for some folks to see what happened and when in relation to what other events.

But yes, Trevor, assume you will get hacked. Essentially, one should plan to fail and then not have the tag "failed to plan".

[FreeTard]

Excellent, well written and honestly refreshing article.