I cannae dae it, cap'n! Why I had to quit the madness of frontline IT It took a massive hack attack against the United States government resulting in the theft of up to 14 million 21.5 million records to make me realise why I want to quit working in IT. Over the past year I've significantly drawn down my involvement in day-to-day IT operations, and I'm much happier for it. The US Office of …
...the problem is not management, it's the IT staff. The author was unusually insightful when he says he believes he's more like LaForge than Scotty. We ALL need to be more like Scotty.
We are a logical, methodical, analytic group, so let's analyze this: These things we know -
- management/business types will automatically cut our budget by 30-50%
- our deadlines and resource requirements will be similarly cut
- we will be dealing with this 'solution' and it's issues for a long time
Assuming these facts as stated, wouldn't the following be a logical response:
- Increase budget estimate by 45-80%. Resulting budget after 30-50% cut will be 90-125% of actual need.
- Increase lead times and resource requirements similarly to result in similar results following cuts
- Always include professional services, training, documentation and knowledge transfer in budget
If we ALL decide to adhere to these principles, then there are no options left to the business types. Anyone not experienced enough to follow these guidelines will regularly fail to meet deadlines, SLA's and budget limits. They will have a short career and remove themselves as competition.
It's in the hands of the geeks to change our lot and that of our fellow workers. Only we can change what needs changing and it's we that need changing, not the management team. They do what they do and we need to acknowledge this and act accordingly.
Oh, and document EVERYTHING including any objections or warnings when your budget is cut too much to do the job. It's my guess that Katherine Archuleta is either being paid VERY well to take the abuse she didn't deserve or she did NOT document her concerns, objections, warnings when her budget was cut to a level inadequate to the job.
Remember: "It's not them, It's us!" - We can't change them, we can only change ourselves.
Just my 2 cents.
There are three stages here. 1) Learning the truth. 2) Accepting the truth. 3) Being in a position to do something about it. It's only in the past few years that I started to get in a position to exit, and doing so without screwing over some good people in the process took time to orchestrate. I'll not dwell on how long it took me to go from "learning the truth" to "accepting the truth" because that's more than a little depressing.
Alas, the root of the problem is that the majority of the people who control the money (managers, investors, VCs, and legislatures) did NOT get where they are by being rational about what results can be expected for what cost and schedules; instead, they got where they are by twisting the dial to 11 (and gluing it back on when it broke off the shaft), getting good-looking results for long enough to be promoted/bought out/off/elected, and running away from the inevitable wave of catastrophe that follows at a distance behind them wherever they go.
There ARE leaders who don't work that way, but since true leadership ability is much rarer than bluster, bluff, and amorality atop a crumbly foundation of short-horizon opportunism, such leadership is unfortunately rather uncommon.
It is useful to think of it as another version of the tragedy of the commons.
GigglyPuff, you forgot a very important point - get the people that work at the coal face up to the top floor to make sure that budgets and time scales are adequate.
The only problem is to ensure they keep up with what is going on, which is why, although I own the company, I do regular night shifts and go out on service calls just like my engineers. I need to know just what my staff face and what help they need to do the job.
I learned all that early on when I worked in engineering before I worked in IT. It did cost me some jobs where I'd tell management what was needed (with the requisite padding for negotiation). They'd start slashing. Eventually, I'd say "you're out of your mind... fast, cheap, and works... pick 2". Being shown the door because they said they'll find someone who would do it on their terms wasn't so bad. I heard stories afterwards about a guy who replaced me at one place. All I can say is, I was glad I was gone.
The idea that the people being thorough, accurate and honest are the 'problem' is perverse.
The problem is with the 'management types' not understanding how IT is not just another cost-centre and can't be treated the same way they treat the other areas of the business.
Perhaps more IT people should learn to speak 'management' more fluently but it is, again, perverse that this gets so turned-around that it becomes IT's responsibility to understand management rather than management's responsibility to understand the operation of the component parts of the organisation they are supposed to be leading and guiding.
Of course, in any situation you have to evaluate what it is that you can do to achieve the results you need, but that does not effect the kind of systemic change that is needed.
I join the list of agreers, but having fell into this industry as one of the Sinclair generation (there was nothing else that was going to happen for me, considering the economic situation at that time), I am vocationally a square peg but I've made the best of it.
I've done my time as one of the blind ambitious and discovered I really didn't like the dangling Damocletic blade and the best skill I ever learned to survive in this industry is how to find a quiet corner from which I can focus on doing my best work. I don't stick my neck out any more, I just turn out good stuff.
I dream of the day that I step off the carousel and fill my life with music and travel, I just hope I'm not too old and still have my health when that day comes.
Working on it.
<snip>
"I dream of the day that I step off the carousel and fill my life with music and travel, I just hope I'm not too old and still have my health when that day comes.
Working on it."
I did that years ago and I am all the better for it. I am sitting in 40 degree heat, filing pictures from the past month that include trips to London, Cyprus, Spain, Gibraltar, Lisbon and Madeira.
Nowadays I have to dweal with complex issues such as what day is it? and what shall I make for breakfast. Life is indeed good.
"Perhaps more IT people should learn to speak 'management' more fluently" An issue with that is that whilst Managementese and ITese share many common words, they apply to different meanings and concepts. Where in IT we say something like ............ ahhh, who cares, Simon Travaglia and Scott Adams say it all anyway.
I think that the take home message for all of you stakeholders out there (management speak, like it?) is that when an Oracle like one of the afore mentioned give forth, we should listen and pay a dutiful observance and then get out the carpet, shovel and quick lime.
No other part of the company is just another cost center. Each part thinks it is unique and ought to have special consideration. But the other parts of the company eventually come to understand that they have to communicate with management and understand the operations of the company. Only IT seems to persist in the belief that management needs to learn to speak its unique language.
This is our, or perhaps your problem to fix, not management's.
@tom 13.
Spot on. The change has to be driven not expected.
I've worked across disciplines and every dept wants (needs) X to do y, and they rarely get it. One difference with IT (generalisation warning!) is the response that 'management' doesn't understand. In most cases they do understand but have more important priorities. Saying that they don't understand when really they just don't agree with you won't get you anywhere.
Sitting in your darkened corners or bitching on blogs won't change a thing. Get yourselves into management and drive the change.
I thank you all for reading this and take the downvotes as a compliment.
Step 1 - Management asks IT to ensure systems are secure.
Step 2 - IT presents researched, costed solution to Managment.
Step 3 - Management requests system to be provide 40% cheaper.
Step 4 - 40% less-effective/resourced solution put in place.
Step 5 - Everything run perfectly in perpetuity - yay management!
Oh, wait . . .
Yes, management often have 'other priorities' but actually it's really just one: do more with less. That's great, from a business perspective, but sometimes more takes more. Or at least can't be done with less. Some things in IT absolutely can but security is not somewhere you can cut too many corners and security of a massive and very sensitive collection of data is certainly not somewhere you can afford to skimp.
Great that 'management' has other priorities but if security of sensitive data is not a high priority then their priorities are wrong. It's not: "hey, let's replace all out SANs with new flash-only arrays" or "hey, let's give everyone iPads" or "hey, let's upgrade the helpdesk ticketing system to make life easier for our staff".
>Perhaps more IT people should learn to speak 'management' more fluently
The issue is temperament. People drawn to IT usually like to be precise and correct. They are the kinds of people you want dealing with machines and data which require precision and accuracy. "Management" likes expectations met. This is how consultancies survive, you put a layer of Management between the techies and the customers' Management, who pad the budgets and the time-scales. Yes, it far more expensive and takes longer, but it gives the customer's Management a warm fuzzy feeling when projects come in "on-time" even if "on-time" is far later than a non-padded project would take.
Agree, I always overestimate the costs and time of a project, for those little niggles that always show up and when I hand back a bag of saved cash, the bean counters smile, the bosses smile and I have sneak off with a smirk on my face and a pat on the back. Unethical, probably, warranted definately !
The problem with inflating the budget is that the accountants now pore over every line, looking for accuracy of estimates. They do this because management believe that all budgets are getting inflated and in any cash-strapped times they want to cut back not just the inflation, but the actual real cost. So inflating your initial figures will eventually get found out (and more probably before work begins now than after the project fails - unless you're working on Universal Credit).
As for the main article I'm one of those honest types too, one who finds it difficult to stretch (or compress) the figures with fictions and make up a believable story to support them. Unfortunately, it seems being able to bamboozle management with smoke and mirrors and trick them into seeing wonderful $$$ where there are none is becoming more and more of a required skill for all jobs (of any kind in any industry) these days, not just the 'sales teams'. It seems that that kind of trickery isn't something IT types like those of us who read El Reg have aptitude for.
The problem with putting in an inflated budget, is that people know and expect that your budget is inflated, which is why they try and cut it down. It's a silly charade, rather like negotiating on a price for a car you are selling. You want £5000 for it, so you price it as £5500. Buyer knows you want £5000 for it so they bid you at £4500. You go through a ridiculous ritual exchange of numbers and end up at the £5000. Ridiculous. Just cut the BS.
Won't work.
I've been sitting on the other side of the table in a non-IT context. We knew the guy asking for money. We knew he was planning to overspend his budget by 10% figuring that was allowed. So we cut his budget by an extra 10% so that when he overspent by 10% he'd still be about where we needed him to be. The people who weren't planning to overspend their budgets by 10% got what they asked for, at least in as much as we could fund them in any given year. The thing about budgets is that the guys running the numbers aren't really setting them. They are reporting them. They know how much money the company is going to generate and they know what all the Wants the company has. But Wants always exceed Income, so something has to give. The budget planning process is supposed to be about doing that rationally.* Oh, and for the record, the guy we knew would overspend was in a high visibility, high PR, medium impact on on "business" position.
The budgeting guys are as smart as you are. And they're watching your behavior on the numbers, not the way you set your budget.
So if you're getting your budget cut by 50% and can't make do, find another job. You've either got an asshole for in finance and you won't be able to change it, or a problem elsewhere in your IT management system which you probably won't be able to change that either.
*To the best of my knowledge the only time we shorted IT was once when I overlooked one of the needs we'd have for storage on a server. As I was the one submitting the request on that one, I can't really blame the budget committee for that one. If I had asked for it, I think I would have gotten it.
@ Hollerith 1
"Do you not think that science and technology are just are riddled with politics as any other place?"
Sure they are - in universities and research centres - but that's why i don't work there...
University grant system is perfect to shady pseudo scientists and promotes the importance of "who can get funding?" instead of "who can do science?"
FOR THE RECORD, OPM has been telling its vendors to do things that it does not even do themselves for YEARS before this happened. I know because I have seen it myself.
It's right in their background investigator contracts to do everything with 2FA, Anti Virus, VPN, no browsing from work computers, locked down hardware and software, etc.
This proves that OPM was already aware of the issue and their inability to use any common sense on their own networks proves that they are incompetent bumbling fools all the way to the top level including their director and the man who is supposed to over see them, the President.
Fire them ALL!
An audit said, the OPM systems are horrendously insecure: turn them off. Archuleta said, we need them for work: keep them on. Then they got hacked. Then Archuleta goes in public and goes, at least her security initiatives let them detect the hack. A year after it happened.
The Malaysians with machetes that just take your fingers...
http://www.theregister.co.uk/2005/04/04/fingerprint_merc_chop/
The largest security measure ever taken (and which is clearly visible from SPAAAAACE) is the Gret Wall of China. It was defeated by bribing gate guards.
Ultimately there is no security that will resist a strong enough and capable enough force, but that'sw still no reason to not try.
No the largest security measure ever taken was the Berlin Wall and all the Steel Curtain installations. Although they were designed to avoid people getting out, not getting in, it actually worked well enough - very few were able to cross it, and all of them needed clever and risky efforts to achieve it.
It worked because it was continually improved - any "bug" that led to someone escaping was analized and fixed - even if it was expensive. Guards had better treatment than most people living there, so they had little incentives to be bribed (and punishment would have been hard enough to be another incentive), and were anyway properly "vetted" before being employed, to ensure they were loyal. Also, it employed an obsessive 24x7 continuos control.
Like the Maginot Line, the Great Wall of China (and other large defensive fortification) were designed to counter large dull frontal attacks, ignoring the cunning ones. The the Wall and the Curtain was designed not to counter a large attack force (atomic bombs would have taken care of it), but exactly to stop people "hacking" it and escape the "communit paradise".
In IT, there's exactly the temptation to build Maginot Lines - because that's what generals or executives like, but attackers needs just to find a weak spot to penetrate, and if behind the front line there's nothing, it's almost impossible to "counter attack". It becomes the equivalent of a "deep strike" attack.
"A state level attacker who has the capacity to subvert the firmware on hard disks, routers and the like in transit, if not before they leave the factory."
If that is sincerely your concern then working around those requirements requires controlling those elements of the supply chain yourself. Either by having the ability to write your own firmware/replace the operating system on your routers or by buying a firm who makes them from the ground up and rolling your own from scratch.
I never said it would be cheap. I said I could do it. And you know what? There are plenty of companies out there who make their own routers and an ever increasing number that make their own flash drives/flash arrays. That includes the firmware. So yeah, it's doable.
This post has been deleted by its author
This post has been deleted by its author
Yeah but how can you be *sure* that those companies haven't been got at?
By owning the companies. If you own the companies you own the code. Do external code audits...like you should be doing with all the code you own anyways. Never trust anyone. Not even yourself. Everything and everyone is a potential point of failure. Build as many checks and balances as you can with the resources you have. Then try to get more resources.
But keep off the "cloud", you don't own that and you don't know who does or who has access to it. And do your Chinese made routers have back doors? And is that a key logger plugged in to the back of your machine.
Oh God I'm so glad I don't have to do this anymore.
Type it on paper put it in a safe and throw away the key.
“Just because you're paranoid doesn't mean they aren't after you.”
― Joseph Heller, Catch-22
Very true Trevor but you must also look out for your team just as they must look for each other. If everyone backstops everyone else, while it might take a little longer to get the end results those results work. The client is happy, the engineers are happy and even the accountant is happy and on that happy note you get on with the next project expecting the same happy results at the end of it.
By owning the companies. If you own the companies you own the code.
You are absolutely sure that the VHDL compiler they used to cook up the masks for all the chippery or the microcode for your CPU is not compromised? That Japanese factory making the tantalum's didn't place a transmitter inside?
I'd say You can't. Even if you buy Intel, you can't. Even if you could build a trustworthy AI capable of holding all of Intel's design information in it's head and simulate it's operation with quantum-level resolution, you can never be sure. Because you would be long dead before it was done. There simply isn't enough time in the universe to x-ray every component, check every single bit, verify all code and confirm that all of the design tools are not lying or hiding information.
You have to assume that the operation is compromised and then work out what the consequences are and how to mitigate this.
You are absolutely sure that the VHDL compiler they used to cook up the masks for all the chippery or the microcode for your CPU is not compromised?
No. Which is why I talked about ensuring that you write firmware that presumes you can't know this and tries to compensate. By doing that you have done everything humanly possible and can stand up in front of a judge and say so.
"hat Japanese factory making the tantalum's didn't place a transmitter inside?"
Yes. Yes, this one I can think at least three ways to verify conclusively.
"You have to assume that the operation is compromised and then work out what the consequences are and how to mitigate this."
Which is exactly what I said. But you also work to minimize the number of different points of compromise so that you have fewer potential holes through which nasties can get you. Security is a comprehensive affair that should be done in depth.
This post has been deleted by its author
It is very naive to think that you can solve every security need. If it was that easy then big companies wouldn't be getting hacked. They would pay big $$$ for the black box to secure themselves.
The reason you can never be fully secure are the meatbags who write the code, they're not perfect and have faulty code. The meatbags who work for you, they're not perfect (and like to click on stuff), most places get hacked due to social engineering.
Remove the meatbags, do a full audit on all code (ALL OS's - servers, desktops, phones, routers, switches, etc), using an AI since those pesky meatbags had to be removed, to verify you have no new vulnerability whenever a new technology comes out and you might stand a chance of being "better secured".
Read the article. Nowhere did I say I would solve every security need.
I merely said that I could build the best network that has ever been built, if the resources were provided. That includes counters for every known security problem, policies/procedures that limit new problems for occurring, incident response plans to mitigate damage when breaches do occur and resolution plans to deal with breaches once they have occurred.
Now, bad code, state actors slipping things into hard drives/switches/etc...these are all easy to solve. Expensive, yes, but these are known issues that can be worked around. Automated testing can be built to look for them. Mitigation programs designed to handle them. If you know about an attack vector you can plan for it, assuming the resources are there to do so.
This includes social engineering. It even includes some thigns I can't talk about related to automated incident response because I'm under NDA with several companies developing next generation technologies.
Suffice it to say that yes, security is actually not that hard. It's spectacularly expensive, and the experts required to implement the things you need to be properly secure are in high demand, but it's all perfectly doable.
That's the problem. It is doable. Worse: I know how it's doable. I can detail for you every single corner cut, every compromise made, every bent copper clawed back in exchange for deepening the risk pool.
You can't guard against what you don't know, but you can absolutely can put in place mitigation and response, compartmentalization and...and...and...FUCK IT. ENOUGH! I'm not going down this goddamned rabbit hole one more time.
Look, companies aren't willing to pay money to secure themselves. Sony wasn't. The US Government wasn't willing to. Many health care providers aren't willing to. Over and over and over and over, up and down the whole damned list.
Every week I have sysadmins from the largest companies on earth telling me very blunt, honest tales about how they have raised flags about things they KNOW are issues, but which management utterly refuses to address. They want me to write about these things in The Register, but somehow keep them completely isolated so that nobody can trace the leak of info back to them.
Government malpractice? Pick a fucking country! SMBs? Cloud providers? SaaS providers? Startups? You name a segment, I'll tell you tales of cut corners that will make your blood run ice cold. Corners they know they are cutting, but take the risk to cut anyways because they delude themselves into thinking that the risk of incidence is low.
Christ man, you read about these things here in The Register every single week! It's now gotten to the point that most of us just tune it out because the frequency and scope of the digital apathy and ignorance is so astoundingly staggering that we, as pratitioners of the art can do nothing but weep.
Then we go to work and pretend that same restrictive penny-pinching bullshit approach to everything is somehow not leaving our precious networks vulnerable. Or we fellate marketing (and oruselves) with the trumped up idea that by using public cloud computing we will somehow offload all risk and responsibility to a third party provider, without, of course, reading the EULA which very explicitly is Nelson Muntz says "ha ha" with both middle fingers in the air.
It is not naieve to think that with the right resources a competent administrator can build the best network on earth. Not impenetrable, but damend close, well monitored, segmented, compartmentalized, isolated and with incident response for when it is inevitably compromised.
What is naive is thinking that anyone will ever be given even a fraction of the resources required to do so, or that any of us are even remotely secure unless and until we do.
And who takes the blame when the hammer falls? When you don't have the incident response you should have? When you are pwned by a known vulnerability, or you didn't have the latest security measures due to budget cuts? Your boss? Accounting? The shareholders?
Nunh uh.
You. The systems administrator. Every single person reading this comment does not have the resources to secure their networks enough to be able to stand in front of a judge and say "I did everything I could, your honour". The best that they can hope for is to document each and every incidence of resources being denied, log strenuous objections and keep paper copies of it all locked away in case you end up in front of that judge.
And if you don't? You just leave room for the attorneys of your employer to blame you. You should have known. that's your job. By not objecting you either didn't know - and are thus incompetent - or you didn't object, and thus committed malpractice. Either way, it's your fault.
But no, sir. Nobody is willing to pay "big $$$ to secure themselves". That's the whole goddamned problem right there.
This post has been deleted by its author
The stuff I've seen is a little bit more advanced than that. Some of it's in beta with stealth companies. Some of it is just tip-top-secret squirrel because it actually works for now and the current "big fish" customers pay the startups a truly obscene amount of money to ensure that only a very limited number of people have access to the technology. (I.E. not their competitors.)
Security products and services are a disgusting business. Companies aren't willing to spend a lot to defend themselves, but holy crap will they spend money to thwart their rivals.
This post has been deleted by its author
I'd guess at automated probes back, followed possibly by throwing a few exploits at whatever you detected
Nope. Nope nope nope. That is illegal on so many different levels I just...that's a great big huge bucket of nope.
"Incidentally, I've been hatching evil plans to get at your bombproof "networked product" factory ever since you mentioned the concept; favourite so far is build the nastyware into the case."
Why bombproof? Hardware is cheap and cheerful. Source from multiple suppliers. Write your own firmware. Have it checked by teams in different countries and foster some spirit of competition between them.
Expect that people will try to compromise your hardware (building nasties into the CPU/ASIC?) and try your hardest to write stuff that will detect it. Shut down if required, work around if possible. Have network gear that doesn't trust what's attached; always look for suspect traffic, etc.
No single point of failure. Not even in your supply chain. Someone bombs your motherboard factory? That's why you source from multiple places! Etc.
This post has been deleted by its author
"I meant 'bomproof' figuratively. With the sort of distributed supplier chain you're describing, compromising it would be relatively easy, if the assailant also had a decent budget. It'd be another "You have to be lucky all the time; whereas the other team would only have to be lucky once" scenarios. Depends greatly on what sort of product, who the customers are and what useful information is passing through."
Interesting viewpoint. I'd love to buy you a beer and go back and forth about it. I think there's merit in both approaches. There are just two different things that need defending against: 1) external actors attempting to "poison" the supply chain and 2) loss of one of more elements of the supply chain. (I'd argue that a "poisoned" supply chain component can be treated the same as one that is destroyed or embargoed.)
So what is the optimal distributedness versus risk solution? I could spend quite a bit of time in front of a whiteboard having fun with that calculation. :)
This post has been deleted by its author
This post has been deleted by its author
>" The stuff I've seen is a little bit more advanced than that. Some of it's in beta with stealth companies. Some of it is just tip-top-secret squirrel because it actually works for now and the current "big fish" customers pay the startups a truly obscene amount of money to ensure that only a very limited number of people have access to the technology. (I.E. not their competitors.)"
If you've seen this top secret stuff then write about it in detail, you're a journalist after all, otherwise it is just another fisherman's story. Without proof, it is meaningless drivel
Come on. How simple are you?
If Trevor were to write about this stuff, two things would happen.
1. He'd get sued for breach of contract (the NDA is a contract).
2. He's get excluded from this sort of information in the future.
In fact, he's probably on shaky ground even admitting that he's subject to an NDA, if they're worded like any of the ones I've been subject to in the past.
So if he did, he would be shooting himself in the feet, both of them.
If Trevor were to write about this stuff, two things would happen.
1. He'd get sued for breach of contract (the NDA is a contract).
Actually, some of the stuff that's out there might well never make the light of day. It's so very deeply hush hush that sometimes companies get bought up just to keep the tech from competitors. And most of it is damned good. I'd be far more worried about well paid character assassins basically ruining my life than any legal consequences. There are ways to perfectly legally ruin a man's life. I'd rather not attract that sort of attention.
2. He's get excluded from this sort of information in the future.
Which is why I comply. By being one of the few who comply, I get to have input in early stage products and help design go-to-market approaches such that there is at least a chance this technology will be made available to my people (the SMBs) at a price they can afford. I then usually have a shot at getting a launch exclusive review.
In fact, he's probably on shaky ground even admitting that he's subject to an NDA, if they're worded like any of the ones I've been subject to in the past.
This is really the weird part. Yes and no. I probably would get in deep cacapoopoo for letting on which companies I was talking to in the context of this conversation. But general innuendo that "I know people who know people who know things"? That actually works out rather well for the startups in question.
People who need the kind of tech we're discussing (or next gen storage tech, or next gen SDN tech, etc) e-mail me. I pass that info on to the startups and their folks do background checks and maybe reach out, get an early customer.
The world of stealth startups is weird.
If you've seen this top secret stuff then write about it in detail, you're a journalist after all, otherwise it is just another fisherman's story. Without proof, it is meaningless drivel
You're absolutely correct. You either trust me that I have seen this stuff, or you don't. It you do then you can trust that when I am allowed to write about it, I will. If you don't, then none of it matters, does it?
It is very naive to think that you can solve every security need.
I think you're missing the general trend of the article by focusing too much on the detail (don't worry, that's normal for IT people). Given enough budget you can come pretty close, especially if you can keep some funds aside to club the bastards who still find a way in (not all solutions are technical, and I have been in some pretty decent setups :) ).
The problem is "enough budget" - there is a prevailing belief that IT people think like political managers, and so get funds slashed because they were too honest, instead of being considered factual and get the funds.
I agree with this article, 100%. The one time I was able to put something together without anyone trying to slash my budget it resulted in a platform that is still running more than a decade later without failure, and without being hacked despite being a pretty juicy target. I've also seen later projects die because they only got 50% of what was needed which drove pretty catastrophic decisions such as the use of cheap coders, a mistake some of these projects and business are STILL paying for in downtime and code maintenance.
I like sleeping at night too, and I have 2 choices to achieve that: (a) no longer caring or (b) making sure it's done right. I'm not wired for option (a), but you don't always have option (b). That is what the article is about.
No they don't. Here's an economic truth for you that everybody denies:
Nobody knows how to make a pencil from start to finish, but the stores are filled with them.
http://www.joshharness.com/2012/10/nobody-knows-how-to-make-pencil.html
Same thing applies to electronics.
This post has been deleted by its author
This post has been deleted by its author
Except that in the scenario discussed the items are coming from the factory to you for internal use. You can then reload clean firmware (actually, every end user should be doing this for every product anyways) and running your test suite on the product before you deploy.
Since you own the company that designed it and employ the devs that code it and the multiple teams that develop the testing suites there should be no way for tinkering to go undetected.
This post has been deleted by its author
I was talking of spiking your product after it has left the factory en-route to the market. You have to do it sometime and that's a weak point.
Except it's not. If you have known-clean firmware to load on the device after it arrives at it's intended location and a set of test software/hardware devices to look for compromises then it doesn't matter if someone compromised it en route; you'll be able to detect that because you know exactly what it is supposed to behave like and you can both load clean firmware and test the behaviour to all conditions.
You seem to feel that owning a company is some sort of ward against evil. I would suggest that that is not the case.
And you're wrong. Owning the company means you own the designs of the hardware and the software. You can have those designs and software independently audited. You thus know what clean firmware should look like and how devices should behave under all circumstances. So even if you are compromised by the workers in your assembly plant, or compromised en route you have the means to test for compromise and even to potentially correct it.
There is no requirement to trust anyone. In fact, if you are doing it right, you don't trust anyone. You establish multiple teams each in different jurisdictions, each serving different masters and with different personality mixes whose job it is to find flaws and compromises. There is no single point of trust or failure throughout the entire procurement process.
You're outsourcing all over the place and each time you do that you're introducing potential holes. It gets to be a fractal coastline thing very quickly; impossible to police properly
Wrong. My procurement design doesn't rely on trusting anyone. It doesn't even need to have many suppliers. Two for any given component will do, each with their own logistics arrangements. Two (preferably three) teams of auditors auditing firmware, designs and creating test suites, each in different jurisdictions. Done. The chances of compromising all of them is effectively zero, and you can always add a hidden auditing team that the rest don't know about for extra paranoia.
When stuff arrives onsite you look for compromises. You absolutely don't trust that it somehow wasn't compromised en route.
If you're assuming an essentially unlimited budget then you have to assume that for your potential attackers. How about if your opponent had a secret chip fab and somewhere to make custom batteries? They could get up to loads of shenanigans
I do assume an unlimited budget for potential attackers. if they compromise things, regardless of if that compromise is done in firmware or in hardware a proper suite of tests designed by people with access to the original designs of the hardware and source code of the firmware will be able to detect the compromise. Doubly so if there are multiple independent teams who are creating separate tests.
You seem obsessed with the idea of prevention as the sole means of doing security. This means you are terrible at security. Nowhere are you looking at detection, monitoring, mitigation and incident response. The fact that an item might be compromised isn't the end of the world. You simply need to detect the compromise before being put into production, or during production if it slipped by. You need to mitigate the damage that any compromised item can do and you need to plan for the fact that you will inevitably miss some and how you will deal with individual breaches.
I can reduce the possibility of compromised equipment being put into my datacenter to damned near zero if I own and operate the supply chain. (Prevention)
I can detect almost all compromises that do slip through by having access to the hardware designs and the source code of the firmware and software. (Detection)
I can further reduce the impact of any compromised equipment by designing my network so that no individual component or piece of software has access to all of my data. (Mitigation)
I can continuously test and monitor all equipment, firmware and software to ensure that it is behaving as expected and immediately trigger alerts if it does otherwise. (Monitoring)
I can immediately lock down my network and trigger security audits and/or alerting of authorities/customers/insurance/etc if a compromise is detected. (Incident response)
You will not catch 100% of incidents this way. You will catch 99.9999%+ of incidents this way. You will also be able to stand in front of a judge and say that you have done literally everything humanly possible to protect your customers' data from harm.
What's more, that's all perfectly doable on the budget of a mid-sized enterprise. None of it requires "unlimited" budget to accomplish. None of it is even particularly hard.
You just have to understand something about information security and best practices, which your comments lead me to believe you do not.
This post has been deleted by its author
What I thought we were doing was wargaming a 'perfect' networking product to look for chinks as a fun thought experiment. Bear in mind that I do not know the size; shape; purpose or likely customers of your product; nor do I have any idea what precautions you are taking. From the above post, I gather it's some sort of data centre. Up until then, I was assuming some sort of mass-produced physical product.
Which is kind of the point. I had mentioned on several occasions in our thread that A) I had never claimed 100% ability to prevent all attacks and b) the purpose of the exercise was to ensure that you, as the implementer of technologies could ensure that your own network was secure. And that you would do so - in part - by owning the supply chain for your widgetry.
Just a second there matey. You said you had that bit pretty well covered at the start; so I was trying for ways of accessing the kit without tripping any alarms.
No, I didn't. Please go back and actually read the thread. You were arguing your own points and not what it was that I was actually discussing.
Based on what you said in the above post and going with tradition; possibly a combined attack might work...simultaneously getting at one or two of your star-chamber auditors and also trying to spike your monitoring software.
The liklihood of your being able to get both (or more) auditing teams as well as the designers and/or the manufacturing is pretty small. And if you did - and the super secret squirrel one that i threw in for funsies - I can still stand in from of a judge and say "I did everything humanly possible, ma'am." You're up into "living to see the asteroid that wipes out humanity" territory of unlikely there.
Either way, I still accept it as a possibility in my design and there are still countermeasures and mitigation and incident response. Because it's actually possible to implement those things without spending too much (on an enterprise budget).
But again - it depends upon the product and the customers. Dribbling selected data over a time period might be the best payoff; or maybe showing up with a couple of guns and emptying your warehouse into the back of a lorry.
And you're back to "you can somehow pwn my network if you can pen the stuff en route". Wrong. You'd have to get the stuff en route, both public audit teams, the secret squirrel audit team, have the previous version testing software also have been pwned, and find a way to bypass automitigation and auto-incident response to get the data out. Even if you did, you'd get a small amount of data. That's a whole fuck of a lot of effort for not very much.
Whatever. Anyway, I could return the serve with some personal abuse and aspersions on your professional competence or I could wander off and do something different. On this occasion, I shall go for the latter, I think.
Oh, please, don't leave everyone hanging. Do demonstrate your ability to pwn the proposed design. But, before you do, please actually read what the idea is, so you're arguing what I'm discussing and not your own, completely unrelated idea of what I'm discussing.
Cheers.
This post has been deleted by its author
You did say that the network would have monitoring, mitigation and all the rest. First order of business for a potential evil attacker is gaining some sort of access. Thinking of ways to do that and sidestep as much security as possible sounded like a fun thought experiment
I agree that it is a fun thought experiment. I certainly enjoy kicking holes in such designs. I'm merely saying you're attacking the wrong end. If you own the folks who make the devices then guarding against tampering with the devices themselves is trivial.
This means that if you want to attack the network you need to get closer. Your attacks - and your reconnaissance - need to be at the point of deployment, not at the point of procurement.
I'm positive there are vulnerabilities here. There is, after all, only so much isolation and mitigation you can do. Systems have to interact in some fashion. How to you do that so they can, but are still as isolated as possible? How do you do this in a manner than can change in an automated fashion so no one person can know the whole design?
These are the weak points.
My poking at you was to get you to see this. To do a broader security analysis of the design and move your focus away from the easily defended procurement and towards the areas of the network where there actually are real questions.
That would have been a much more interesting debate because there are some very real limits to what's possible with today's technology. Even "new" application designs present problems. Any legacy apps would be huge security holes.
I have some thoughts - application proxies, mainly - but it's the area where my knowledge hits its limits and I would have to bring in a series of specialists to help me work out the fine details.
This post has been deleted by its author
Pretty sure that your biggest problem is going to be people
So you design everything in such a way that noone is 100% trusted. Which is what I've been saying all along...and was really hoping for a great discussion on how one might go about that.
mentioned earlier that IT knowledge and loyalty-inspiring charm weren't exactly synonymous and we spent subsequent posts proving exactly that. Now you can mitigate that with good working conditions; but that only goes so far.
Where did I claim to have loyalty-inspiring charm? That's a purchasable commodity. A network architect doesn't require it directly. Picard didn't have to deal with children on his ship: he had Riker for that. Etc.
And that goes for parts of the supplier chain you own.
Holy fuck, you're back on that again.
As you said in the article, a project of this type requires leadership and a company-owning leader who pisses people off may quickly end up in a worse position than someone more personable who has never been near the place.
Well, yes and no. You see grown ups don't sabotage their workplaces because they don't like their boss' boss' boss' boss. Good HR can root out most of those who would before they do and good network design can limit access of the hoi polloi, with only the most trusted (and well vetted, well compensated) individuals having deep knowledge of more than their segment of the network and/or access to multiple areas.
In addition, quite frankly, when someone is as obstinate or stubborn as you have been - not to mention unable to read - I'd simply let them go.
A better approach might be to send in a personable member of the team to negotiate a fixed-term contract that includes all the data/diagrams that you need.
Uh, no. That would leave your designs in the hands of someone else and raises all sorts of interesting rights issues. The goal is total control. You can't control what's in peoples' minds, but you can bind them by contract not to disclose and you can expunge their access to any design materials when they are not actively working on a design-related project.
and as a bonus, anyone who is trying to attack you by spiking hardware and the like will have to do it all over again at regular intervals; thus making it more expensive
Wrong. it makes it easier for them to do so, because you've created multiple soft targets that have information about your designs. You're far better to control the supply chain and still design your testing and validation regiment to expect potential compromise. Reduce the risk of compromise at the outset, but test for it anyways.
I put it to you (and this is not a snark, although it absolutely would have been if I'd said it 8 hours ago) that "If you own the folks who make the devices" is the worst possible way of approaching things...you're already inciting revolution (or at least mumbles of "fuck that guy rhubarb, rhubarb, rhubarb") before you've even plugged in your first box.
You're really, really bad with people, aren't you? Funny how it seems to be fairly easy to get qualified, talented, relatively loyal people to work for you if you pay them well and get them to work on projects they enjoy. I don't have problems with "rebellion", and hundreds of millions of other businesses don't have problems with rebellion. There are entire disciplines devoted how to treat people right to ensure you don't have rebellion. You could also - holy shit - listen to your staff regularly and find out if they feel they need anything.
And there's trust (although this may just be a different name for the people problem). You have testing and monitoring kit; but who wrote it? You have testing and monitoring kit; but who wrote it? The software stack it runs on? The hardware? Do you need monitoring software to monitor the original monitoring software? Who writes it? And so on.
As discussed eleventy billion times already, multiple independant teams who are given the design materials and tasked with coming up with independent testing regimens. They are not related to the original design team at all.
This is ground I've been over dozens of times. And you're still obsessed with the kit going into the business as the point of attack. Holy wow, man. Holy fucking wow.
Questions along these lines end up in a recursive loop and your brains running out of your nose. </I.
No, they're really quite straight forward. As a matter of fact there are quite a few very simple bits of game theory that apply here. They even give you the optimal number of independent teams, etc. Verifying supply chain is not hard if you own it. It's really, really not.
<i>For an enterprise of this type, it'd probably be better if you took the Merlin role and appointed someone else to do the King Arthuring.
No, Merlin had to read.
You'd also need someone truly stellar in HR...one of those rare ones who are very good at reading people.
The ability to read English and retain it would be where I'd start. We would then proceed to see how much charisma was required from there...but honestly that skillset if not that difficult. There are entire business schools full of them.
Getting the right core team together would be the make-or-break of the whole enterprise.
No, no, no and 10,000 times no. This is absolutely wrong. The whole point of security by design is to design out any single point of failure, including the failure of individuals. You don't need a stellar core team to run a secure, successful business. You need one to run a business that will rock wall street and perpetually exceed expectations.
There are literally thousands of examples of large enterprises around the world that are well run, stable, steady businesses that do things in a secure fashion. They don't make the news because they aren't prima donnas, they aren't high-stakes wall street derivatives stocks but many of them are household names.
If you design your business to rely on the charisma and personality of individual members of your corporate team you have already failed at information security. Everyone in a company is disposable. Even the CEO. That's proper security. Nobody can be indispensable. Nobody can be in a position to "leverage" the company. No one person - not even the CEO - can be allowed to have full security access to anything.
Policies, procedures and best practices determine how operations are carried out. Changes to those policies procedures and best practices are researched, audited, vetted and tested before being implemented.
It means the company evolves slowly. It means they will never be on the bleeding edge. But it can mean - assuming the design is correct - that they will be secure.
Anyone who is "exceptional" is a threat to the stability of such a company. Exceptional individuals have no place in the smooth running of an organization. They may be useful in research and development, but not on the implementation side.
None of this is a dig, by the way. I'm almost certainly worse at this hoo-man stuff than you are. People can also be considered as exploitable flaws, however, and a bit of introspection does no harm.
People are exploitable flaws. But the biggest risks are in ongoing operations (and the people making those operations go). New equipment can be vetted and tested and verified before being put into service. Any behaviour that deviates from modeled behaviour can be/should be analyzed. Equipment can be deployed in test/simulation environments before going into real ones.
Individuals responsible for design of equipment should be isolated from those designing testing. Those implementing testing should be separate from those implementing production and from those who designed the tests. Those who deliver the goods should be separate from everyone. There should be a "chaos monkey" group internally whose job it is to try to break things. Talk to Netflix about it and you'll understand the benefits.
But the people who are doing day-to-day production. Who are working the help desk, who have access to backups, administrative privs, commit privs, push privs, deploy privs...all these people are threats. They need to be categorized. They need to be maintained. They need to be well cared for, kept happy and - above all - their activities need to be closely monitored and documented so that if they attempt to screw up you not only know about it, but you can replace them at a moment's notice.
That said, that doesn't mean you have to be the evil overlords. You make it clear to people up front that you are a secure environment. They will be monitored. The company doesn't care if they watch porn while waiting for something to break. The company doesn't care if they listen to music or drink coffee at their desks.
The company does have issues with communications with the outside world during office hours unless they agree to allow that communication to be monitored for corporate secrets getting out. If they want to type sexy somethings to their significant other, that's fine: but it's going through the corporate network, not their cell phone, and the content will be analyzed by computers.
Make sure the corporate policy doesn't prevent them from typing sexy sweet nothings, and that corporate policy prevents anyone other than security teams from accessing those messages. respect privacy as much as possible and provide as relaxed an environment as possible, but make it clear that there are concessions to security.
If they don't act against the company's interests then they are guaranteed a job as long as they perform adequately. If the systems detect them acting against company interests a specially qualified, vetted individual trained in discretion and personal privacy ethics will examine thier suspect events/traffic and determine if they pose a risk to the company. The individual will be informed of the event and information about whether or not the data was false or positive will go back to the algorithm team to make the machine better.
That's the best design I have for keeping operations teams satisfied, but I am still not sure if it manages the balance quite well enough. And it is here where, if there is a failure in my design or a breach in the company that it will occur.
This is why I would personally bring experts in to pick apart various stages of my design.
That said the design is based on a lot of research. Failures and successes of other companies. Every single security expert I've talked to - and most that I've read - are adamant that the biggest risk to any company is ongoing operations. Not procurement.
What's more, the procurement design discussed here ad nauseam is one that aligns not only with the best expert advice, but with game theory as well. I simply do not understand why you seem so obsessed with the idea of compromising devices as opposed to compromising the people who will be safeguarding and using those devices every day.
I've only been on the periphery of security for a few small firms and have been involved in DR and some penetration testing. Some sites have gone from home-grown to NTLM-style security, to SAML with multiple outside connections.
I'm in total admiration of the folks that can juggle so many technologies as well as stay on top of the current ones and the current batch of threats.
You can throw several boatloads of money at an IT shop, even one with great sysadmins, without totally solving the vulnerability problems. It takes training, experience, probably consulting, conferences, and lots of hard work.
No wonder I can't find anyone to fill a nice little coding job with lots of opportunities. Young people are obviously smarter than their CVs show, and they have decided to find a job where they don't sliced and diced for a handful of gravel a day and blamed when the house of straw collapses :-)
I wouldn't mind, and I agree completely with the sentiments expressed, but as a slightly unusual small business I have always strenuously fought the boss (read wife) to ensure we don't cut corners with our IT.
So I have a nice little job with good resources and no one interested in filling it. Apart from a few wannabees who 'can do Word' and 'want to be a games programmer'- the painful result of completely misguided IT education policies :-)
When I go home at night I know I've done a solid job, given the resources at my command. I've made suggestions, worked hard, applied the very best google-fu I can summon.
I used to care to the same degree you do Mr Potts. It can make you a more valuable, harder working employee - until you burn out or quit. Letting go of it all is hard, but my health is worth more than my wage. I'm glad you escaped that hell, but there are other ways out too.
"[..] but there are other ways out too."
I retired - after letting some ambitious youngsters think they were slowly usurping my role. I knew a few years back that I wouldn't be there forever to keep pulling the chestnuts out of the fire. So I went at a time of my own choosing.
Different AC here.
I changed the way that I worked. I became a diligent but apparently unambitious grunt, prepared to do what I was asked to do without being in a position of responsibility, and let those above me take all the sh*t.
OK, I don't earn the megabucks, and occasionally I get asked to do things that are really not deliverable, but I work short(ish) contracts (at least that's what they're supposed to be), so can move on when I need to, and I can see myself working like this, probably moving down the hierarchy as I get older, my requirements for income reduce, and my skills age, until I can properly retire.
Add a yearly requirement of 35% of the initial project cost for upgrades and repairs. Do not under any circumstances go below 20% when dickering with the accounting trolls. At that point, have them zero out the upkeep budget for that project, call it a legacy system and refuse to support it.
What does Katherine Archuleta have to do with Front Line IT ?
Nothing that's what.
Gov IT isn't the whole culture either so nix that as well.
Some of us are very well paid Techs and love the Industry and steer away from Jobs that resemble sweat shops or panic on the USS Trump. If you happen to to think you're still IT while having worked with Mr Jobs then you're a little confused.
All Hail The WOZ !
Corporate to SME IT: "I want you to secure your systems against intellectual property theft, other group companies have been hacked (sic). Here are the rules which you must follow, No I don't understand it either, I got a consultant to write it (who copied it from ISO 27001) but you do, you are IT."
SME IT/Cyber Security Manager to MD: "Group want me to implement as security system, for us its impossible in terms of money and effort but we can make a start at once: you will have to stop sending our intellectual property to your personal server at home which is outside my control, and start adhering to your own policies"
MD to I T Manager: "No way, apart from the cost I don't believe we need such protection, its all bullshit. OH and I need Admin Priv so my son can install games on my work laptop, OH and can you change policies to that I can get to the football scores on the Internet and watch porn on my computer "
I T Manager to MD. I might as well give up now then. Bye Bye!
This is the situation in many companies, where IT is not a core function, and even many where it is.
No idea, I've never seen a Star Trek film.
Used to watch the TV series a long time ago, but can't remember any episodes where box ticking was all they needed. Did they have a disaster/business continuity plan - No - don't get me started.
For a long time I was convinced that the guy who wrote Dilbert worked for my company.
Worked 'near' the security people at a medium-big oil company. Watched their actions and words for years. It was all very 'correct'.
The company followed best available practices. (as revealed by their vendors) They used the best tools available. (those funded) The people individually checked all the boxes with their certifications (self-funded)
The security guys could not defend the company. What they could defend was their own efforts. Within their purview they did everything in their power. They certainly had my respect. Their efforts individually were not inadequate, but they well knew the total combined effort across the corporation was inadequate.
And they knew - knew - that they had been penetrated at will, more or less quarterly. The company as a whole figured this out - a few quiet inquiries from the C-suite made it clear even they knew - as the people on the other side had too much obviously internal information, not just re: negotiations but their technology/techniques/timeframes were known to low-level counterparts.
The company took the only action possible to them to 'cure' that situation. They slowly sold off any projects in the area and backed away from any further interests in the resources of or near to a certain large Asian state. They could do nothing at the turnabout that it was their own resources - technology - that had been extracted.
There is the characterisation that to get by when "living in the big city" you keep your head down, walk fast, look as threatening as possible when challenged, and be ready to run like hell. You *don't* attract the attention of gangs, unless you are surrounded by your own gang.
That network you've got protecting you... surrounding you... do you really think it'll stand up against the big gangs? We are everywhere living in the worst parts of the worst big city. Be ready to run like hell. Individually if need be.
That network you've got protecting you... surrounding you... do you really think it'll stand up against the big gangs? We are everywhere living in the worst parts of the worst big city. Be ready to run like hell. Individually if need be.
Absolutely not. But that's why proper security is about far more than prevention. You need the following covered, at a minimum:
1) Prevention
2) Detection
3) Monitoring
4) Mitigation*
5) Incident response**
6) Penetration testing
7) Randomization
*Compartmentalisation/isolation/segmentation of data so that no one breach can pwn your whole network or all relevant data.
** You will be pwned. Accept this. Have plans of action to deal with it.
You will eventually have a security breach. Make that reality part of your security plans. Too many IT "professionals" think that security stops at "prevention". There's a hell of a lot more to security than patching and firewalls!
Also, without 2), everything else doesn't work. You need to detect the sooner you can if something bad is happening. It of course rerquire a proper 3) system, and good human eyes and brains to understand what the monitoring systems are finding and reporting, and to tune them continuosly.
The deadliest attack is the one that goes unnoticed for long enough to become a metastasis you can no longer fight.
There's not yet a fully automated "fire and foget" solution that enables a sysadmin to be lazy, sleep (or watch porn...) while waiting for the ring bell and the big red light to turn on, while the "Ultimate FW/IDS/IPS/AV/<put your acronym here>" does all the hard work and churns out the report of its activities.
My questions would be: Why was this information stored on computers accessible through the internet? Why weren't the systems isolated once it was known they were compromised, which was apparently moths before the data was taken. Why was this data unencrypted? You can talk about budgets, but 50 shades of stupid was needed to get to this point - and money doesn't fix that.
Why was this info stored on computers accessible through the internet? So many possible reasons, but let me pick one: an intranet or 'knowledge library' shared by offices scattered across the country or across many countries. Or a VPN. Or some other way the C-suite wants to be able to share info among themselves. Unless you are shaving a messenger's head, tattooing in the info and letting him go by armoured car to the next office, how can the system not be pwned in some way? This is at kiddie level.
A couple of things here.
Firstly, getting the budget and support you need is a sales problem. IT is very bad at sales. This means that IT gets dictated to, instead of being able to find out what people actually need (per the article) and deliver on that after getting the appropriate level of funding.
Secondly, the business knows IT pads their budgets. IT have trained lines of business to cut their funding because for the last n budgets, IT asked for $Xm and only got 70% of $Xm. Instead of saying "Ok, choose which projects you don't want" IT says they'll do their best and makes do. LOB figure they get what they want when they argue the budget down, (and are rewarded for it), so guess what happens? The consequences are far removed from the action, so look who gets left holding the bag.
Get out there and sell IT as something worth paying for. Doing things the current way isn't working, so maybe instead of insisting everyone else change, perhaps IT could try changing things under their own control?
Go make friends with the head of Sales and get them to teach you how to sell yourself.
"In order to be good at IT you must first master business, sales, marketing, procurment, etc. etc. BTW, the pay is shit, you get no respect and people on the internet will tell you you're a failure if you aren't capable of doing the job of 15 departments full of people for less than the average mortgage payment with a smile on your face."
Thanks for remaining me why I quit, Justin.
Ah, no. It doesn't have to be you personally. That's why we work in teams/groups/autonomous collectives.
This is all a succession of managers' fault for not supporting their staff with the resources they require. An IT department with internal customers has a bunch of functions not purely technical: HR and finance, for example. Either their own, dedicated people, or they use some shared service. It's not the DBA or sysadmin doing HR, right?
But how many IT departments have you seen doing any marketing or sales? Why not?
@jpwarren
The problem with that is that, in the places that have a significant problem with IT of the "underfunded and under-resourced" variety, that kind of User Requirement Identification will end up being a timesink that's just as hard to justify as the rest of IT's operations. Try and do it in a hurry and you'll get useless feedback (because frequently the users aren't in a position to give you technical details about what they need - which is fair enough, part of our job is to identify those requirements and provide ways to meet them), but when you then spend significant time on it, you fall into the same trap of being told to stop wasting your time on anything that's not part of core operations.
At best you get someone at a Service Delivery Manager type level who handles this part of things for you, but the problems are usually more pronounced in organisations who don't see a need for SDM type roles...
when you then spend significant time on it, you fall into the same trap of being told to stop wasting your time on anything that's not part of core operations
Indeed. New alpha male manager, within a couple of days of having been brought in sidewayslet out of the cage, wanted to have our document wiki killed because "nobody uses it (he asked a couple of people in his office whether they had heard of it) and in my deep experience lack of use means that user requirements are not being met". It happens to be the one place where things regarding IT are actually being documented, a step up from the "headless but feeling extremely efficient" chicken syndrome.
We, the IT grunts, own the world. Literally, There is not one damn human activity on the entire planet that IT doesn't touch. And certainly all the lovely things that make modern civilization.
So why are we letting the clueless fucks dictate to us? We OWN them dammit!
I often say it's time to organize an IT strike, and turn off all systems for a day or two. Maybe only then someone will realize today IT is the nervous system of the whole society, and not just a nuisance you have to accept because somehow Internet connectivity is nice to have.
Because IT bods are useless at seeing the larger perspectives and trends. See, back in the day I worked with the "lawful interception interface" on some telephony servers.
Did I insert a back door? No. Why not? Because I didn't think about it, I was thinking 100% about getting the code perfectly right, not slightly wrong.
Now that I am on the "money-side" of things rather than the technical, older, more cynical, I see money, opportunities for spending it, and, priorities change, I can appreciate how the value of a slight mistake could be worth so much more than anyone paid for the flawless product - certainly would be just revenge for my last pay review - and I do wonder about "the nerds", how naively loyal they are even in spite of how they are treated. It's like kicking puppies.
Then we should build something better out of the ashes. There's too many organizations, governments and corporations addicted to cheap tech now, and the pace of tech change has steamrolled over us all. There's no time anymore to reflect on the problems or properly budget for security, never mind self-examination of what works and what doesn't. Who knows what the workplace will be like in a decade. I'm with Trev on this, exit stage left ... I'll mop floors!
Firstly let me agree with everything you said. As an IT manager (who was previously an engineer) I am usually brought in to rebuild IT teams destroyed by incompetant IT managers or worse, destroyed by malicious pricks at the top slashing IT to the point of collapse.
But. This comes down to money. In an ideal world, you'd be allowed to build the perfectly robust and resilient infrastructure with unlimited funds. The problem is that IT has to be paid for by profits earnt by 'the business' - sales of stuff. This puts them (in their minds at least) in charge of how much of their profits the are prepared to 'waste' on IT.
Our job in IT is to explain the value in what we do - the fact we are the guardians of IP, the defenders against malice, the builders of product, and maintainers of service. We need to explain that you CAN cut corners in these areas but there WILL be consequences. We should stop papering over these risks they force IT to endure and call them out at the highest level. We should allow the risk to happen from time to time to show the reality of their decision, rather than catching it every time. We should cover our ass and document our concerns and highlight the risk and impact of being ignored.
As much as none of us entered IT for the politics or the finance or risk management, the reality is these are tools you need to have in your toolkit as well as the technical knowledge you possess.
Scott was (will be?) lucky because he has no need of money, and has no competitors. His problem is just time.
In this century, we still need to cope with money - and unlike IT needs, most companies have in place policies and controls for money spending and budgeting (people time means also costs).
Increase the costs/time too much, and your project could be easily axed or lose a bid. If it is accepted, and you can do it spending less, the first time or two you will be congratulated. Do it more times, and your forecasting and budgeting skills will be questioned - while nodoby usually has the skills to assess if you released/implemented a crappy product. Delivering means money flows in (or stops being spent), thereby that's what is importand, not what you really deliver.
Until this mindset changes, there's little you can really do but looking for niches where it had already, ot look for something else.
is "IT Business Alignment". Which, by itself, encapsulates the issues that can occur when management doesn't understand IT and its important to the organisation.
Yes, IT should attempt to align its procedures and operations around the organisation's goals. What management often seem to miss in this regard is that the reverse is also true, at least for any organisation where any kind of data storage and processing is part of their workflow - which means most of them, to some extent or another. Data security and integrity are areas where, based on understanding relevant legal principles, it is the responsibility of the business to align itself with IT (or, more specifically, how IT implements controls for things like Data Protection Act compliance).
Picking the higher education sector as it's one where I have a fair amount of experience - getting management to understand the extent to which trying to stop researchers from storing research data in their Dropbox account can be really challenging, because a lot of the time they view IT as "those chaps who are sort of computer janitors". Despite the fact that, depending on the nature of the research data, such actoins can actually end up with the department/university on the nasty end of a sizeable lawsuit - especially when dealing with people who don't understand why they should need to encrypt the confidential medical data with which they work because "I set a password on my Dropbox account, so it's protected, right?"
You can get some goodwill if you demonstrate flexibility where possible in aligning other policies and services around people's requirements; but ultimately management buy-in and support are vital, and unfortunately if your organisation doesn't have a (good) CTO and/or an ex-IT pro in a senior management level, you end up dealing with a lot of silliness. And nobody wins in the environment where budget requests have to be needlessly inflated only to be slashed back down to what was needed in the first place - that's wasted time on everyone's part that could've been better spent on doing something to further the organisation's actual goals...
You easily solve this problem by resigning if you aren't allowed enough budget to do what you think is best, instead of getting a well-paid job and hoping it'll all be okay.
You can't do that, however, if you aren't ethical or expert enough in your field to understand the situation.
I'm with you Trevor. After 30+ years in the computer business, working at everything from developer to network management I'm fed up with the whole business.
This goes beyond security. It seems that nothing works right anymore--hardware, software, you name it. And please don't get me started on the quality of documentation these days.
For the last 5 years or so it seems I spend all my time on the telephone talking to idiots, trying to fix problems caused by their crappy products. No one cares about quality anymore. It seems everyone is so busy trying to become a billionaire by coming up with the next "Yo!" app that nobody is left to mind the engine room.
I started out life as an electrician. I'm ready to go back.
An app with a single purpose of sending 3 characters across the intertubes was fundamentally flawed. Granted there were dependencies such as contact lists... but bringing the kid into our IT fold the fact that we can't even get 3 characters from one device to another securely and safely shows the flaws in our industry.
If we built houses, roads or made food we would be worse than the charlatans currently doing those functions who are using our products and services.
This was one of the most accurate articles I've read on a tech website, or any website for a long time, real hitting nail on the head stuff. It was hardly rocket science either, just the big secret we all know and pretend to forget for our 7+ hours a day.
Dear PHB,
While I appreciate the need for the company to return a profit to shareholders in the coming year I think your plan to cut the XXXX budget is counter productive.
The cuts impact the reliability and security of the system and place the companies profits at risk.
A days outage of the sales application will cost an estimated n zillion dollars. A serious security breach
could irreparably damage the reputation of the company and lose long term customers forever.
In order to comply with past budget cuts we have cut costs and improved our operations. However any further budget cuts will seriously effect our ability to provide a reliable and secure service.
The decision is your to make.
Regards
S Neaky
Ops Manager
I'm now at PHB level, the fact that you have put "companies" rather than "company's" undermines your attention to detail and raises the issue of whether I can trust your judgement on this matter. On this occasion I note your concern however I do also need to cut the budget and I'm now more convinced of this than before I read your email.
Love 'n hugs
PHB
P.S. I'm also looking at your use of "days", confirming my initial fears which are then compounded by the odd break in that line.
@John:
The problem is, it's not "point-missing nit-pickery", it's ammunition for someone with an agenda to push that allows them to dismiss those who (quite possibly for entirely valid reasons) oppose that agenda.
Yes, it's daft, but OTOH that's humans. I've worked in environments where the opinion of the only person qualified to make a decision on the technical impact of a configuration change to IT systems was ignored by management because management were all senior academics and the person from IT didn't have a PhD - and while they would never outright say it, they made it clear that they simply did not take seriously the opinions of anyone without at least a Masters to their name...
Today the fed govt is spending roughly $13 billion on IT security and yet we are still seeing breaches by China and others. Today we have all these various IT departments and they are free to do what they want. So now we do not have a uniform security implementation among various branches. One issue is how to manage this without creating another massive bureaucracy. One way would be to have regular round table meeting among the various IT dept's. Another possible option is to have internal security auditing for the whole federal govt.
Good what a bunch of primadonnas IT staff are. Chances are that if your company is treating IT staff like carp they are treating everybody else in the company like carp. In my experience companies carp on everybody equally, not just IT staff.
The general problem stems from the fact that many people in management start drinking their own cool-aid. Nobody wants to work for a manager who goes around saying that they are incompetent (I did meet a CIO of another company who did this), however managers should know the limitations of their staff, technology and staff. The problem is most people don't, it's not new read Feyman's "What Do You Care What Other People Think?" to learn about how it can happen no matter how well your funded. t's also not just in IT it's everywhere, yeah you can leave IT, you'll still have to deal with it.
The main problem with people is everybody wants to feel special, everybody wants to have the Answer, Trevor wants to be the guy everybody turns to with THE answer. Once upon a time maybe this was possible with IT, computers where relatively simple and systems where understandable by one person, now this isn't the case. There is no such thing as perfect. You should have learned that years ago. Everybody is under-resourced and there are never enough hours in the day.
The real question is, does it really matter if Billy Nobodies tax affairs get published on the internet, a power station get hacked, a hospital systems gets compromised or the latest movie gets leaked. Some of the previous example are clearly very important and need protecting others really don't matter that much and are first world problems. Sometimes the solution is simple don't plug critical systems into the internet (not always but mostly possible). Spending money on securing them is money that can't go to paying doctors, then a hack can be embarrassing not life threatening. The problem again is that people drink the cool-aid, they hear somebody like Trevor say I can make a secure system so the start connecting critical systems into the internet. They take a few precautions then as is human nature things get forgotten or it turns out really expensive and something goes horribly wrong.
The problem again is that people drink the cool-aid, they hear somebody like Trevor say I can make a secure system so the start connecting critical systems into the internet.
Firstly I believe it is Kool-aid, and is a reference to Jonestown. I may be wrong.
Secondly, Potty actually says he likes to think he can make a system secure, but goes on to state later that he can't. Its quite obvious he can't. Nobody can. And he'd certainly not be bailing out of the industry if he actually believed he was a top gun.
Lets say you had the ultimate sysadmin, a hypothetical being who could secure absolutely the IT estate of a company.... one crap developer will accidentally have him rooted six ways from Sunday before he can saw "Aww shit". Security is a team sport... and no matter how good the team, nobody wins all the time. Worst case, it's the XKCD pipe wrench applied to someone with legitimate access.
"Secondly, Potty actually says he likes to think he can make a system secure, but goes on to state later that he can't. "
I never said I could make a system 100% secure. I said I could build the best network out there, given enough resources. Nowhere does that mean that the network will be 100% secure. It means that it will be as secure as is possible and tha tit will be able to cope with the inevitable security breach.
Learn to read.
The resources part of your article is one of the reasons why I left a purely IT environment. The other was the complete lack of any funding for training whilst everyone else swans off on this course and that course. I now work in an Industrial environment, designing, building and programming machinery and systems. It is still stressful at times, but I find it much more rewarding, but not quite as lucrative.
Despite more than a decade of attempting to change that, when pressed, I crumble and cough up the real number. Business types, thinking they're being clever, will then tell me to go forth and make do with half the amount I need.
It always entertains me when people start talking about a decade in terms of it being a lot. Its about 1/4 or 1/5th of a career. Time enough then to be useful and to learn a thing, but not so long as you have nothing left to learn and no improvements to make.
Perhaps, as for most of us (me included) your technical skills exceed your powers of persuasion? A colleague recently asked me how many ways I could influence the manager to whom I report - in the sense of bringing them to my way of thinking. I got to about 4 distinct types. He had more than 15. My technical skills far exceed his, but if the manager making the decision is influenced into doing things his way rather than mine, then my tech skills count for little.
I've spent 20 years never having enough resources and I just can't take it anymore. I need to sleep without the nightmares. I need to be able to go to work every day without having legitimate, clinical panic attacks during my morning commute
In all my years of doing this, I don't think I ever met one single person said to me "I have enough resources to do the job right". I don't think anyone in any profession has ever told me that. Between not caring, and caring too much, lies a vast plain of potential.
For anyone but the very top recruits in IT to ask probing questions and investigate the company they are being hired by is viewed as arrogance.
I've successfully done this in the past. My trick for it is you can only ask them 3 questions, one has to be about something from the interview. Doesn't sound like enough, and it isn't, but its just laying the groundwork for further enquiry.... and that is what the elevator ride back down to the security desk is for. It won't get you a full understanding of their position, but with enough practice you can spot a lot of red flags that way.
Padding budget estimates and timelines is not the best way to handle these issues, in my view, though you do need to do it a little. Take a modular approach - identify critical success factors and operational risks and produce a plan based on these. As the budget is reduced, have them remove some of the CSFs, which also removes the cover from the associated operational risk. Managers don't like documentation when that documentation covers your ass not theirs. Changing the budget allows them flexibility, but automatically adjusts the scope obtainable and the risks in doing so.
It always entertains me when people start talking about a decade in terms of it being a lot. Its about 1/4 or 1/5th of a career.
If you haven't learnt a skill in a decade of trying, I'd suggest that it's not one you're going to manage. Trevor is clearly talking about something which he has repeatedly tried to do better in, yet failed. Your point would be valid(ish) if it was a skill which he had simply not had time to work on, and was now about to start. And finally, I think that quarter of a career is a significant time in a human's lifespan, especially given the frequency of changing paths means you might have moved to an entirely different field now.
The rest of your post is bang on.
If you haven't learnt a skill in a decade of trying, I'd suggest that it's not one you're going to manage.
If you haven't learnt the difference between learn and master, then I suggest you keep practicing.
Trevor is clearly talking about something which he has repeatedly tried to do better in, yet failed.
His past failures are no guarantee of future results, unless he has stopped learning from them.
And finally, I think that quarter of a career is a significant time in a human's lifespan, especially given the frequency of changing paths means you might have moved to an entirely different field now.
Nobody would disagree that it is significant. 10 years is time to learn anything, but master nothing. The ancient tea masters or the orient would spend a whole life learning nothing but how to brew the perfect cuppa. It's a skill you can learn in 10 minutes, but most certainly not master in 10 years. Technology, well, that's a bit more complicated than tea....
A look at the comments and endless past articles shows this to be an important and increasingly timely topic. But history suggests it is only going to get more important, more timely and IMO we as a society are not going to be addressing the fundamental issues anytime soon.
To support that opinion look at the quantity of blood that flowed before engineering standards and codes were developed for other parts of our infrastructure. Empires and societies literally fell and collapsed, blood literally flowed, the wealthy lost fortunes and their own lives before those finding themselves in charge determined that physical infrastructure should be build properly, built to the conditions the infrastructure would experience.
Even then it took generations.
In the meantime we have to follow orders, even if those orders include putting the equivalent of a skyscraper on a foundation not suitable for a mini-mall. If you don't feel comfortable doing that, if you can't rationalize away your role when things collapse and blood flows, then maybe this isn't the time or industry for you. Or me.
"In the meantime we have to follow orders, even if those orders include putting the equivalent of a skyscraper on a foundation not suitable for a mini-mall. "
If you were an engineer doing that you'd be prosecuted out of existence.
In IT it's much harder to refuse but you _can_ send an email saying that in your opinion this is unwise for XYZ reasons and be sure to retain a copy.
As other posters have said, managers hate it when the documentation is covering your ass and not theirs. For some people I deal with, I _always_ carry a recording device - because they will always deny having said something when it means they'd take a hit (and deny the denial when played the recording).
This post has been deleted by its author
and they do not appear on balance sheets
This just means that the "tool" of balance sheets is way out of whack for the applications that it is being put to. And that the insurance companies are in dereliction of duty (otherwise the premiums for "insurance against getting reamed" would hurt where it counts). And that the "industry" is totally immature ("oh we got hacked... sorryyy... here is a free credit alert for 1y" ... *wristslap*).
But immaturity in the 21st is seen as "refreshing". So there.
"What you do not spend on IT infrastructure will be taken out of your wallet by careless/malicious users, hardware and software failures, power drops, acts of God, etc., but you will lose more to downtime and outages than it would ever have cost you to spend on infrastructure, spares, staff, training, etc."
There, fixed that for you.
In short: run your environment, or it will surely run you.
Good points. But about trusting Google software over GM etc, Google's culture of beta is good enough would give me pause. Plus the Google self-drivers require $250K conversion kit, the route has to be mapped anew every day, and a qualified driver to take over at a moments notice of failure, seems more stressful than driving.
Getting an IT system working right is all about technical skills and attention to detail.
Getting a budget approved is all about people skills, negotiation and (frequently) knowing where the bodies are buried.
It's very rare to find all of these skills in the same person. The interface between the good technical folks and the horrible, gooey, backstabbing world of corporate politics is critical. You need someone who can understand and support the IT folks and also play the corporate game - and people like that who don't want to take advantage of the IT folks are like rocking-horse shit.
It's very rare to find all of these skills in the same person.
This is the CIO's role but in the end it is impossible to be on both sides of the divide. It's like being a capable, old-school general in the 3rd Reich's bonanza of power-grabbing. You will be served sooner or later.
Sane corporate management should be all about avoiding backstabbing, literal or otherwise. It is the CEO's role to suppress such behaviour and he is in dereliction of duty if the turns a blind eye or encourages this kind of bullshit. The sociopathic form of "management" is very prevalent but that doesn't make it okay. Indeed, it indicates that the company should be left to its fortunes ASAP.
If this were not the flowery epoch of historically unprecedented experiments in bailouts, free money and casino capitalism, reality would assert itself. We will have to await the supercrash to apply the bleaching. Won't be long now.
You are completely correct Trevor. My recommendation is to try to save as much of your own money so that you can retire as soon as possible. The companies are only going to cut IT spending even more. The stress can or will take a toll on you physically and / or mentally. Eating junk food or sugar to keep going and alcohol to wind down in moderation is fine but the stress can push the body too far. Watch out for depression and see a medical doctor if needed. The people here and many others do care about you.
I almost got fired because I kept on complaining that the expensive off site recovery process didn't work. It took three long years before things were corrected. Every lightening storm still makes me jumpy.
having worked in for the Fed. briefly I can say that most of the IT staff is probably not as skilled as the author thinks. The way the hiring is done for the USA government is as follows:
1) internal employees
2) vet's
3) special minor groups
4) any one else out side the above 3.
The job requirements are secondary to the selection of the personnel that is hired. So if some one whom has 10 years direct experience applies for the same position as some one with 1 ( but is in group 1 thru 3), the person with 10 years experience will be pasted over. The issue with this that just like all other US companies the Fed does not do any training. So even if you have a person that has a lot of drive but little experience apply the odds of their skills increasing outside their own efforts is very low.
So many good points and insights on this thread - it pushed me to my first post here (I've been a long time reader).
Just like with the environmental or the financial issues, the human society prefers to kick the traditional can down the road instead of taking meaningful action. Sadly IT is mostly a race to the bottom nowadays, and nobody wins in the long run - the products and services are getting more complex but less secure/reliable, and both the employers and the employees get more and more frustrated. I truly liked the idea of an IT strike to bring attention to an increasingly unsustainable approach but I just can't see that happening. Turns out train, bus and taxi drivers have (much) more solidarity! Could very well be an evolutionary quirk.
I've read all comments but I did not see any mention of the staggering numbers of Chinese and Indian people willing to come and take over your/our "sh1tty" IT jobs, while complaining a lot less. Heck, I'm sure most of them would even be grateful... at least for a few years. So yes, I could definitely leave company X or Y because of my work principles; meanwhile, while I'm looking for the next, "ideal" place to work (i.e. earning no salary while I do this) most companies are willing to get the cheaper candidates, maybe with less experience and principles. So where does that leave me? Yeah, I thought so. Sadly this is the game, you/we either play along or don't. No employer is there for *your* benefit first. The way I see it there are only a few options:
1. learn the unwritten rules and play the game along - if doing so doesn't clash (too much) with your inner principles
2. go mental
3. leave and start your own business (if you have the idea, skills, experience, money and/or guts), where you're the one calling the shots.
Just my 2 pence, happy to take any flak.
"I also suspect – and there's some proof for this – the OPM's IT efforts were spectacularly underfunded for the task to hand."
They won't be anymore.
A parallel: For the last 6 years I've been periodically warning the regional office of the chain of corner stores which has one a couple of doors down from me that their CCTV system is inadequate and will be utterly useless in the event of an armed robbery.
Last weekend that scenario happened (the crims jumped the morning shift when they arrived to open up) - and as predicted the CCTV system proved utterly useless. Net result is a couple of highly traumatised staff (having a sawn-off shotgun shoved in your face isn't pleasant) and the crims likely got away with it.
Less than 24 hours later the entire system was substantially upgraded. I suspect the words "vicarious liability" were bandied around and I'll bet they haven't told the police or their insurers that they'd been previously warned about the state of the setup.
interesting thread,
one of the few on el reg in the recent past.
I read with interest what happened and how it all slid for the author - and what , who etc can be held accountable - see the irony in the work accountable.
More often than not those who make the decision have no idea whom their decisions will affect and those who will carry those cans will be those a shade above those who .. therefore ultimately leave through choice or .. are exitted / part of a new solution.
In the case of outsource of course the "can carriers" are lost (exitted) early on.. and the lower orders are cherry picked .. in the first round - if that is they're still around.
Mind you - there are still compliance people out there (and i've seen them) with teeth so sharp they'll shut you down and your company will bleed (perhaps not in public) till it hurts.
They are few and far between - but they are out there.
Alas in the uk of late - they've been .. well let's say dismissed as candy floss .. with next to no power, which for us a shame as we have no allies when it comes to - using the term - no ... this will impact you adversly if YOU DECIDE to do that.
Long long time ago - i worked for a company who encouraged you to say no - and advise why if you NEEDED to - and would back you up for saying so.
Needless to say both that culture and that company were snuffed out a while ago...perhaps it now exists in the cloud(tech).
still - thanks for all the comments - thise one's been cool so far.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
