Big IQ play from IT outsourcer: Can't create batch files if you can't save files. Of any kind The end of a damp weekend (for the UK at least) heralds a new instalment in our ongoing series of Register reader confessions. Welcome back to Who, Me? Today's story comes from a reader Regomised as "Alan" and concerns the time he was instrumental in the accidental near-shutdown of an entire department of Her Majesty's …
As you may recall, WIPRO had an embarassing "security incident" a year or two ago. I was one of the users on the ground who felt the repercussions - because the miscreants had used Powershell to gain a foothold, they simply blocked Powershell EVERYWHERE to "Improve security".
Apart from of course not being able to run or develop anything any more (hi, automation guy here), I also was blocked from even opening my .ps1 files in notepad, so I couldn't copy the work to an off-domain dev machine and continue the work. Months that took to sort out...
Office software's ability to access files anywhere on a system a decade or two back was well known. I'd used it myself - so there had been no excuse for leaving that open.
And no one with an ounce of sense turns off core functions (like save and print).
All of which does rather suggest that issues around outsourcing, and the kinds of people who run public IT must be pretty long standing.
And then we wonder why public projects never seem to work, let alone on time or within budget.
It only saves money IF the job is one that doesn't need to happen regularly. For example, if your business has 2 photocopiers, outsourcing photocopier repair is the only thing that makes sense.
Outsourcing a 5-man full-time IT helpdesk or the like is ludicris because you're just inserting a needless middleman between you and those 5 employees. So either they get paid badly and the turnover is terrible or you're paying way to much (in most cases it's both).
And then there's the idealogical aspect. In private companies it's referred to as concentrating on their core tasks. Which makes absolute sense until you think it through. i.e. the same level of management is needed to manage a contract as is to manage an internal team because you still need to make sure that the job is done satisfactorily. maybe more because an internal team only needs normal management, not the forensic skills of managing external contractors' performance. Whether companies who outsource services do actual do that level of contract managing is a different kettle of ball games.
In local authority services it's idealogical and financial. It's a way of getting staff on crap pay and conditions.
My experience of this in local authorities is probably replicated in commercial organisations, - I'd guess - based on what I've been told-. is that the frontline staffs' own managers have to devote too much time to this work, because no one higher up has the competence, knowledge or interest in doing it. The ones that did were all got rid of, or went to work as contract managers for the outsourcing companies. And yes, I've known people who used to manage local authority site staff, IT teams etc who've gone on to become account managers with big contracting companies. Taking their skills and knowledge with them. And come contract renewal or any (inevitable) out of contract work they're like gamekeepers turned poachers.
Yup.
Once worked at a (nameless) design agency, who were asked to build a 'market-offerings-comparison' type site for their main market by several different clients. (mainly because the incumbent offering was utterly pants).
3 years later and the system is the premier system of it's type in that market, we've got millions of rows of market-specific data and the IT and WebDev side of the agency is still the 'also-ran' department when budgets and resources were handed out.
D'oh!
I work for a national IT services company and it’s the same here.
I’m a break-fix technician, one of the guys who fixes that nice hardware the company has sold/leased to clients.
But we don’t make the company money, so we’re only seen as an expense.
Nope, it’s the sales team that makes the company money, we are just the ones who make sure the clients don’t leave so you keep making the money.
Oh yes, when I worked at an MSP, we had companies that were impressed by the staff, response time, proactive protection, etc. And we had "infrequent fliers" who only called when something broke very badly, often so rarely that it would be companies that I'd never heard of because the last time they spent a penny on anything IT was long before I started there. Now they have ransomware or a dead disk array, and it turns out their backups haven't worked in years and antivirus was long expired, because it was too expensive to get someone out to take a look at their little network even once a year or so, or get it set up to send basic system alerts. They end up finding out how much more expensive it can be for your business to shut down and rebuild it, especially the accounting side.
One of the problem with outsourcing is that it is taken as a solution to the wrong problem.Often outsourcing is seen as a way to save the cost of doing something. While it should only b a way of saving on the burden of doing that thing, but knowing well it will cost you more.
Yes. This is exactly right.
Not sure of that. I worked for a small council and we had 7 or 8 software people and 3 pfys and and MBA manager who knew fuck all about bits and bytes. There were 3 old hacks who had managed to resist all attempts to change their contracts to make them legally obliged to train an outsourcing company to do their job. We kept the boss at arms length and serviced the 'customers' very well. I was amazed to discover that other councils of similar size generally had ~100 outsourced people crawling over desks and filling in forms.
We used to get things done before they had even got the first planning meeting for a project and there was generally no budget because we could make it work on the stuff we had.
The reason why other public projects may not work is because people like Did0 get involved. People who are not interested and have no investment in the project in the first place. People who seem to be hell bent on making public projects fail and fail loudly.
I had that some years ago.
Roll out new machines across the estate, all machines must have external I/O access disabled, all USB ports disabled, all machines to have no expansion connectors.
Roll on to Vehicle Testing, where a clued-up manager realised we were about to destroy his entire business by removing the ability to connect any of his testing equipment, and kicked us out before we did any damage.
... I've had the fix the end result of a junior admin deciding that things would be a lot more secure if he changed the permissions of the contents of the likes of /bin, /sbin, /etc and/or /var. Usually when they get this brilliant idea, they manage to do it recursively ...
It's more secure, all right.
I accidently changed all the file properties to read only across an entire hard drive. I'd intended to do it on a floppy but typed the wrong drive letter in the path. I realized my oopsie when my Win3.11 suddenly locked up tighter than an accountant's personal purse strings. I fixed it by restoring a backup of the system I'd made earlier that morning.
Posting anon because I still feel embarrassed by the !DOH! event...
Security is quite easy. All you have to do is sort out ownership and responsibility. You simply cant fuck with anything that you dont own or are responsible for and should not be allowed the ability to do so. get management to agree to this, set up the appropriate security groups to provide the necessary and sufficient security levels and within hours you will discover management would rather do away with security altogether than not be able to breach their own rules and spy and interfere outside their pervue.
I'm surprised at an outsourcer swiftly implementing a change rather than responding with the standard: "This is not covered under our contract and must therefore be charged at our (outrageous) daily rate." But, this being the civil service, probably the conversation did take place and produced the response: "Don't worry, the taxpayer will cover it."
Probably near renewal time and the outsourcer mindful to present a good customer impression by doing everything asked for promptly.
The customer not being exacting and specific was just a bonus for the outsourcer as their account managers could up sell and offer to do that security role thereby saving the company future issues as the security team clearly where not experienced enough.
>>> our tale takes place some decades ago
Even the best and worst outsourcers had to learn from their experience. Having worked on both sides of the story, going far enough back (some decades?) the service was generally "all inclusive" of routine and stuff that could be considered day-to-day support and maintenance - which was what you expected as a customer of a managed service.
As time went by - and I guess little projects got larger and larger - they realised that this was a revenue stream they should exploit. Working for ten years before I retired from one of the big outsource players, in a bid and contract review team I knew most of the cash generating "arrangements" that could be used in outsourcing contracts! Many customers' buyers got wise to this in second (and later, if you can belive it) generation contacts which led to some "interesting" negotiations - and subsequent disputes!
Personally, when TUPEd to one of the big players in 2001 it took a lot of "eductation" to convince me of the way that the Request for Service process should work, as previously I'd been part of a central team delivering consultancy, service development and project management to the various businesses in the group as part of corporately-funded overhead.
Looking for pennies in the coat pocket to pay for the extra services ...
A similar issue:
Users were dropped into the Informix ISQL menu system at log-on end logged out when they quit the program. However the menu system allowed them to shell out by hitting '!'.
Solution? A quick program to mimic the menu system using the same sysmenu tables but without the shell-out option.
If you believe the Daily Fail, the 'IT Glitch' or 'Computing Error' was due to an Excel spreadsheet becoming maxed out. Not so much an IT issue as a PEBCAK one.
But why does the civil service feel the need to spend billions on software if the users just resort to using a spreadsheet anyway? Are they really that incompetent ?? (We all know the answer to that one)
Excel is the real software that all companies, big and small, run on. Most likely someone had made a spreadsheet or three, and when the Big Consultant came along, was told they were a critical part of operations. So they just created a rickety workflow around the spreadsheets.
Heck, they might have just created them on their own. Excel is involved somewhere in almost all Rube Goldberg IT disasters.
> Excel is the real software that all companies, big and small, run on.
It can work nicely in inappropriate functions. I once simulated an entire mobile phone network in it, without resorting to macros/custom functions/plugins and came up with reasonably similar numbers to a horridly complex C++ simulator.
YMMV, obvs, and the example of Track & Trace running out of cols/rows is a complete shit-show
The real WTFF is somebody thinking that a sequential series of items should go horizontally. 5000 years of scratching something down on some sort of media has concluded that lists go in the order of time's arrow, that is, *down*wards.
I mean, Exell even moves the damn entry point *DOWNWARDS* as you enter each item.
I once simulated the entire cooling system for an all-electric ship we were developing. It consisted of four separate cooling fluids in interconnected systems, three of the secondary systems were contained within the equipment, but the primary coolant was seawater. It took into account different areas of ocean, different seawater temperature and salinity, and even changes in barometric pressure. It consisted of several dozen (I forget how many) worksheets, each of which was fed by and fed back to another. Took me many months to refine, but eventually I could input any conditions the Electrical Engineers requested, and the spreadsheet would (eventually) spit out a summary of the operating conditions of the whole equipment. I still have that spreadsheet on a floppy somewhere, but have no idea whether it would still work on more modern computers.
If the information in there is anywhere near mission critical it should be removed from the spreadsheets and stored safely in a db. The spreadsheets can then read that info from the db safely and with security. the calculations need to be in the DB too if possible - stored procedures are not hard and can provide a level of abstraction that means you just change the SP when there is a minor DB change and you dont have to re-write 4000 random spreadsheets around the place.
Spreadsheets have their place (in /dev/null?) but some find them useful but they are damn near immune to good software practices that have evolved over the last 60 years and if people are going to fuck with them its worth making sure you can keep proper control over any useful IT in them given they are so easy to fuckup.
Also, why wasn't he punished? The process usually proceeds as follows:
Think wait a sec, I wonder if I could.....
Test run/POC
Report serious security risk to management.
Management ignores it.
Escalate the issue.
Get reprimanded/fired for "Hacking" or not following the chain of command.
Wait several months.
Then and only then will the broken "Fix" be applied.
Also in my experience getting a security "fix" reverted is impossible.
Working on a system at Scotland Yard, users were not supposed to be access anything other than the application, which loaded automatically at boot etc.
One persistant user discovered that if you hit F1 to load the windows help file, there was a "Run" option in the File menu, which meant that you could run any application or utility if you knew the name of the executable. This would give you access to such things as Progman.exe, Fileman.exe, command line etc.
Where I used to work, we had a machine in reception that was used to enable visitors to access a specific website. The way our IT systems guys implemented that was to set the home page on IE to the particular site, then disable as much of IE's interface as they could, and set IE as the Windows shell.
Fine, in theory, and actually worked quite well until one night, I was working late, and got a phone call from the front desk. The security guard was panicking because he had somehow accessed a porn site on this machine. The company had strict rules on this, and there was a camera pointing at the computer. He had a *lot* of popups on screen when I got to the desk and the machine was clearing struggling. I ended up having to close them, and after hitting the windows shortcut for closing windows well over 100 times, I eventually managed to close all the windows. Still had to reboot to ensure the correct web page was showing.
I once worked in a consultancy team for a very well-known city firm (stockbroker), you would recognise the name were I so indelicate as to write it down.
Well, the SysAdmin set us up with accounts on the workstations, each with a profile, and we got on with our work. A few weeks in, the head of IT security asked to check the internet history on my profile, not having accessed the Interpleb at all, I was somewhat surprised to discover some rather explicit pink sites in the favourites section. The head of IT security, however, was to surprised. The SySAdmin had used his own profile to set up ours and every one had the same favourites. He left a few weeks later.
AC coz, well, don't want any chance of the client being recognised.
This type of 'solution' was the standard procedure for the IT technician back when I was in high school.
Students kept opening the command prompt and running "net send * " messages during lessons. Rather than locking down cmd with a group policy, they tried to remove any way of just opening the command prompt. This included blocking Visual Basic from running, which meant that *literally every week* the A-Level IT teacher had to go down to the IT office and yell at them to unblock VB again so that they could actually teach lessons.
One of his other excellent management decisions was to set up web filtering in such a way that it only worked through Internet Explorer. If you brought in a USB stick with Firefox on it you'd circumvent the filtering entirely. To this day I have no idea how he managed that.
That IT technician has been a guiding beacon for me on how NOT to behave now that I'm in a similar role
One of his other excellent management decisions was to set up web filtering in such a way that it only worked through Internet Explorer. If you brought in a USB stick with Firefox on it you'd circumvent the filtering entirely. To this day I have no idea how he managed that.
I suspect this was done by setting the proxy address in IE. IIRC, this is (or was) separate to the system proxy settings.
Not the right way to do it™.
Well, Feynman did find a way to steal the last two numbers of a three-number combo on file cabinets, making it easier for him to crack the whole thing, especially since the locks would tolerate a few clicks either side of the official number. He would idly fiddle with the lock while engaging the owner in conversation, which he was good at, unlike many geniuses.
The true revelation was that many safes were still on factory default, and the default was widely known. When Feynman asked a locksmith about his secret, the locksmith just let on that he usually tried 50-0-50 since most users never bothered to RTFM to reset the combo, and nobody enforced changes. Sound familiar? Plus ca change. Anyway Feynman managed to get 1 safe in 3 open that way.
That is actually the correct response. The time and effort (and cost) of replacing all the safes at Los Alamos with high quality ones that Feynman could not open would have been impossible to justify. In any case, having a cleared safe cracker available in the event that someone forgets their combination could have been really handy. Besides, not letting Feynman near a safe does not prevent the owner using it.
I don't think that's the correct response. I have not read his account, although I've just put it on my reading list which is a LIFO stack so I will have soon, so I'm going with the summary outlined above, but in that case, it was a terrible response. If the comment I'm referring to is correct, the reason he could break in was that people were using a small set of combinations and the locks were shoddy enough to take several possibilities. The second problem is expensive to fix, but the first is not. Ban the default combination and require people to change it, with an explanation of how to do so randomly. That should dramatically worsen the chances of having a code which unlocks all the safes or brute forcing a small number of possibilities while distracting the safe's owner.
In addition, this was a government project with a massive budget holding state secrets; if security isn't relevant there, what is the point? Blocking one person from accessing safes protects you against that person, who already proved he was on your side by reporting this instead of stealing the information and choosing his next nation of residence. It does not protect you from anyone who read his report, heard it from him, or figured out the same thing. If they're not on your side, you won't find out until after they've exploited the problem. This is what happens to the least sympathetic of data breech victims; they know there is a problem, know why there is a problem, know how to fix the problem, don't fix the problem, and people suffer as a result. Don't do it.
When trying to second guess this kind of thing, one would do well to remember that the world was a very different place in the mid 1940s. To the vast majority of people, safes were just that ... safe. If it was under lock & key, it might as well have been on the moon. It was unreachable by mere mortals. In the other direction, the common mind-set did not include locking the front door of your house at night. There was no need.
Feynman discovered that in the three-digit combination locks, you could be one out, either way in each number of the combination for the lock to open. He also discovered that when the safes were open, which they were for most of the day while his colleagues were working, if he fiddled with the tumbler, he could figure out the final digit (plus or minus one) of the combination. He would write it down, and store it on a piece of paper that he kept hidden inside the lock on his own safe. People, at least the scientists did use 'random' combinations, rather than leaving the safe on the delivery setting, which is why he had to play about with the locks when they were open. It was the military personnel who sometimes did not change their lock combinations.
I stand by my comment above, replacing the safes would have taken too much effort and been too costly at the time, and keeping Feynman away from the safes was the most effective counter-measure that could be implemented quickly. In fact changing safes would not necessarily have helped, as a Colonel had a super secure safe, which had exactly the same make and model of lock, so getting safes with superior locks would probably have been a nightmare, both to do and to explain (easier to sack Feynman, probably).
Again, it doesn't help against anyone else. If he could do it, anyone else could do it. If he was going to do anything dangerous, he'd have done it before, or instead of, telling someone about the problem. It's missing the point and badly to think about sacking him; either you don't care enough about the stuff in the safes to go to the expense of updating the locks in which case you can ignore the problem, or you do in which case your attack landscape is anyone and everyone who could conceivably get to the safes. It sounds like they completely ignored this, and given that their project was being spied on by Soviet agents which they'd rather not have know the information, they probably should have put some thought into it.
It turned out that Feynman's skills were occasionally in demand. Of course he never let on that he knew any of the numbers of the locks: he closed the office door behind him, did his practised shenanigans to open the lock, then let a decent amount of time expire before welcoming in the impatient horde. He did however highlight one of the biggest security flaws: basing the combo on a number that someone else could discover, in one case the birthday of a daughter of the owner of the locked item.
The full story is in "Surely You're Joking, Mr Feynman", as told by the man himself.
IMO its a book that should be on every techie's bookshelf, partly because its very funny and partly because his approach to problem solving is well worth knowing about. Its excellent, readable history too.
His safe-cracking and code breaking exploits are fully covered in two stories, "Los Alamos from Below" and "Safecracker Meets Safecracker".
At a previous employer I found a problem where the supposedly secure corporate WiFi would let any device connect to it.
Raised it with management and then spent a few weeks working through the fix so that only company issued devices could connect to this network.
My job title was network security engineer....
If you find a breach raise it, management might not want to do anything about it but it will have to be recorded and if breached they will have to answer for their inaction.
In a similar vein I was testing rbac on Cisco switches, trying to get a limited command set for the helpless (help desk)
Turns out that missing from the document was the note to add the logout and exit commands to those you were permitting. Otherwise they could check the ports but never close the session down.
Fortunately I was doing it in test and making sure all was good before rolling it out.
The real problem isn't ham fisted security poliices, its the overal corporate policy of having IT work for corporate, they tend to be in the same management tier as services such as HR and Legal. This has two unfortunate side effects. One is that they get their input directly from senior management, the other is that you can't argue with them. Their unique cocktail of Imperial Directive and Not Really Understanding How Systems Work (plus Not Understanding the User Base's Requirements) means that developers, especially those who work with non-Microsoft tools (that is, "tools unrelated to day to day computing tasks on Office or simlar") spend a lot of time figuring out workarounds for their latest policies.
I've worked in numerous companies and this was always the rule. There was one exception -- an IT group who's leader actively sought to understand our problems and requirements and who worked with us to match Imperial Directives with real world needs. This manager was cherished, a rare breed indeed. The more usual situation was to have the newbe type turn up with new broom and institute networking policies based on his understanding of how networks worked (and failing to notice that a) the company made these products and b) we were the people who designed and developed them).
Sounds like where I am now. You won't find any of the embedded software IDE's in the list of "tools you can easily install from the server without calling IT", nor any of the multifarious compilers, scriptoids, magic builders etc. that we have to use from time to time. It doesn't help that many install programs ask for admin privileges whether they really need them or not. A notable exception is VS Code which installs in user space without being an admin-botherer, remarkable considering it's M$. You can also get some Eclipse's to install if you extract them from a zipfile and do your own jiggery-pokery.
Not Understanding the User Base's Requirements
That's kind of where I left the world of computing.
For a couple of decades there were no "proper" school IT staff supporting primaries.
It was types like me, semi-trained semi-amateurs who ran staff training and supported schools.
Towards the end the schools and my own teaching service had proper local authority IT teams to call on and servers and fancy stuff like that. I slowly began to be less necessary in that role.
But just when I thought I could forget about it and just concentrate on doing my own real job various kinds of new educational and and administrative programmes started to appear. The one thing they had in common is that they weren't teacher friendly. Either not doing the job that was needed or just failing to work for teachers. Classic problems such as having a compulsory field that only made sense or needed information that was only available to, say, the social work team.
Or a really good piece of educational software that defaulted to saving kids' work in a restricted area ( programme folders) where there was no save access, and also suppressed error messages so that no one knew where the work had gone ( which was basically into thin air). Added to which access to the teachers' section where they could change the default was somewhat obscure, to say the least. It took enough teacher time learning the educational aspects of how to use the programme with the kids without a whole extra burden of trying to figure this out. Needless to say a lot of software never got used in any of the many schools I used to visit.
So I found myself trying to a) help teachers around this ( including changes to defaults where possible) and b) sitting with the designers and administrators of the software and saying things like "No teacher, or indeed anyone who wsn't a Social worker/Psychologist/etc would ever think that this sentence meant what you seem to think it means." or "You can't make that a compulsory field because teachers don't have that information" and so on.
But the biggest group was software that just didn't do what the teachers needed it to do, because someone somewhere didn't realise that this wasn't how teachers worked. One, I can't remember what it was, needed teachers to be able to leave the classroom "just for five minutes" for some kind of setting up tasks. Like, yeah, you can just leave 30 7 year old kids when you felt like. What ever it was it never got used.
The outsourcer knew that blocking all avenues for DOS commands was impossible if users were allowed to save files, so they showed them what has to happen if they really care about that particular security measure.
i.e. it was a way to get the government to abandon, modify, or stop trying to enforce a security measure that was incompatible with being able to actually use the PCs.
Why the down votes?
When you know something inside out and line management instructs you to do it in spite of your protestations that it will have wide ranging consequences because of a,b and c - where does responsibility lie? Society.
Sometimes you have to let stuff break to reinforce that it is critical infrastructure and needs to be respected. This is how people learn and always have done. Its called learning from your mistakes.
When your mistakes have always had negligible consequences you are much less inclined to take advice from others because their advice has no reference value in comparison with your experience. The lessons learned through experience have been 'low value'. If, not when, the big one comes its cost is understood.
Hence the expressions 'learned a valuable lesson' and 'valuable experience'. This is why allowing people to make mistakes early on gives them an appreciation of the risk space they inhabit. The shame of it though is that 'learned experience' takes a long time to influence 'organisational culture' unless its an experience.shared simultaneously and widely. Even so, a widely experienced phenomena might bring about undocumented behavioural changes - because the common experience means it doesn't need to be discussed or recorded, everyone knows. Until a generation arrives that has no direct experience, no first hand knowledge - and no inclination to learn from the past because they need immersive experiences to pique their interest. They can't learn from black and white media.
Thats why the morons parroting anti-vax rhetoric, 5G and windmill cancer 'risks' are so dangerous. They haven't lost 13 out of 15 children to polio, cholera, typhoid, TB, smallpox, plague, famine or eating the wrong mushrooms. When the apocalypse comes they'll be looking for wifi and amazon prime to survive.
A company I worked for banned the use of a query tool in interactive mode. So our IT Manager ran a batch file that changed what he thought was unimportant data in the production database. He got a bollocking for that. Later, after I took over his job, I found out that the changes blocked the purge of old data and cluttered even our enormous 20Mb disk.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
