The future is now, old man: Let the young guns show how to properly cock things up We straddle the worlds of IT and telephony in this week's episode of Who, Me? where a reader fails to consider the tinkering of someone too young to know better. "Al", for that is not his name, was looking forward to a well-earned retirement after a career spent at an IT giant working on everything from compilers and operating …
Had a very experienced network engineer make a "small" change that took out half the network (one that he mostly built).
Fine - we all make mistakes. His was not having any change paperwork raised (despite a company wide email stating that no paperwork meant no change).
But rules are rules, right? So we lost an experienced network engineer....
Since you mentioned the Police National Computer ... years ago I worked on a project to support access to the system from some new boxes. Why that project was needed is a story in its own right, thanks to a salesman "misunderstanding" the requirements, selling something totally wrong for the network, and then we had to bail him out.
But in order to test and demo our work, we had to send queries to the real PNC. But because of security/privacy concerns, the only query I was allowed to run was to lookup my own car registration.Unfortunately for readers of the Reg it didn't show as stolen or written off. But I did wonder if analytics (even in those days they did some, although they might not call it that) would show a concern about repeated queries for a particular vehicle.
"thanks to a salesman "misunderstanding" the requirements, selling something totally wrong for the network"
You mean unloading an obsolete bit of kit that provided him with a large commission for shifting the archaic boat anchor?
Seen it all to many times ... Yet another reason why Management and Marketing should have absolutely no hand whatsoever in purchasing new networking kit.
I used to do support and installation work for a company writing HSE training software. Sold to big councils, police forces, fire services, banks, etc.
Usually my first task on site with a customer was finding out what special extras the salesman had promised but forgotten to mention to the programmers. The little changes like promising a Welsh Police force to have everything translated into a bi-lingual edition. So on the Go Live \ Install day I was usually explaining to the devs of the changes that had to be hacked into the code ASAP.
As to security - haha, don't make me laugh. This was the kind of developer who wanted to always leave every folder fully writable to all on the web application he was writing.
This is, sadly, rather common.
I've always tried to make developers develop on systems that are configured as they would be in production.
Salespeople just chase the money. Users want the latest wizzy crap, regardless of whether its actually supported or not. Developers just want to make it work. Security want it to work securely. Support teams just want it not to fail.
I'd not limit them to keeping out of the purchase of networking kit - almost anything. I've known them screw up the purchase of printer paper. That was when office printers were tractor fed and the company bought paper that tore down to A4 size. One mangler was persuaded to buy from a new supplier who could sell the paper for ~£1/box less (about 10%) - a lot when you consider they bought it by the van-load. Unfortunately, the new paper tore down to quarto size - almost every document across the organisation now had to be reformatted when printed. Of course, the reformatting was done before printing a second copy as you didn't find out it was wrong until the first copy had been produced. Paper use increased, not quite doubling, but a lot more than 10% - although the real cost to the company was the extra time spent.
No, keep management and marketing as far away from anything...
"One mangler was persuaded to buy from a new supplier [...]"
In my computer bureau days - some customers provided their own printer paper. One particular weekly run was hated. It used a multipart carbon paper which had a remarkable tendency to come apart at the sheet perforations. We had to spend time doing test prints to get the alignment looking safe - before we even dared to risk the live run.
We often made the comment "Someone had a nice lunch out of that salesman".
another reason why Management and Marketing should have absolutely no hand whatsoever in purchasing new networking kit anything.
There you go, FTFY.
If you cut their hands off, they'll find it a lot harder to get themselves in all sorts of trouble!
(icon, just in case any readers think I'm a firm believer in the Saudi justice system)
Management should, and arguably must, make the final decisions. BUT.
The problems appear when the management is of the "I know better" brigade who don't actually work as team leaders, but have an arrogant belief in their own abilities to decide everything, and this type tend not to think that there are any service specific aspects to be considered. To them managing a team in one industry/department is no different to managing any other team/industry/department and they make the same decisions because to them every situation looks the same. Or to put it another way, they just don't believe there are any unknown unknowns. Anything they haven't seen is presumed not to exist. These are the "seagull managers" mentioned elsewhere on El reg recently.
I worked for the vendor who used to supply the PNC and one of our salesmen on the account was likeable enough but managed to find himself on the front page of the Sun for walking out of the PNC with a removable disk pack full of data under his arm. Without going through the formalities of seeking any permission to do so. During the Falklands war.
Proably the same bloke as the one who sold the unsuitable kit.
thanks to a salesman "misunderstanding" the requirements
Back in the early 80s I was an academic computer scientist and used to drink in a pub just over the road from a firm of dodgy box shifters flogging early PCs. Someone had told them what I did so one evening one of them asked me what the difference was between multiuser and multitasking systems. I attempted to explain at a level that someone who was basically a second hand car salesman with a new job could understand, and then naively asked why they wanted to know. The reply was "well, we've just sold 2 dozen MS-DOS boxes on the basis that they're both multiuser and mutitasking".
Heh. Many years ago a colleague set up an inter-agency computer system whereby - for the first time - one agency could directly query the document stores of another.
"We need a document to retrieve" says My Colleague.
"How about my driver's license?" says My Other Colleague.
"OK" <enters number>
17 unpaid tickets showing. MOC went white as sheet as he realized that not only was he a scofflaw, everyone in the building would shortly know that.
Everyone else stayed silent until he has left the office "for an early lunch" at which point hilarity broke out.
"Speaking of the Police National Computer, you CAN'T delete data from it accidentality or deliberately, at least according to the Police"
They are liars, then. Even a WORM drive can have specific files "deleted". Simply copy the files you want to keep to another WORM drive, then thermite the first drive. Simples. (Yes, I have done this, legally, for reasons.)
"who are still illegally holding records which by law they are supposed to have deleted by now."
How much are they paying which Judge, and does the Press know?
"How much are they paying which Judge, and does the Press know?"
Yes, the Press know. I'm pretty sure El Reg has covered it too.
I don't remember the exact details and can't currently find the story, but IIRC it was related to GDPR and retention of unneeded data such as where people where arrested "on suspicion" and the released without charge. There's a limit on how long that data can be kept and the Police still have it today, and that data can show up in security checks, even though the person may well be completely innocent.
"it occurred to me that the engineer interface into the exchange would have been a prime hacking target for obtaining free calls and anonymity on the phone network."
Yes, it was. That's why sensible people had it locked down unless it needed to be opened for a short time ... and then locked it down again immediately the need passed. Sometimes this was as simple as unplugging the modem attached to the executive port ...
"Times were a bit more innocent 20 years ago."
No, no they weren't. That kind of innocence and naivety was lost by the early 1980s ... The Morris Worm of 1988 was a boot to the head of those who hadn't got the picture quite yet. Any hold-outs with this kind of open system by the year 2000 were idiots whistling past the graveyard.
That kind of innocence and naivety was lost by the early 1980s
oh i dont know , when the FBI were kicking in the doors of Steve Jackson games on behalf of the NSA and their newly formed hacker taskforce in about 1993 they found a little nest of some of the more famous hackers, still enjoying a worls of virtually no security.
Mainly default passwords i'd imagine.
Uh ... you do know that it wasn't the FBI, it was the Secret Service, it wasn't in '93, it was in '90, and all they found was games, and a user guide to a specific game purporting to be a "hacker's manual", not actual illegally obtained material, right?
Jeebus ... has it really been over thirty years? I'm gettin' old ...
When I worked on VOIP systems, I often had to deal with support queries like "why is my exchange getting hammered by SIP registration requests?" The reason was always because the relevant ports hadn't been closed properly and the box was under attack from persons unknown trying to get free calls.
Operator Error occurs in every industry. Usually starting with the Boss, Owner or YetAnother C* bellowing "I DON'T CARE, JUST LET ME DO IT!".
Somewhat sadly, the proverbial lathe chuck-key never embeds itself into said muck-a-muck's forehead when he flips the switch inappropriately. Instead, others suffer.
::sighs::
Pun not intended.
The Morris Worm only prevented the most egregious examples of misconfiguration, it didn't trigger a sea change in re-architecting software.
In the early to mid nineties most operating systems still had a load of holes, some of the popular mail server products were file sharing based, and the computing landscape was pretty diverse. Towards the late nineties everything was starting to consolidate around Windows and a small number of Unixes, and exploits moved from academic misadventure or determined hacker, to hacker groups illustrating holes in an attempt to convince companies to architect secure-ish software, to more widely available exploits and script kiddies.
I will agree that by 2000 if you were operating an insecure system you were living on borrowed time. Internet access was widespread, and companies were starting to get on the ball, including automatic software patching by companies such as Microsoft. I've just shredded some of my work notebooks from the late nineties which included a list of patches to manually apply to NT (probably 4.0) to make it secure, a different world!
Indeed. In mid to late 1990s the Scripters of Coffee had fun showing the holes in HPUX of the time. Some of the exploits were horrifyingly simple, one was deep systems stuff shell to perl to PARA-RISC assembler cascade. I used to enjoy trying the attacks I understood on my sandpit. AIX had a few, one in particularly being useful when someone forgot root password. I would not dream of repeating this now outside of virtual machines. One does not know just how much is in payload, connecting tho where. Especially as even CPUs have their own OS inside.
"it didn't trigger a sea change in re-architecting software."
Actually, yes. It did. For one thing, it triggered the formation of CERT by DARPA.
By the late '80s, most OSes might have contained exploitable holes, however those exploits were no longer accessible outside the corporate network. (Assumes adequate capability in charge of the firewall, of course.)
The corporate world consolidating on Windows just made life easier from the crackers perspective ... Monocultures in computing are just as bad as they are in farming, and for the same reasons.
Contrary to popular belief, Internet access was widespread before 1990.
> Contrary to popular belief, Internet access was widespread before 1990.
Your definition of "widespread" seriously lacks. Windows 3.1 didn't have it TCP/IP built-in, and the package for it came around 1992. Same applies to DOS. If your definition of "widespread" is "the network spans around the globe, even if less than 0.1% of the population in the handful of participating countries have access" you seriously got a weird viewpoint.
"Your definition of "widespread" seriously lacks."
Does it? By 1987 email addresses were normal on business cards. I have a Pagemaker template to produce "camera ready art" for 10-up business cards from that year. It includes an email field ... dBase classes of the era strongly suggested including an email address field in corporate directories and the like. If that's not wide-spread enough for you, what is?
I had recently managed moving our Siemens PBX, Telephony servers & call centre servers into a shiny new server room. I had purchased a network card for the system's UPS so we I could monitor it along with all the other services (knowing something is misbehaving 30 seconds before the users being a Good Thing (tm)).
So I rang the suppliers techies and quized them on procedure, did the change thing, as no downtime was needed I proceded on a Monday morning with a Sunday hangover. I put the UPS in to pass-thru mode prior to fitting the card ... Ker-Thump!!! ... both racks went down. It took me most of the day to recover everything.
Basically our sparkies had not wired in the pass-thru. I was quized by the IT Director and the Finance Director and while they agreed the outage was not my fault I was unfortunately the PM on the system move so I should have checked the sparkies work. Obviously the dept thought it was hilarious.
A bad day ended better, went to the pub.
As someone who has just purchased a small PBX (Panasonic TAW848) for non-work experimentation, I'm getting a kick out of this.
First thing I did was to take out the SD card containing its programming and make a copy of it. Then verified that it would work, by installing it and power ing up the system. I've also got the PC programming software (runs over a USB cable) which shows the current setup, lets you read the logs, etc.
These things are quite inexpensive now, since nobody uses landline phones anymore...
"nobody uses landline phones anymore"
Oh yes they do - whenever privacy, low failure rate and local power independence are required. Until of course OpenRetch force us all onto fibre. There's a lot to be said for line powered comms with point to point connectivity.
I do recall when I was Application Support Manager for a sizeable telecoms company and found, quite by accident, that the installation engineers had deliberately installed a support line in to the main switch. The switch it self was HUGE, thing mainframe and add extra cabinets. These forward thinkingguys had put this modem in so they could remote dial in to the system and patch any faults. But they hadn't thought of any security. None whatsoever. You dialled in to the number and you had a console, albeit at 9600 baud, directly in to the internals of the switch operating system. From this you could do anything you wanted: Want a new number? Easy. A free-for-life phone line? Simples. Whats more, this went in at such a high level nothing was even logged so we never knew if this was hacked.
That's why we used dial-back modems.
Cognizant Engineer dials in, enters "password" for modem. Modem hangs up. Modem looks up "password" in a list, and dials back the appropriate number. Cognizant Engineer's modem answers, makes the connection, and is then presented with a login for whatever bit of gear needs twiddling, this time (hopefully!) with a proper login/password pair. Once set up, it's easier to do than to describe. Side benefit is the owner of the bit of kit being worked on pays the long-distance bills.
That's what I used to do with systems I supported. Multitech modem, with dialback numbers configured. Bonus was that the company paid for the call (except the inital call to initiate the callback) which with metered calls could easily add up.
As you said, dead easy to setup. And yes you could set it up so that you could also provide callback number with your "password" for the modem to fial back (only to be used at your peril of course).
The ILR station I worked at in the 1990s had a telemetry connection to the transmitter(s). This was some kind of Epson portable computer thingy with a built-in tape drive which used dictaphone cassettes. The modem used some really odd kind of three-tone modulation which normal modems couldn't understand. Didn't stop people trying though, and we heard everything. Strangely musical - more so than a normal modem anyway. Occasionally you'd get people misdialling and wondering why they couldn't order a takeaway or book a taxi.
M.
My ISP has TR-069 set up on the modem/router they sold me. (Due to the peculiar connection, the modem is essentially only available direct from them.) The idea is that they can rewrite the configuration if needed. Two problems, though. One is that they could upload a totally different firmware to the unit without even asking first, including "features" like sending copies of all INTERNAL traffic back to the ISP. The second, and far bigger, issue is that they didn't bother to block access to the port from the internet at large - anyone in the world can connect to it. And there's no way to turn it off. (Which is why I have a separate router just past that, as the only device connected to it.)
The second, and far bigger, issue is that they didn't bother to block access to the port from the internet at large - anyone in the world can connect to it. And there's no way to turn it off.
Seems like you need to rewrite the configuration so it is possible to turn it off.
Tried that. It wouldn't allow me to forward the port to some other (non-existent, obviously) internal IP address. The only way I found it was when it rejected a broad "forward all ports from x to y" setup, and I narrowed it down to one particular port number. I eventually found an undocumented, user-should-never-see-this "configuration" page for it - which still couldn't turn it off (though I did mess with the settings in a way that hopefully will prevent connections).
I tried calling the manufacturer... who wouldn't talk to me because it was an ISP-branded device, all support **MUST** be provided by the ISP. And the ISP's techs had no idea what I was talking about, as they're Tier -1 support, and TR-069 isn't in their script.
One of these days, there's going to be a major news article about how some cracker got the TR-069 password to an entire ISP and turned all that ISP's modems into a botnet. And me without a destination email address to send an "I told you so!" about it.
"Was there an option to play thermonuclear war?"
That was just a batch file located on the local machine. Forced the modem to make all the correct connection noises, but didn't actually require a telephone line. Echoed the "responses" from the "far end" to the console. The version that used ANSI.SYS[0] to draw the screen was almost convincing.
[0] Or NANSI.SYS if you had clues.
I heard that some young chaps wanted to move some solid state disks about 6 inches without powering it down. These were about 5ft high and 3 ft square and heavy. So the lads following the best health and safety procedures put their backs against it, and pushed with their legs. This was fine, till one of them found he had caught his belt under the recessed emergency power-off "pull knob" and did an emergency power off as he stood up!
If you knew how, you could program them to walk by themselves.
When I was at DEC, one of the guys learned to make the washing machine sized disk drives "walk" across the floor ... I had to fire him when he did it in front of Ken Olsen, who was visiting our lab. Was very hard on the hardware.
I've seen something similar happen on an ICL2900 mainframe many, many moons ago. Guy managed to catch a reboot switch with the cuff of his suit jacket while moving a terminal around. 3 days it took to get the server back up!
I never wore a suit jacket or had my sleeves down in a machine room after that.
I remember it like it happened yesterday - but that was over 35 years ago. I feel *old*!
"I never wore a suit jacket or had my sleeves down in a machine room after that."
Shit, I never even wore a tie! Ever get a tie caught in a cooling fan or a line printer? There's a reason that ties were fair game for anyone with a pair of scissors at most early Silly Con Valley companies.
The only use for a tie is as a handle when trying to shake sense into the wearer.
Not in the land of Gorton it wasn't.
Dress code? Motorhead t-shirt and bike jacket one day, suit & tie the next. Just to keep people on their toes.
Thos dev hall "white" coats. Some of them had been unwashed for so long that they were devleping civililzations in the pockets...
Takes me back though.
The last 9-5 I interviewed for (in 1989), I was wearing my racing leathers. When the interviewer queried my choice of "uniform", I pointed out that he had asked me to drive up from Palo Alto to South San Francisco by 10AM ... and had called at 9AM. I knew I could make it on the bike, but there was no way I was driving the Bayshore without armor ... I got the job.
The 9-5 prior to that, I wore the same outfit, for similar reasons. When queried, I responded along the lines of "are you hiring an engineer or a fashion plate?" ... They made me an offer. I counter offered, they hired me at my price point.
I was talking to a "white coat" at ICL about 30 years ago. I asked "Why the white coat?". He said it was a holdover from the days when tens of thousands of relays created a lot of dust, and high voltages concentrated that and the naturally existing dust ... and then they threw in paper tape, punch cards and mag tape, all of which create more dust. So the coats were to protect the clothes of the wearer.
I pointed out that that was hardly a necessity anymore ... He just smiled sadly and said "We're British. We like uniforms."
That reminds me of the R&D manager who backed up the company NetWare servers to a weekly tape. He took that home too.
Then one day they needed to access the backups due to a disaster - ceiling collapsed and flooded the servers. It was that day he learnt all about how he needed to wind tapes on past the headers before using them... many useless tapes in a box. And all those wasted hours on taking the backups.
The problem is not just headers. One place I worked, they used to cycle tapes weekly. So each day the company chief typist (who'd been given no training) took the next tape from the rack, inserted it in the drive and pressed rewind. As soon as the tape was fully rewound it ejected, and the typist thought that meant it had finished backing up even though it ejected almost immediately. As no training had ever been given to anyone, nobody spotted the problem until they needed a backup. All the tapes were fully rewound and blank.
When I was IT manager for a division of a university, I had one guy working for me who had been there for years and was clearly going to stay there until retirement. So every month a clone of the full backup was taken and he took the tapes home, and brought back the previous month's tape. That was what we could afford for "disaster recovery".
We both agreed if there was ever a big enough disaster to take out both the building we worked in and his house three miles away, those who were left alive would no longer give a damn about their files.
For many years, a long time ago, I took home a copy of all our confidential files, because the powers-that-be couldn't understand the the idea of providing off-site backup*. Eventually I stopped, not because there was off-site back-up, but because there were then rules about taking this stuff off-site, and by then there was at least back-up to a separate USB hard drive that could be swapped and put in a different room.
*The risk of losing all those records seemed far higher than the risk of having these data stolen, particularly as there was some sort of password protection (not, if I recall correctly, encryption).
In the dim and distance past, I was working for a company that had an "office" (Jolly pad) in London. and because this was an office it needed to be connected to the Centrex phone system.
As a PFY at the time I was sent down to make sure it worked, with senior management not being trusted with such things. I was assured that it was all installed, I just needed to make a couple of test calls, job done. My plan was then to change my ticket back to Aberdeen to the weekend, find a cheap hotel and see the sights of London.
I get there, the fibre is not installed the MUX is not installed and the patching is not finished... Several calls to my Manager and heads started to roll and I was to remain on site until the work had been finished.
A couple of days later the hardware was installed and all was ready and "working".
It was fine if you just used one phone, but would hang up calls if you tried such ridiculous things as making/ receiving another call on another handset.
Another couples of days working though it and lots of angry calls from me to Cable and wireless as each of their fixes failed to fix the problem.
Finally sorted about 2 hours before the CEO who definitely wouldn't want to see a PFY in his "office" arrived.
Returned back to a pat on the back and very loose interpretation of what was acceptable expenses.
I always said I would retire when someone could do my techie trouble-shooting job as well or better. There came a point when I realised that some ambitious youngsters were trying to cut me out of the loop - to boost their careers towards being managers. I cleaned up their self-made disasters for a while - then I decided it was time to retire gracefully and leave them to it.
I was responsible for giving out free calls for a month by mistake.
Decades ago I worked for a midsized, long gone, telecom in the US. We got our call records on paper tape, maybe a half a foot to a foot wide, which had 32 bits (not bytes) per call. For a reason I can't recall, we needed to start using one of the bits. It wasn't marked as used so we claimed it and told the switches to set it to zero (programming the switches to use it would come later).. You can guess what's next.
This was the dawn of computer to computer telephone file transfer and our call duration bits were designed with person to person in mind so they started to overflow and someone decided to use the bit (that I thought was free) to handle this. As a result, the billing for that month gave customers free time on our network for whatever files they were sending.
I was exonerated because the programmer who made the change hadn't documented it (and had left the company).
There was a time when I was nominally responsible for multiple site phone systems. When I started, these were Nortel PBXs, some in place for upwards of 20 years. No one had any Nortel training, and the documentation was... spotty. These were robust systems, but nothing lasts forever, and getting parts was becoming difficult. My CIO didn't want to spend his training budget on Nortel classes, so all I had to go on were notes by predecessors.
So, a monthly phone bill from one of the smaller sites suddenly rose from well under $1k to over $40k. Someone had called into the PBX remote service line, taken control and had made a lot of calls to Hong Kong and Singapore. I was promptly thrown under the bus for allowing this to happen. The CIO figured that making me the scapegoat would take the heat off him, and happily told the Finance VP that I was to blame.
Alas for him, I had my CV and my emails showing he'd denied me Nortel training (or even software manuals). I had had no Nortel experience prior, he'd picked me to expand the network and switch everyone to VoIP. That CIO was nothing if not a skilled politician. He switched gears instantly when he realized that he'd go down with me, and managed to use the hack to get the VoIP project budget increased. That kept me and my mates busy for years.
My fix for the Nortel hack problem was incredibly low tech, as the systems really didn't have security in mind when made. I disconnected every remote service line, marking it very clearly. The few times I had to access those beasts remotely, I'd call up the local tech (or power user) and have them plug that line in, then call to have it disconnected again afterwards. No, I never did go to Nortel class, but finally got a programming manual, which I gratefully binned along with the last Nortel PBX.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
