Techie's quick cure for a curious conflict caused a huge headache Welcome, gentle reader, to the safe place we call Who, Me? where Register readers tell us about the times when their brilliance shone a little dimmer than was needed to reach a brighter future. This week, meet "Bruce" who was once an IT Helpdesk Analyst for a "large bluechip healthcare organization" in Australia. That sounds a …
Don't you just hate it when the technology is actively out to get you.
Sure, naming a single random PC after the company is somewhat maximising the chance of something unfortunate happening, but the chances are still so slim that it'll cause any major issue that it just reinforces the point that computers are vindictive buggers and You Can Never Trust Them.
Especially in a company that installs a web server on every PC and has no use for it.
That was a brilliant idea there, or a wonderful absence of checking that the image had only what was actually needed.
In either case, if I had been in charge, someone would have been raked over the coals for that (not Bruce, I hasten to add).
How many people does it take to make a logic bomb?
- A systems programmer who puts their utilities in /u and makes /u the first directory in the search path, as some of the utilities have the same name as (and supersede) system utilities.
- A user who typos `ls > grep` in /u, instead of using "|"
No real problem so far. /u/grep isn't executable, so is ignored.
- A systems administrator who decides everything in /u should be executable...
It's surprising how much of a Unix system depends on "grep" and will fail if you break that utility!
Your point stands and you have my upvote but I should think /u would not have been left world writable by an otherwise competent system's programmer. The scenario you describe should only be possible to the same extent that the user accidentally hit enter after typing the same thing in /bin
A good point - The end-users got an application menu rather than a command prompt, so it was someone with root privileges wot dun it.
`ls | grep` is also an unlikely command, but that's what I found when examining "/u/grep"
As an apology, I'll offer one of my own "Who, me?"s [1]: `last | grep reboot` [2]
Except I inexplicably missed typing "grep", and "reboot" doesn't ask whether you really meant it...
[1] I make a new-and-interesting major mistake about once every 7 years. I'm overdue another one..I've warned my boss they really should fire me before that happens!
[2] Other, safer commands are available, e.g. `who -b` or even `last reboot`
I am *currently* working in an organisation (also in Australia, ironically) where the default server build - deployed as the baseline on every server - included IIS until literally this year. I had kicked up a fuss about it for at least 2-3 years until I finally convinced the sec team to take an interest.
Of course, nothing has been done about the 100s of existing servers still running IIS services for no reason.
Don't you just hate it when the technology is actively out to get you.
The more capable the device, the better it is at Resistentialism.
computers are vindictive buggers and You Can Never Trust Them
I'm convinced that computers start to work properly when I'm around because they know all the horrible things I can do to them. One hard stare and they mysteriously work again..
Keep them in fear!
(Much like my cats do with my dogs)
Who hasn't ?
Just a few weeks ago, I was at a customer site during a migration to Exchange. I was asked to add an Internet Address to all the groups in the Name & Address book. The format was supposed to be <nameofgroup@companyname.com.
I'm not an admin, so although I found the request curious, it is not for me to speak up on such a matter. I created the required script, ran it, and all appeared to be well.
For about five minutes.
Then calls started coming in to the Helpdesk. People were receiving weird mails that they shouldn't have and didn't before. It just so happens that some groups had a group name that was very similar to some redirections defined on the internet side of the portal, and anything coming in from outside was now sent directly to all the names in those groups.
Oops.
Needless to say, I was asked to revert the change post haste, which I did in record time.
I then left them to devise some new format which wouldn't create a new dogpile of issues. Maybe by checking incoming redirection lists first ?
A migration *to* Exchange?
Especially if your main MTA is sendmail based and one of your sub-comanies stealthily 'migrates' over to Exchange 5.5 - which claims to be able to do SMTP pipelining but does so in a way not compatible with the SMTP standard (surprise, surprise) and so only bothers processing the first email in the pipeline..
As happened to me many years ago. Fortunately, sendmail allows you to over-ride the facilities offered by a remote SMTP server so, once I'd realised what the error was, it took me 5 minutes to research the format of the over-ride, update the sendmail.cf and recompile it and push it out via NIS
Yep, I've done something like that before.
I reset a little wifi router on the office network back to factory settings, so that I could log in and reset the password that someone (cough) carelessly lost. I learned a few things from this small change. A) What a DHCP server was, B) most home routers have this enabled by default, and C) it's a bad thing when you suddenly have a second DHCP server on a network that doesn't talk to the first and hasn't been configured with the correct network information. I found all of this out when PCs on the network started dropping off randomly. The IT Director asked if anything new had been plugged into the network, and I remember the sinking feeling I had when I remembered the reset router...
BTDTGTTS
With a previous hat on, we'd sometimes recycle old routers as access points and/or switches - which is fine as long as you a) disable the DHCP server, and b) make sure it's not using the same IP address as you real router.
On one occasion, we got a call from a customer - they'd had a power outage that had shut down their server, and after powering things up again, some of their PCs weren't working properly. As "the network guy" I was dispatched to site, and found the affected PCs using addresses from a different IP range. On consulting colleagues, it transpired that one of them had indeed used an old router as a switch - but because fo the way Windows DHCP works (not RFC compliant I might add) the clients are very "sticky" and stay with the main server. That is, until the main server isn't there and they go lokoing for another DHCP server ... Yes, my colleague had neglected to turn off the DHCP service, but the network had been fine for weeks.
Elsewhere, I'd been sent to clients' sites to find a motley assortment of old routers being used as access points & switches - some in parallel, some cascaded, but generally not working. Bonus points for adding Sonos devices that then create wired-wireless bridges to further confuse the picture.
The bottom line of all of this is the usual: the moment you start working with an outfit that feels it's too big to comply with clearly defined open standards you know it will eventually hit you in the neck.
Name resolution, Kerberos, SMB, heck, even the simple principle of not installing and running more than you actually need to preserve resources as well as keeping things safe - there is that one company that always thinks it knows better and will happily ram their approach down your throat. But hey, it keeps people employed..
Two quick downvote on tales that admit causing problems with extra routers and DHCP.
I wonder if there is someone here with a bad conscience about doing the same thing but never admitting it; now they're worried that you'll give the game away by describing the symptoms, making a colleague think "Hey, that sounds just like what happened last month then Joe plugged in that router, which he swore he knows how to configure".
After seeing some quite inexplicable downvotes over the years, I'm convinced that there are some commentards who get some weird buzz out of hitting the downvote arrow, and another lot that have been staring at a monitor too close and for too long so can't really see what they're doing.
I'm convinced that there are some commentards who get some weird buzz out of hitting the downvote arrow
I have long hypothesised that we have a Phantom Downvoter as it's frequently only 1 down vote on an otherwise heavily upvoted comment.
There are apparently commantards who think anything positive about space exploration is a bad thing, and a waste of money.
There are other commentards who have alternative opinions about physics and hate anyone who can debunk them.
There are still other commentards who think that the entire multi-national space effort is a hoax.
There are even a few commentards who hate it when a post makes it clear their favorite SciFi TV show got science wrong.
I'm sure I missed quite a few.
Bottom line: If you are an expert on a given subject, and post about it, there will always be someone here on ElReg who will become outraged and furiously hammer that downvote button. Just ignore it and move on. Or do what I do ... block[0] the upvote/downvote display. It's fairly useless, so no loss.
[0] Do you think that addblockers only block adds?
One nameless, faceless blob of grey goo[0] recently admitted to downvoting at least one article just because it had too many upvotes, and it hinted that it had done the same in the past. It didn't say how many upvotes were "too many".
Another claimed it regularly did the opposite.
::shrugs::
[0] Most people just call them "AC".
Oh God. DHCP. We were building an industrial controller based on a well known RTOS. We used a DHCP server on the controller to allocate IP addresses to I/O. If the configuration indicated there was only one I/O unit on a connection we auto allocated a single IP address.
IO was connected to ETH2 through to ETH5. ETH1 was used for the supervisory (programming) network. In the early stages of development we had the DHCP server enabled on ETH1 and in error connected the controller not to the private lab network but to the company one.
You can see where this is going. When everyone came in in the morning and switched their computers on our controller gleefully handed out ‘192.168.10.10’ to anyone who asked.
The issue was fixed very rapidly and an edict sent out that no unauthorised items were to be connected to the company network.
sounds eerily familiar.
Quite some time ago, when I was still green and a little wet behind the ears, I thought it would be a good idea to bring in a spare wi-fi router I had at home into the office, so we could get wi-fi. Plugged it into a spare network socket, and all worked quite well, until we started getting reports of devices dropping off the network. Sure enough, was acting as a rogue DHCP server. Lots of panic in IT office, i blushed, stuck my hand up and unplugged it. Problem caused, problem solved.
I had someone do a similar thing and it took me an hour to figure it out.
One network segment stopped logging users on, anything that had been left running was fine but anything turned on that morning was failing - because said machines auto connected to the domain (and I didn’t have local credentials) I couldn’t see much. So after checking all the network and dhcp relay settings I was a little confused.
Then I got my own laptop out and fired up wireshark, I had already logged in on a working segment so had my usual access. All looked fine until I noticed the dhcp packets were coming fr9m the wrong address. It turns out someone decided to use whatever (it was a long time ago) software was needed to configure thin terminals, this software ran its own dhcp server and it had been plugged into one of the production segments.
Once unplugged and some reboots later all was working, the server had a static ip and was turned on during the previous afternoon when everyone was in the office and working so no dhcp traffic was needed until they rebooted the next morning.
Similar thing a few years ago. Mid-morning, workstations were dropping off the network and a quick look showed them using 192.168.1.x which wasn't the workstation LAN range. Not having particularly intelligent switches at that time we could only isolate bits of the network and ask people to reboot to see what address they got while someone went searching for any rogue SSIDs. We finally traced it to a meeting room where we found a previous generation BT HomeHub hidden behind a cupboard and plugged into a network port. We recalled that the Sales Director was constantly bitching that his phone or laptop couldn't get internet in parts of the building, and he had previous form for plugging a cheap switch in behind his workstation to get a connection for his Mac, which we didn't allow on the LAN, and which slowed his machine to 100Mbps so a core application was timing out.
We took the offending device to the carpark, took turns at reducing it to scrap with a lump hammer and left the remains on his desk. Naturally he went apeshit until I pointed out that the company had lost some 200 man hours of work owning to his unauthorised network modification and I would be quite happy to discuss it with HR. He learned his lesson this time.
A) What a DHCP server was, B) most home routers have this enabled by default
I've got 3 DHCP servers on my home network - two Win2019 AD server VMs and, as a server of last resort [1], the firewall. Each has a (non-overlapping) range that's big enough for all the devices on my home network.
Which reminds me - must re-instate the guest/IoT wifi network..
[1] Which actually hands out the most addresses since it's a single-use server rather than a VM running under proxmox.
Back when I started my first proper IT job back in 1998 I was sent to the US head office for 6 months training. As part of my training I had to set up and install a Communication Server (a server that talked to third party systems). I was pointed to a clean PC, given the Software and an instruction manual and told to get on with it. Simple enough.
The first step was to install Windows NT Server on the box, which I dutifully did. I then proceeded to install our own software on the newly installed NT Server. While merrily working away installing and configuring our software on the nice new NT Server, I noticed out of the corner of my eye, the service manager race past my desk (which was quite impressive as he was getting on a bit). And then he raced back in the other direction. And then our internal Systems Administrator ran past. And then back again. Soon lots of people were running about the place looking rather agitated. The next thing I know, the Service Manager pops over to my desk, taps me on the shoulder and very calmly asked if I had just installed NT Server…
It turned out the manual I was following failed to mention that when you install the Windows NT Server you MUST set it to a Secondary Domain Controller, otherwise it decides it’s the Primary Domain Controller and takes over from the existing Primary Domain Controller. I had just managed to take down the entire network in our Head Office!
Luckily the company was very easy going so I never got in trouble. I think they were more impressed that I’d managed to install and set everything up from just reading the manual (as the manual failed to mention setting the Server to a Secondary Domain Controller I suspect I was the first person to manage it). Obviously, the manual was very quickly updated.
This post has been deleted by its author
Becoming a PDC was never the default in NT4.
First it would have been in a workgroup.
Then you could add it to an NT4 domain.
You could then make it a PDC or BDC and many times I saw people who had lost a PDC with no adequate backup think that they could just install from fresh and promote the new machines as a replacement and then it would work well (wrong - promote one of the BDC's).
I would suggest that there was something in the instructions that were either wrong or misinterpreted but no... imagine the chaos if every time you joined an NT4 server to a domain it automatically tried to be the PDC.
It is fairly obvious that the instructions were wrong .
Wrt defaults: when you get to the point of actually making it a DC, was the default to be primary or backup?
My random guess: the instructions were written as the company's first NT server was being set up, so they took you to setting up a DC, and as the first one had to be primary the author never even considered choosing BDC. Then those became "The Instructions" for setting up an NT server (well, they worked okay the last time).
My memory say quite the opposite! A Windows NT 4.0 Server has to be PDC or BDC right during installation, when you just fiddled with the network in text mode setup. You cannot change from "Member" to "PDC" or "BDC".
You describe the improvement which came with Windows 2000 and Active Directory.
This story reminded me of the olden days in the last millennium when a bunch of PCs could be networked as long as *nothing changed*. The SMB protocol was obviously written by Sparticus as every PC tried to say "I'm the master browser" (No, I'M the master browser) whenever it was turned on.
Being too averse to running a windows server, a Linux machine was introduced with a high enough OS_LEVEL to keep all the clients at bay, which worked fine most of the time, but still there were some weird and wonderful browsing issues, and incredibly slow network access. It transpired that the outgoing (but still online) Novel Netware server was providing another form of name resolution, but it depended on the sales department's PCs being booted up before anyone else - those machines had both IPX and MS stacks loaded, resulting in the strange behaviour. That was cured when all file sharing was migrated to Samba, and the Novel machine put out to grass.
Despite many crates of Single Malt Whisky (as it is correctly called) has filtered into my brain, I'm still triggered by browsing issues ...
The workaround at that time was to use a WINS Server, which always wins that master browser election. Today I still see that thing sometimes when taking over a new customer -> Instant nuke, stop and disable service, uninstall a few weeks later. And asking my coworkers why they haven't done that years ago. Funny side effects possible if WINS wins today!
You're quite correct, of course. IIRC Samba at that time could also act as a WINS server as part of its duties, so that was how that particular battle was won.
Like I say, it's been far too many years for me to remember the details as it's all NFS on my post-work home network, and there are no windows machines here either. Even the VMWare server with Win10 VMs has been powered down for years.
Beer on a Monday afternoon, because I can.
Whenever "that name is already in use" crops up I always add an 'X' on the end and try again. It at least means I have a record of what it was when I come to investigate a better resolution. And, if I need to add more than one 'X', it's reminder I really should start looking at resolving things properly.
What's now popped into my head is wondering if I could apply for Musk sponsorship?
"Just plug the cable in the machine, and the other end in your router" says the salesman.
Ok.... here we go..... and........ nuffink... well ok DCHP enabled on the router so we can get a basic system up and running.
And still nuffink.
hmmm head scratch.
Try again.. nuffink... hmmm maybe its the machines.
Delve into the mysteries of Heidenpain networking.......... DCHP option not enabled... please see your machine tool builder "Well you havent paid for that option to be enabled(see its not just car makers that can be dicks about having options built in that are not turned on unless you pony up some $$$$$)
Right... bit more brain storming and success 1st machine is on the network. now for the second and its a no-go.
Lets delve into this one............. more head scratching until we notice the IP is set by the machine and its using the same f'ing IP as the first one...... and they're all set to the same IP because thats how they test them at the factory....
Dont even ask what happened when I tried to mount the server shares as drives on the machines..... <shudders><starts with the flashbacks><screams a little>
You just described my first network and hardware
My first network action was managing to take down our whole token-ring by short-circuiting the central bus. Walked out of the network closet to hear a whole roomful of PS/2 machines going 'click, click' as they tried to grab a token that was never going to be there..
Now add in a buggy server direct from the factory right out of the box
We (at the same company) bought a load of shiny new 386 desktops from Trigem in Taiwan. Took 6 weeks to arrive by boat. The first 50 were unboxed and set up on the build room bench, plugged in to all the usual gubbins, then the room power was switched on.
Cue 50 melted power supplies that had been set to 120V at the factory, rather than switched to 240V as per our order. Copious magic smoke, including the stuff coming from our managers ears! The sales droid was summoned and told they *would* be replacing the units for free, *RIGHT NOW*. Even if they had to withhold stock from other customers.
The rest got their power supply switched over to 240V and built successfully. We never bought any more because the build quality was fairly crap, especially compared to the IBM PS/2s we were then using.
Oh - and Compaq.. At the next company I orked for, we bought some shiny Compaq rack-mount servers and UPSes (and installation service from Compaq).
The installers put the UPS units in the *top* of the racks, making them fairly unstable. The on-site director took one look at the setup and made the installers do things properly. I've never trusted manufacturer-supplied installers since.
Choosing the name of the company as a "temporary" name is the most colossally mind-numbingly stupid idea I can't even. If he worked for me I would have fired him, not for the result of taking down the entire corporate network but because after exhibiting such poor judgment I would know it was only a matter of time before he would do something equally stupid again in the future!
Oh what a shame that would be.
Every bit of useful wisdom/ experience to prevent future mistakes is gained through "stupid" blunders.
So you would just hire someone else who has not yet gotten that one time blunder over in done with and so get a repeat performance from another actor.
You have much to learn about developing people into effective useful tools for your company.
There's a difference between an honest mistake borne of ignorance and rank stupidity. One type learns from their mistakes, the other repeats them again and again.
That is a management lesson you learn the hard way - by giving a stupid person several chances and getting burnt over and over for your trouble. Only then do you see the difference and know which type of person is capable of learning from their mistakes and which will be doomed to forever repeat them.
So you acknowledge your managerial bias but do nothing to offset for it?
That's blatantly irresponsible thinking.
You keep going you're going to be completely alienated from reality.
That is the most common pitfall of those in managerial positions.
Socially normalized delusions that interfere with basic grasps of reality that eventually become a chronic learning disability.
I would suggest for your own mental health and retainment of intelligence to switch to working in production to offset your experiential balance of perception to reduce your delusions.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
