My previous employer decided that they wanted to bring in an MSP.
We were INCREDIBLY understaffed and this was their solution to "Well, actually, you're just too busy to spend time babysitting the boss when he can't open Teams, so we need more" and rather than hire, they decided to try to get an MSP. With an unstated and repeatedly-denied purpose of replacing in-house IT, I'd like to add. It was so blatantly obvious.
Anyway, they brought on an MSP, despite my objections, and that MSP proceeded to lecture me on my job, and claim to know better than me about everything, and tried to take over everything (it was clear that their brief was "take over the IT" while that was always denied in meetings, etc.).
So... I let them. Not without objections, and clarifications, and pre-warnings and I-told-you-so's, but I let them do it.
It turned out hilarious.
One of their "network team" (they had a "team" for everything, but those teams were always busy with whatever they did, which meant that the slightest query always went back to the MSP, lingered for days, then came back half-hearted with no time spent on it, a bill for doing that properly, and "no, our other guy you're paying can't do that, it has to be the network team") literally lectured me on NTP servers. One of the most trivial and relatively unimportant things ever... we had no need of time sync beyond basic domain operations. But they decided we were "wrong" with our "non-standard" deployment, because we were using a local NTP server and one of our remote NTP servers was one they hadn't heard of.
What they failed to take account of? The local NTP was a literal Bodet NTP radio clock sync device, the kind used in stock exchanges and railway operations. It was designed for site-wide sync and someone (*cough* me *cough*) had bought it as part of an all-site tannoy-like system because it synced time for free to all the units, used GPS and radio for timesync, and provided a local network NTP server that was certified to some ridiculous accuracy. We never needed it, but it was already there and cost us nothing... why not use it?
No, apparently, we had to use time.windows.com.
Then they argued about the remote service and demanded we replace it with NTP Pool servers. They gave me some huge bluff piece on it, and there was a LOT of time wasted on this, especially for something that we absolutely did not need. That's why they tried to lecture me on how NTP works and why the pool was better, and how to configure our NTP. In the middle of which they specified settings which were both insecure but also... that included a particular NTP pool server (in an incorrect way to address it, I'd like to point out). I let them argue with me some more. Then I told them. That's my server. It's literally mine. I operate it. I joined it to the pool. It's been there for over a decades. It's one of the more reliable in the pool. It handles more NTP traffic than the entire commercial network we were using for that employer. Every day. It's literally my personal server that I operate outside of work. You're telling me to use MY server. Then you're telling me how MY server is configured and how it works and that I should use it.
And the fact of the matter is... we already were, via the use of NTP pool. They just didn't understand how it worked.
I had similar run-ins with them on all kinds of issues. They replaced our perfectly-functional intra-site VPN with one that literally didn't work. I know why it didn't work. I told them. I had even pre-warned them, and dropped hints at every opportunity and told them explicitly half a dozen times. But their "network team" never understood that they had to route additional subnets over the VPN or those subnets wouldn't work on the remote site. They were trying to pretend they could interpret Wireshark traces. They were trying to pretend our networks were undocumented (I literally pointed them at the existing, working configuration in plain text). They were trying to pretend that we were doing something completely impossible (It's bloody working already!!!!!). Etc. Etc. But if you don't route those additional subnets over the VPN... then the VPN isn't going to bother to route that traffic.
Everything on the main subnet... fine. Everything on another VLAN/subnet... never transited the VPN. Access control stopped working. Telephones. Printers. Digital signage. Anything on another subnet didn't work on the remote site from the second they put in the totally-unnecssary VPN of theirs. The irony was... their VPN box was literally just a VPN box. To operate, it had to sit behind the routers at each site that... had been running the VPN between those sites. So we had had to turn off the VPN functionality on the router, install two expensive boxes behind them, configure port-forwarding etc. for VPN ports on both ends, and then have the boxes route the VPN traffic... badly and incompletely.
After six months, my employer got tired of the constant arguments and people complaining to me about stuff not working at the remote site (and I just filed tickets with the MSP... not my problem!) and told me to back it out. Ten minutes later (I had saved the config), the VPN worked and all traffic routed properly and we threw away the VPN boxes they'd made us buy. I mean... I had literally told them what was necessary and told them that we routed several subnets over there... not once did they ever put in any additional config to route those other subnets.
Similarly, they installed an new high-availability router device. Massive, expensive rack-mount thing. I asked how they intended to deploy it. They said it had to sit directly behind the main gateway. Okay. Well... we have two gateways, you know. Because we have two leased lines to the Internet for redundancy. And we use both in failover. They said it had to sit directly behind the main one. But what happens when we're in failover? We get no internet, that's what. Again, "the network team know better than peons like you and they've spent months designing this and you're just a guy we intend to put out of a job".
(shrug). Okay.
Because of power and other problems, that site would failover about once a week. There was a reason we had two leased line, two routers, at either end of the site, on independent power supplies. So within weeks, it failed over to the other device. I asked why it wasn't working was still in place. But obviously they couldn't contact their device. Everything else was still working (because we designed the network to work like that, and they were well aware of that) but the router was now entirely out of the loop, sitting on a dead gateway. Worthless.
There were MONTHS of that. Literally MONTHS of accusations flying around about how we must have turned it off deliberately (I honestly didn't need to sabotage the idiots, they were doing quite well by themselves!). But ultimately, they realised... this wasn't going to work. Not only had they spent months putting in a device in a terrible configuration, but even the "HA" portion of it literally never worked. Not in a single demo. Not once. Never. They even made us run 100Gbps fibre between the routers SPECIFICALLY for HA heartbeat. Fibre was fine. HA never worked.
They did something else similar. Bought an IDS/IPS. Attached it to one gateway. It never detected anything. Literally nothing. I kept complaining. And complaining. And complaining. And it kep getting escalated but they assured me it was all working. I got it in writing. MONTHS this went on.
It was at that point that I pointed out that the device they were supposed to install was still sitting in the rack, uncable and unpowered. They'd racked it. And that was it. Then they tried to blame me, but I had not only a trail of evidence, but I'd deliberately pointed it out to my boss who had - sensibly - not said a word and let them drop themselves in it.
When they did cable it and turn it on, it didn't work. Why? It was only monitoring one gateway. Fine while that was the active one. Useless when it wasn't. Our network was unprotected 50% of the time. They claimed that it was fine and they'd checked with the manufacturer and it was a supported config. Strange that. Because I had a written statement from the manufacturer (thanks BlueDog!) who had agreed to talk to me when they realised I was the customer, that they had SPECIFICALLY warned that MSP (the reseller) that they would need at least two such devices, and it would be worthless without. They sent me the email chain. Where the MSP dismissed that and just ignored it, repeatedly, against BlueDog's advice.
Whoops.