It is always DNS!
been there done that, got the blistered ears from users yelling that the Internet is broken.
205 publicly visible posts • joined 11 Mar 2014
I recently had to replace 2 network switches and use the backup link for a site that experienced a near miss. It appears that the strike caused a surge in the existing link (ethernet cable) that blew up the network switches at each end. Fortunately, when we ran the link cable, we ran 2 in case there were issues with one.
I had a problem with storage on my servers being constantly full, and not because we had underspecced the storage. Turns out one of our users had stored his p0rn collection on his user drive, complete with category folders, luckily no kiddyP but still not what you expect in a professional environment. He was tapped on the shoulder and told that it needed to be removed COB next day or it would be deleted. We also chatted to a number of staff that were storing music and videos in their user areas. Our storage usage dropped by half.
I worked for a stockbrokers, and because the cleaners never cleaned under the desks, we tried to avoid having the Pcs on the floor. this would then result in having powerboards on desks.
Whenever we specced new desks up, they had 2 RJ45 sockets and 4 power points for each desk. Then we would get the users that pushed their keyboard up to the PC case and were constantly hitting the power button ;)
Funny but I have been listening to a lot of customers asking what we think of AI and when are their jobs going to be replaced? For many of them the moment we can get an AI that understands what they are doing and can translate it into an invoice, they will be gone :)
Fortunately for those of us that keep the wheels turning and the fans whirring, this is not going to be any time soon. As i said to someone the other day, they will still need someone to physically change hard drives and fans on all those servers :).
The company I worked for managed the accommodation on a work barge that would be towed out to sea when there were cyclones threatening. The original internet was provided by a Satellite dish on the top of a hill, about 300 meters away from the barge tie up point. An ethernet link was run down the hill and connected across the gangplank , and was regularly damaged because of the 10m tides! So when I first started we suggested that we use a wireless link between the top of the hill and the barge. Attaching the receiver on the barge was easy, the receiver at the top of the hill was more of an issue as the site managers were very strict about H&S. If I did it the correct way it would have taken 3 weeks and copious paper work to get permission to climb the ladder on the outside of a container and attach the aerial with an ethernet cable and some cable ties. So one morning, one of the managers drove me to the top of the hill with the required equipment and looked the other way :) 15 minutes later our wifi link was up and it stayed that way until we decommissioned the barge 3 years later.
In these cases, I have found that the occasional "Dry Run" ( say once a month, about the end of the month when accounts are trying to generate their important reports for the C suite) can be invaluable in pointing out that these lines should be maintained regardless of the cost. Illustrates to Accounts that they are important and regardless of the cost, necessary, if only to keep the accounts department running in case of a disaster.
migrated an options trading system to new hardware and updated software, which took all night, ended sleeping for an hour in the boardroom before people started turning up for work. At the point when I finished, the historical data had not been moved to the new server.
Of course stupid ( me) forgot to copy the historical data across to the new server, and shut down the server. As it was options trading, the only thing the traders cared about was that their trades worked. backups only backed up the day's trades and not the historical data.
Then the excrement hit the rotating air circulation device. A high profile court case needed access to the historical records that had been wiped when the server was re-utilised. We tried getting a data recovery firm to look at the drive and see if they could recover any data but because it was on an array, no dice.
In the end the company took a hit because it could not produce the records.
only thing coming up is the following for Azure
Summary: Starting around 20:19 UTC on 7 February 2023, a utility power surge in the Southeast Asia region tripped all of the chiller units for one datacenter offline. While working to restore the chiller units, temperatures in the datacenter increase so we are proactively powering down compute, storage and networking resources to avoid damage to hardware. All impacted infrastructure is in the same datacenter, within one of the region’s three Availability Zones (AZs). Downstream services that have been identified as impacted include Azure App Services, Azure Backup, Azure Cosmos DB, Azure Database for MySQL & flexible server, Azure Database for PostgreSQL & flexible server, Azure Log Analytics, Azure Red Hat OpenShift, Azure Search, Azure SQL Database, and Azure Virtual Machines (VMs).
I.e. due to dodgy power and the chiller units not being on filtered power, they dropped out and now the fan heaters in the the server room are driving the temperature through the roof. Even with the doors open and large industrial fans.
So this is affecting South East Asia,
I used to ride daily to my work (started at 5am during ESDT ) and when it rained, my kit was hung on a coat hanger at the rear of the rack. Always made sure that it was not dripping first. Those Lenovo servers certainly generated enough heat to keep the room cosy even with an aircon set at 20 degrees (Celsius)
Having worked everywhere from a Stock broker to a small 2 man outfit, I can safely say that ties in Australia are only on salesmen, politicians and lawyers. Most of the workplaces in Perth ( where summer temperatures regularly exceed 40 degrees celsius (100 F for the seppos)) have loosened their workplace dress standards to polo shirts, or open collar business shirts, simply because ties just end up a mess or dragging in something.
I was scoping out Seek and there were jobs going for data managers and big data engineers on this project a few years ago. The base station for the data management is at Curtin university, quite close to where I live. One of the Professors in charge of this project was a Joey leader at our Scout group so as you can imagine, we had some fun STEM activities :)
Generalisation here, but I have found in the past that most IT departments have at least 1 or 2 ASD people who the department totally depends on to keep their sh!t going, that are not interested in the social side of work and just want to do their job and get back to their cave. I worked for 14 years with an IT manager who refused to attend Christmas parties because he did not want to witness the depravity that went on ( working for a stockbroker during the "Wolf of Wall Street" years).
Had a server room air con fail because it was running too cold and the condensor coil turned into a solid block of ice :) In a multi storey office block so building maintenance were none too happy that we were dumping our hot outlet into their building AC to resolve.
We had to run one of those big portable refrigerated units that had a water container for the condensate, that had to be emptied every 6 hours. So for a weekend I had to drive to the office every 6 hours and empty the container. Fortunately it was only a 20 minute drive and the traffic was good over the weekend, but it felt like I was doing the new father bit again. I did get paid a handy bit of overtime though :)
This sounds like the Scout method of training, i.e. plan do review, where you explain what is expected, give an example, get the youth member to replicate it, keep them repeating it, then get them to train a younger member. Finally the review is done to ensure that there are no areas for improvement in the methodology or training.
My wife constantly volunteers me to fix other people's IT issues. When I get home after a day of dealing with IT problems that I get paid to fix, the last thing I want to do is help someone sort out why their Iutlook is not connecting to their email. "Have you changed your password recently? Did you clear the old password from the credential manager?"
I want to know how they are going to tell if the gmail account is US based or not. I have had my gmail account since Google started, and I still get spam for someone in the US with the same surname, and matching first initial. Given that Google just got sued over geolocation services on Android, I want to know how they will confirm that the gmail accounts they are spamming are based in the US (I was repeatedly spammed by various Texan Republicans, who I just blocked during the last US election cycle. )
I believe this happened a few years ago when there was a female sales manager who managed to get Simon's managers to sign up for whatever waffle of the day she was selling. I believe she outwitted Simon a few times, with the end result being that she was promoted out of dealing with anything within BOFH's purview , thus saving Simon from having to resort to the carpet roll and quicklime.
Given that we have seen "Tron", "The Matrix" and any number of variations that indicate that people living in a VR world miss out on what is happening in the real world.
Add the problem of motion sickness, and being tied to a computer with sufficient processing power, and we are still not at the point that technology can be said to be progressed.
Having worked for a Government department that used Banyan Vines ( because anything that is good enough for the US department of Defense must be secure enough for our piddly little Australian govvy department) I sympathise with you all. We had to get new versions of the software flown in (back in the days when the Internet was a glint in telecommunications company's eyes, and it was quicker to transport CDs than to try and download them) to get our server hardware upgraded at a remote site. They used Banyan Mail (character based but pretty cool in its day) and had a windows gui that hooked into it (running on Windows 3.11 Sharkmail?)
So come stupid o'clock on Monday Morning and we are trying to install the server and restore the data, and finish about 730 am on Monday morning, only to have some scientist haul me over the coals because they were unable to access their latest batch of spam emails. At that stage I had been running on coffee and chocolate for ~ 20 hours and was in no mood to be polite. So I told her that the mail would be fixed when it was fixed and she would know about it when it finished. I then went to try and get some sleep at the hotel room I had not seen that weekend, only to have the owner start mowing the lawn outside !
This is how accidents on the road happen, trying to drive when you have been awake for more than 24 hours and need to drive for another 3 hours to get home. At some point I think I pulled the ute I was driving into a truck bay and slept in the back for about an hour. Fortunately the parental home was a good hour closer so I went there and slept for 3 hours before driving home.
Got to the office the next day and was immediately requested to report to the IT manager. She asked why she was having to deal with a call from some senior scientist about why her staff were so rude. I then went to town on her with the whole sorry story, and resolved to start looking for a new job that day.
Before I left they had gone the whole Exchange route, with Banyan running on Windows NT for directory services. Some contractors made out like bandits on that contract!
I once had to setup a stock trading system on a computer for a client (back in the days when windows XP was good enough for businesses) and walked in thinking, "this will be simple" only to find the client was Chinese and his computer was using chinese language. I had to guess where most of the settings were based on the pictures for the different apps, and my knowledge of the various control panel apps.
We had foxtel back when the only way to get it was to have them install a satellite dish on your roof and run coax to the box next to your TV. If you wanted more TVs to use it, you had to pay for additional boxes. When Foxtel started it was about the sport and not having to watch ads, but they slowly crept in, initially with program promos but eventually with beer and gambling. When Netflix started, we cancelled Foxtel ( 1. because Murdoch, 2. because it cost 10 times what Netflix did) and never looked back.
Netflix already does the program promos at the end of series and when you pause long enough in the selection screen. Adding ads in during programs is just going to be a pain. However I noticed the other night that there are already "Ad" breaks in some programs e.g. "ShadowHunters" where it goes to a black screen in between scene cuts. I suspect this was in the original programming and not removed by Netflix.
Personally, I prefer to work from the office, as it is a 20 minute drive from home and I sit here by myself, as the boss works from home or his car.
Bonus is that I don't have to put up with the other family members calling me for IT support every 5 minutes.
Add to that the crappy FTTN NBN internet service at home and you can see why I decided to continue working from the office even though I could just as easily work from home.
Saw a rumour recently that Toyota is looking to make the remote lock/start option become a subscription only model, where you pay a monthly fee to access your car. I think this would have been slammed by this sort of law
https://www.theverge.com/2021/12/12/22831105/toyota-subscription-remote-start-key-fob
Working for a national company in headquarters, with around 20 IT staff in an open plan office, the rule was if you walked away from your computer without locking it, then it was fair game. One of our staff made a point of setting the desktop to be pictures of kittens and leaving an email to management telling them that you quit.
Locked my car keys in the office one night as I was heading out. As the last on site I also had a gate key and the alarm code :( So I did what any prepared IT guy does, and opened the door using my Leatherman. (Latch was a sliding lock on an outward opening door).
I have also thought that it was crazy having high security on a comms room door when the wall next to the door consists of plasterboard and insulation.
Most of the electric locks I have seen let go when the power goes out or the fire alarm is triggered. Except at one Government Office I worked at (State Herbarium) in which the collection storage (multiple wings on multiple floors) had CO2 dump systems and the locks triggered on fire alarm . Spread thoughout the storage were breathing masks that would allow you enough O2 to get to the door and exit. locks had to be released from the fire panel for fire fighters to enter.
I worked for a Government department many years ago ( so long ago that the department has split and reformed twice since then) however they had a large data centre with raised floors from the days when they ran multiple VAX systems. The cost to rip out the data centre walls and resize when they went Banyan Vines and then Windows servers was not something they wanted to pay, so the room remained pretty much empty except for a single rack of servers in one corner. When the Vaxes were finally removed, the systems manager decided it was time to clean up the under floor. We pulled out 3 utility truck loads of cables that went to the recyclers.
The good thing was that the room had dedicated airconditioning, so I moved my desk in there, and the only people who could access the room were IT staff :)