Posts by Hazmoid

Square blames last week's outage on DNS screw-up

It is always DNS!

been there done that, got the blistered ears from users yelling that the Internet is broken.

Have you ever suspected your colleague doesn't hope this email finds you well?*

Re: Performance assessments

Lol, I see what you did there :-)

Playing a game of boardroom bingo, were we?

Nobody does DR tests to survive lightning striking twice

speaking of lightning strikes

I recently had to replace 2 network switches and use the backup link for a site that experienced a near miss. It appears that the strike caused a surge in the existing link (ethernet cable) that blew up the network switches at each end. Fortunately, when we ran the link cable, we ran 2 in case there were issues with one.

BOFH: Lies, damned lies, and standards

Re: Association of Servicepeople for Software and Hardware Over the Lifetime of Equipment.

actually it started out as a joke (much like the Australian Research and Space Exploration website https://spaceaustralia.com.au/)

Fed up with slammed servers, IT replaced iTunes backups with a cow of a file

I had a problem with storage on my servers being constantly full, and not because we had underspecced the storage. Turns out one of our users had stored his p0rn collection on his user drive, complete with category folders, luckily no kiddyP but still not what you expect in a professional environment. He was tapped on the shoulder and told that it needed to be removed COB next day or it would be deleted. We also chatted to a number of staff that were storing music and videos in their user areas. Our storage usage dropped by half.

A toast to being in the right place at the right time

desk provisioning

I worked for a stockbrokers, and because the cleaners never cleaned under the desks, we tried to avoid having the Pcs on the floor. this would then result in having powerboards on desks.

Whenever we specced new desks up, they had 2 RJ45 sockets and 4 power points for each desk. Then we would get the users that pushed their keyboard up to the PC case and were constantly hitting the power button ;)

BOFH: Good news, everyone – we're in the sausage business

So ? Is AI the new "cloud computing"?

Funny but I have been listening to a lot of customers asking what we think of AI and when are their jobs going to be replaced? For many of them the moment we can get an AI that understands what they are doing and can translate it into an invoice, they will be gone :)

Fortunately for those of us that keep the wheels turning and the fans whirring, this is not going to be any time soon. As i said to someone the other day, they will still need someone to physically change hard drives and fans on all those servers :).

Cunningly camouflaged cable routed around WAN-sized hole in project budget

many years ago on a multi million dollar project

The company I worked for managed the accommodation on a work barge that would be towed out to sea when there were cyclones threatening. The original internet was provided by a Satellite dish on the top of a hill, about 300 meters away from the barge tie up point. An ethernet link was run down the hill and connected across the gangplank , and was regularly damaged because of the 10m tides! So when I first started we suggested that we use a wireless link between the top of the hill and the barge. Attaching the receiver on the barge was easy, the receiver at the top of the hill was more of an issue as the site managers were very strict about H&S. If I did it the correct way it would have taken 3 weeks and copious paper work to get permission to climb the ladder on the outside of a container and attach the aerial with an ethernet cable and some cable ties. So one morning, one of the managers drove me to the top of the hill with the required equipment and looked the other way :) 15 minutes later our wifi link was up and it stayed that way until we decommissioned the barge 3 years later.

Dyson moans about state of UK science and tech, forgets to suck up his own mess

Re: With two-faced "friends" like Dyson, Britain doesn't need enemies

Especially now she is an Australian citizen :)

BOFH: Ah. Company-branded merch. So much better than a bonus

4GB- could barely fit a couple of photos that you do not want to keep on it, I guess it would be suitable for providing those incriminating photos of the boss in a compromising position with his secretary, to be accidentally picked up by the office gossip.

Defunct comms link connected to nothing at a fire station – for 15 years

Re: Opposite Problem

In these cases, I have found that the occasional "Dry Run" ( say once a month, about the end of the month when accounts are trying to generate their important reports for the C suite) can be invaluable in pointing out that these lines should be maintained regardless of the cost. Illustrates to Accounts that they are important and regardless of the cost, necessary, if only to keep the accounts department running in case of a disaster.

Techie wiped a server, nobody noticed, so a customer kept paying for six months

my fault here.

migrated an options trading system to new hardware and updated software, which took all night, ended sleeping for an hour in the boardroom before people started turning up for work. At the point when I finished, the historical data had not been moved to the new server.

Of course stupid ( me) forgot to copy the historical data across to the new server, and shut down the server. As it was options trading, the only thing the traders cared about was that their trades worked. backups only backed up the day's trades and not the historical data.

Then the excrement hit the rotating air circulation device. A high profile court case needed access to the historical records that had been wiped when the server was re-utilised. We tried getting a data recovery firm to look at the drive and see if they could recover any data but because it was on an array, no dice.

In the end the company took a hit because it could not produce the records.

Welcome to Muskville: Where the workers never leave

I expect it would be cheaper too. ;)

Take the morning off because Outlook has already

Here in +8 land

only thing coming up is the following for Azure

Summary: Starting around 20:19 UTC on 7 February 2023, a utility power surge in the Southeast Asia region tripped all of the chiller units for one datacenter offline. While working to restore the chiller units, temperatures in the datacenter increase so we are proactively powering down compute, storage and networking resources to avoid damage to hardware. All impacted infrastructure is in the same datacenter, within one of the region’s three Availability Zones (AZs). Downstream services that have been identified as impacted include Azure App Services, Azure Backup, Azure Cosmos DB, Azure Database for MySQL & flexible server, Azure Database for PostgreSQL & flexible server, Azure Log Analytics, Azure Red Hat OpenShift, Azure Search, Azure SQL Database, and Azure Virtual Machines (VMs).

I.e. due to dodgy power and the chiller units not being on filtered power, they dropped out and now the fan heaters in the the server room are driving the temperature through the roof. Even with the doors open and large industrial fans.

So this is affecting South East Asia,

No, you cannot safely run a network operations center from a corridor

optional kit drying rack

I used to ride daily to my work (started at 5am during ESDT ) and when it rained, my kit was hung on a coat hanger at the rear of the rack. Always made sure that it was not dripping first. Those Lenovo servers certainly generated enough heat to keep the room cosy even with an aircon set at 20 degrees (Celsius)

BOFH and the office security access upgrade

Re: Cost optimized Stairwell

Is that the stairwell with the timed lights that go out just after the door closes to save energy costs? With no natural lighting?

Server broke because it was invisibly designed to break

ties and their uselessness

Having worked everywhere from a Stock broker to a small 2 man outfit, I can safely say that ties in Australia are only on salesmen, politicians and lawyers. Most of the workplaces in Perth ( where summer temperatures regularly exceed 40 degrees celsius (100 F for the seppos)) have loosened their workplace dress standards to polo shirts, or open collar business shirts, simply because ties just end up a mess or dragging in something.

Square Kilometre Array Observatory construction commences

Proud West Aussie here

I was scoping out Seek and there were jobs going for data managers and big data engineers on this project a few years ago. The base station for the data management is at Curtin university, quite close to where I live. One of the Professors in charge of this project was a Joey leader at our Scout group so as you can imagine, we had some fun STEM activities :)

Job 1: Get the boss on the network. Job 2: Figure out why Job 1 broke the network for everyone else

Re: Sounds Like

As long as it wasn't the Donald's golf course :)

Man wins court case against employer that fired him for not liking boozy, forced 'fun' culture

Good luck getting most IT people involved

Generalisation here, but I have found in the past that most IT departments have at least 1 or 2 ASD people who the department totally depends on to keep their sh!t going, that are not interested in the social side of work and just want to do their job and get back to their cave. I worked for 14 years with an IT manager who refused to attend Christmas parties because he did not want to witness the depravity that went on ( working for a stockbroker during the "Wolf of Wall Street" years).

Hot, sweaty builders hosed a server – literally – leaving support with an all-night RAID repair job

Had a server room air con fail because it was running too cold and the condensor coil turned into a solid block of ice :) In a multi storey office block so building maintenance were none too happy that we were dumping our hot outlet into their building AC to resolve.

We had to run one of those big portable refrigerated units that had a water container for the condensate, that had to be emptied every 6 hours. So for a weekend I had to drive to the office every 6 hours and empty the container. Fortunately it was only a 20 minute drive and the traffic was good over the weekend, but it felt like I was doing the new father bit again. I did get paid a handy bit of overtime though :)

To make this computer work, users had to press a button. Why didn't it work? Guess

Re: Bad design

When one of my relatives was a dairy farmer, he had one of the old magneto phones between the house and the dairy to save having to walk over to check on things. Of course, this was in the days well before mobiles became ubiquitous.

Re: The human race could be safe after all

And this is why Mollyguards were invented :)

Re: Manual is optional,

This sounds like the Scout method of training, i.e. plan do review, where you explain what is expected, give an example, get the youth member to replicate it, keep them repeating it, then get them to train a younger member. Finally the review is done to ensure that there are no areas for improvement in the methodology or training.

NASA regains control of CAPSTONE lunar orbiter after a tumble

Well done that support engineer!

Given that you can't just jump in an Uber and pop around to sort this out, managing to fix this remotely is a brilliant bit of work :)

No, working in IT does not mean you can fix anything with a soldering iron

My wife constantly volunteers me to fix other people's IT issues. When I get home after a day of dealing with IT problems that I get paid to fix, the last thing I want to do is help someone sort out why their Iutlook is not connecting to their email. "Have you changed your password recently? Did you clear the old password from the credential manager?"

Bad UI killed the radio star

Re: Bah!

To be honest if you don't do safe isolation, it's likely you are going to need the toilet if you survive.

BOFH: It's Friday, it's time to RTFM

The worst part is that I know what most of these TLA mean :(

Terminal downgrade saves the day after a client/server heist

Re: Teletubbies

I've done that before, copied people's windows 7 profile and installed it in Windows 10 machines, so they feel comfortable with the new machine.

Mouse hiding in cable tray cheesed off its bemused user

being called out because keyboard is misbehaving, only to have to move a textbook off the keyboard :)

Google gets the green light to flood US Gmail inboxes with political spam

I want to know how they are going to tell if the gmail account is US based or not. I have had my gmail account since Google started, and I still get spam for someone in the US with the same surname, and matching first initial. Given that Google just got sued over geolocation services on Android, I want to know how they will confirm that the gmail accounts they are spamming are based in the US (I was repeatedly spammed by various Texan Republicans, who I just blocked during the last US election cycle. )

BOFH: Who us? Sysadmins? Spend time with other departments?

Re: Good start

I believe this happened a few years ago when there was a female sales manager who managed to get Simon's managers to sign up for whatever waffle of the day she was selling. I believe she outwitted Simon a few times, with the end result being that she was promoted out of dealing with anything within BOFH's purview , thus saving Simon from having to resort to the carpet roll and quicklime.

After 40 years in tech, I see every innovation contains its dark opposite

I think there have been enough warnings about VR

Given that we have seen "Tron", "The Matrix" and any number of variations that indicate that people living in a VR world miss out on what is happening in the real world.

Add the problem of motion sickness, and being tied to a computer with sufficient processing power, and we are still not at the point that technology can be said to be progressed.

Pentester says he broke into datacenter via hidden route running behind toilets

simple fix is to split the piss corridor with a wall.

Of course in this case it was a case of "out of sight, out of mind"

Getting that syncing feeling after an Exchange restore

Try Banyan email for a doozy

Having worked for a Government department that used Banyan Vines ( because anything that is good enough for the US department of Defense must be secure enough for our piddly little Australian govvy department) I sympathise with you all. We had to get new versions of the software flown in (back in the days when the Internet was a glint in telecommunications company's eyes, and it was quicker to transport CDs than to try and download them) to get our server hardware upgraded at a remote site. They used Banyan Mail (character based but pretty cool in its day) and had a windows gui that hooked into it (running on Windows 3.11 Sharkmail?)

So come stupid o'clock on Monday Morning and we are trying to install the server and restore the data, and finish about 730 am on Monday morning, only to have some scientist haul me over the coals because they were unable to access their latest batch of spam emails. At that stage I had been running on coffee and chocolate for ~ 20 hours and was in no mood to be polite. So I told her that the mail would be fixed when it was fixed and she would know about it when it finished. I then went to try and get some sleep at the hotel room I had not seen that weekend, only to have the owner start mowing the lawn outside !

This is how accidents on the road happen, trying to drive when you have been awake for more than 24 hours and need to drive for another 3 hours to get home. At some point I think I pulled the ute I was driving into a truck bay and slept in the back for about an hour. Fortunately the parental home was a good hour closer so I went there and slept for 3 hours before driving home.

Got to the office the next day and was immediately requested to report to the IT manager. She asked why she was having to deal with a call from some senior scientist about why her staff were so rude. I then went to town on her with the whole sorry story, and resolved to start looking for a new job that day.

Before I left they had gone the whole Exchange route, with Banyan running on Windows NT for directory services. Some contractors made out like bandits on that contract!

That time a techie accidentally improved an airline's productivity

Re: Everybody knows...

I once had to setup a stock trading system on a computer for a client (back in the days when windows XP was good enough for businesses) and walked in thinking, "this will be simple" only to find the client was Chinese and his computer was using chinese language. I had to guess where most of the settings were based on the pictures for the different apps, and my knowledge of the various control panel apps.

BOFH: Something's consuming 40% of UPS capacity – and it's coming from the beancounters' office

Beancounter central would not be happy on Monday

I think the phone system would be on bypass as well :)

Netflix to crack down on account sharing, offer ad-laden cheaper options

Re: Maybe

+1 for Cowboy Bebop, was just getting into a rhythm and they cut the show. I wanted to see what happens with Edward. :(

Foxtel is a key example of what happens when you introduce ads.

We had foxtel back when the only way to get it was to have them install a satellite dish on your roof and run coax to the box next to your TV. If you wanted more TVs to use it, you had to pay for additional boxes. When Foxtel started it was about the sport and not having to watch ads, but they slowly crept in, initially with program promos but eventually with beer and gambling. When Netflix started, we cancelled Foxtel ( 1. because Murdoch, 2. because it cost 10 times what Netflix did) and never looked back.

Netflix already does the program promos at the end of series and when you pause long enough in the selection screen. Adding ads in during programs is just going to be a pain. However I noticed the other night that there are already "Ad" breaks in some programs e.g. "ShadowHunters" where it goes to a black screen in between scene cuts. I suspect this was in the original programming and not removed by Netflix.

An early crack at network management with an unfortunate logfile

Re: Quality Issue Management System

There was a training company here called IFAP, Last time I went past their site I noticed that the name had changed.

Buying a USB adapter: Pennies. Knowing where to stick it: Priceless

Re: Seems ok

Someone has been reading too much BOFH ;)

If you fire someone, don't let them hang around a month to finish code

Re: Never risk it

AAAH a true BOFH in the making :)

Half of bosses out of touch with reality, study shows

working from the office.

Personally, I prefer to work from the office, as it is a 20 minute drive from home and I sit here by myself, as the boss works from home or his car.

Bonus is that I don't have to put up with the other family members calling me for IT support every 5 minutes.

Add to that the crappy FTTN NBN internet service at home and you can see why I decided to continue working from the office even though I could just as easily work from home.

Network operating system Dent 2.0 targets smaller firms

Can't wait for the Slartibartfast version :)

As a longtime DNA fan, I have to admit that I am happy HHGTTG should finally be used in this way.

However I think that even small businesses may be worried about something like their infrastructure relying on that small bowl of petunias.

Right-to-repair laws proposed in the US aim to make ownership great again

Car manufacturers are not immune to this.

Saw a rumour recently that Toyota is looking to make the remote lock/start option become a subscription only model, where you pay a monthly fee to access your car. I think this would have been slammed by this sort of law

https://www.theverge.com/2021/12/12/22831105/toyota-subscription-remote-start-key-fob

When forgetting to set a password for root is the least of your woes

Working for a national company in headquarters, with around 20 IT staff in an open plan office, the rule was if you walked away from your computer without locking it, then it was fair game. One of our staff made a point of setting the desktop to be pictures of kittens and leaving an email to management telling them that you quit.

Pop quiz: The network team didn't make your change. The server is in a locked room. What do you do?

Locked my car keys in the office one night as I was heading out. As the last on site I also had a gate key and the alarm code :( So I did what any prepared IT guy does, and opened the door using my Leatherman. (Latch was a sliding lock on an outward opening door).

I have also thought that it was crazy having high security on a comms room door when the wall next to the door consists of plasterboard and insulation.

Most of the electric locks I have seen let go when the power goes out or the fire alarm is triggered. Except at one Government Office I worked at (State Herbarium) in which the collection storage (multiple wings on multiple floors) had CO2 dump systems and the locks triggered on fire alarm . Spread thoughout the storage were breathing masks that would allow you enough O2 to get to the door and exit. locks had to be released from the fire panel for fire fighters to enter.

Re: Out of date building plans

In a previous job, I regularly got to ride cherry pickers and climb up high buildings to investigate and install wireless links. never been on a working at heights course but know how to put on and use a safety harness (thank you to the Scouts abseil team)

Microsoft patches Y2K-like bug that borked on-prem Exchange Server

Re: Year 2100 bug

And a fortune to be made certifying and selling new hardware and software that is guaranteed not to have this problem. Just look at what happened with Y2K.

A time when cabling was not so much 'structured' than 'survival of the fittest'

Re: They had it coming.

I worked for a Government department many years ago ( so long ago that the department has split and reformed twice since then) however they had a large data centre with raised floors from the days when they ran multiple VAX systems. The cost to rip out the data centre walls and resize when they went Banyan Vines and then Windows servers was not something they wanted to pay, so the room remained pretty much empty except for a single rack of servers in one corner. When the Vaxes were finally removed, the systems manager decided it was time to clean up the under floor. We pulled out 3 utility truck loads of cables that went to the recyclers.

The good thing was that the room had dedicated airconditioning, so I moved my desk in there, and the only people who could access the room were IT staff :)