Dev's telnet tinkering lands him on out-of-hour conference call with CEO, CTO, MD

Welcome all, to the merry world of Who, Me?, our weekly trip down memory lane for techies who want to get something off their chest. This week, we hear from "Gavin", who was working at an ISP that had a few thousand point-to-multipoint radio links and had been asked to write a script to backup their configs. "They already had …

[W.S.Gosset]

Re: Well, there was this time...

"Interestingly", that's precisely the tactic used by the chap who burgled/bullshitted the Paris Accord/Convention with fictional data dressed up to present Calamity!/Armageddon! and on which the whole "result" depended. (The "paper" was rushed to publication in a manner and pace never seen before, AND every single delegate received a paper copy of same, and it was several times physically and dramatically brandished by delegates proclaiming how dramatic and urgent the situation was: "see!? PROOF!")

On being sprung using/having stolen a semi-AI-pseudo-model which the authors couldn't get to run twice and get the same numbers, and on being pressed to deliver up his data, he claimed his disk had corrupted.

On being told that he'd got so many genuine scientists so angry that they'd stumped up the money to have it forensically recovered, he "regretfully" informed them he'd smashed the disk with a hammer to protect the data from misuse.

[Alan Brown]

"remote freaking systems"

"And, if the extra traffic wasn't enough, the CPU would crash and then they had to dispatch the technicians "

One thing I learned a LONG time ago about remote routers of any kind is that if you don't have some kind of watchdog on them you're going to have to send someone out sooner or later.

Rigging one up before sending the things out is a LOT cheaper than one roadtrip to push the reset button (or power cycle) afterwards.

[My other car WAS an IAV Stryker]

Some former co-workers may remember when we got a reminder to update our timecards every day at 2:00 via NET SEND if you hadn't updated it before then (policy was to put in the morning's time when leaving for lunch).

But the BOFH(s) on the first floor didn't secure that option from being used by anyone.

After some small tests to see if it worked between individual users, I put together a text list of department usernames and wrote a QBasic script (being a non-dev) that called the command line to send custom messages to everyone in the department, one at a time down the list. I believe I tried it once and I failed, but others may recall the trials.

It wasn't long after, especially after the rollout of Win7 (it was XP originally), that SEND was no longer functional. Both playtime and daily all-hands reminders ceased.

I hope to be remembered for the good things, like the custom powertrain module testing kits I had built, or the graphical "dashboards" that are highly useful in the chassis dynamometer control room to see live telemetry. Now I'm designing wire harnesses instead of playing with powerpacks.

[Spanners]

@My other car WAS an IAV Stryker

NET SEND was a really useful thing. I am now missing it again. Is there anything that will do the job nowadays?

[I Am Spartacus]

Re: @My other car WAS an IAV Stryker

I think the new version is called SMS, or if you need a GUI, WhatsApp.

[Prst. V.Jeltz]

Re: net send

" Is there anything that will do the job nowadays?"

yeah write a script along the lines of msgbox("hi there") , copy it to destination pc and run it from there using psexec.

Unless you mean do the job of messaging every user on the network and getting yourself fired , in which case you need a bigger script.

[jake]

Re: net send

write and talk still work, and should be available on most *nixish platforms.

[Kevin Johnston]

Ahh...EEPROMs

As a newly badged engineer I was working in Systems Test on radar systems and I was given the task of programming and labelling the various PROMs for the systems. This was done with white ink for readability but after final test I would go round and re-mark them all. In an attempt to give this some degree of permanence I would then dab some varnish on to seal it.

By the time we got to my third system I was looking to speed up the process so I had re-marked them all and put them on some cardboard so I could use some spray varnish which was quicker to apply and dried faster too. A couple of hours later I discovered the third feature as the system would not run and I had to re-burn a full set as the varnish was bonded to the legs as a perfect insulator on the first set.

Not my finest hour

[Anonymous Coward]

Backing off

I think the real lesson here is: if something goes wrong when you're talking to a system, back off in some organised way (exponentially up to a limit then just keep trying once a minute or something) rather than hammering endlessly on the door.

I worked for a company which had a back-end database behind some vast farm of caching systems. If the database failed and had to be restarted then the caching machines just sat there and tried to authenticate as fast as they could. This just battered the database to death: some resource (connections?) ran out so it started dropping connections and then the whole system just became effectively catatonic as the caching systems effectively launched a DoS attack on the DB. The answer, a very bad one, was that if the database fell over the entire front end had to be stopped and then brought up gradually. I don't remember if we ever persuaded the developers to change their system to back off if it failed to authenticate.

[defiler]

Re: Backing off

One of our clients used to use a Mac email application named after a bird you'd send down a mine to check the air.

Whenever his password expired, it would try to authenticate over 100 times a second. And that's across the internet - not even locally. His account would be locked in an instant...

We told him to stop using it, once the devs didn't seem to bothered about fixing it.

[Anonymous Coward]

Re: Backing off

If the database failed and had to be restarted then the caching machines just sat there and tried to authenticate as fast as they could.

I did some training at a technical institute many years ago. Recovery from a network outage was a very painful process - the machines we students had were all disk-less, and hundreds of students trying to re-boot net-booting computers at the same time...... (Back in the 311 days, when it would've been rather difficult to stop someone messing with c:\windows on a given machine).

[Wexford]

Re: Backing off

I ran a Solaris farm with NIS+ for directory services in the 90s. A cron job that ran every minute would check a file for updates to apply to mail aliases, then do a "nisping" in a loop to commit until the new alias showed up in the table.

Sadly I once made a change that, when someone deleted their alias via the front end, the damned thing would delete it then do a nisping in a loop to commit until the new alias showed up in the table...which of course it never did.

Once this loop started, the NIS+ database would get corrupted around two days later. It took me a few weeks of daily restores of the entire directory, during which emails would bounce because the user principals were missing, before I diagnosed the problem. I'd noticed a regular ticking noise coming from the server. "What's causing that disk activity?" I wondered and found the looping process.

[Anonymous Coward]

Interrupts

Had two (which both ended with drive deletions..)

1. Whilst editing some assembly language for a program that needed debugging, I accidently missed off a h to denote hex in the interrupt. Set it running and noticed a drive light come on and the machine lock up. Never did recover either.

2. before GIT and CVS, we had a RCS. We were a large dev team and we would often symlink into the repository to get branches and work on code. A colleage had done this and was cleaning up and managed a rm-rf complete with following symlinks. The delete wiped his files and then traversed into the symlink and started on that too. We had backups but it was a lesson to not link into the repository directly...

[Anonymous Coward]

Re: Interrupts

The other week I accidentally deleted all of /var/log/apace2/, rather than the subdirectory of archived log files* I was aiming for. As long as no one needs to check an old log in the next couple of weeks I'm probably ok...

* 40 fecking GB of them. Logrotate? Never heard of it apparently.

[Anonymous South African Coward]

logrotate

Had a Smoothwall that just fell over due to a full HDD.

Narrowed it down to excessive logs - and from there to logrotate.conf whose commands was commented out.

Found it was one of the mods that I installed that just "disabled" logrotate - edited the conf file by hand and fixed it. Was quite interesting. Had to do a manual logrotate first which took quite a while :)

[Prosthetic Conscience]

Re: logrotate

You're lucky, if it was a Fortinet on some models logging would just kill the storage medium after a little while due to the excess IO demand..

[Anonymous South African Coward]

Re: Interrupts

Ahhh, good old assembly language. Very cryptic, but very powerful.

[JBowler]

rm -rf complet with following symlinks

For those of you out there who don't speak UNIX, that post is a Troll.

[Anonymous Coward]

Re: rm -rf complet with following symlinks

Sorry - it was mounted/hard linked. Not symlinked...

[Criggie]

Re: rm -rf complet with following symlinks

If someone's here without at least a passing understanding of unix, then they've come to the wrong part of town.

[NonyaDB]

"Mr. Anon, why does our internet randomly cut out on us?"

"Well, sir, we have a point-to-point wireless 'shot' from our tower here to another tower about 60km away and unfortunately the shot lies directly in the approach flight path of the airport over there so every time a Blackhawk or Apache comes in to land, they literally sit in the beam path for a few minutes and block the signal. Once they fly off then it takes a minute or two for the two modems to re-establish the link."

"Well, can't you move the towers?"

"Afraid not, sir. They're permanently installed."

"Well who the hell did that?"

"The United States Department of Defense, sir."

"Oh. OK."

[Anonymous Coward]

Technicians dispatched to sites

“the CPU would crash and then they had to dispatch the technicians – to four different cities in the UK from just one base. During the night. to dozens of sites.”

What kind of a device requires a site visit if it crashes, shouldn't it reboot after failing to trigger a heartbeat after a set period?

[Displacement Activity]

Yes, alpha particles

<nerd mode>

Cosmic rays cause soft errors in memory chips and general circuit failures. At sea level, 'cosmic rays' are primarily high-energy neutrons. Neutrons are uncharged, so don't themselves cause circuit upsets. However, when they're captured in a nuclei in a circuit element, they produce charged secondaries, including alpha particles, which do cause circuit upsets. See https://en.wikipedia.org/wiki/Soft_error#Cosmic_rays_creating_energetic_neutrons_and_protons, for instance.

</nerd mode>

[Anonymous Coward]

Russian comms hardware? Really?

[defiler]

In Soviet Russia, Internet browses you!

[vincent himpe]

Newly minted silicon

While testing a fresh-from-the fab prototype integrated circuit on bench the big boss walks in exclaiming

'The customer is excited , he wants samples'.

To which the guy testing them responds : And how many do they want ? one, two or all three that work ...

(out of like 500 ... the chemistry was off. new process )

[Anon Ymous 42]

In 1979, as a high school student I was asked to write the attendance program. Each home-room had a punch card for each student. Those absent at home-room would have their cards sent to the main office, these were then fed into the card reader and two lists generated. One for all the teachers so they knew who was absent, the other to the nurse who badgered parents for excuses. I wrote the program so that any card with my last name never end up as an entry on the nurses list. Skip homeroom and I could skip any class that day. I never got busted because I never abused it. After I went to collage my brother, 5 years younger figured this out quickly and got busted within the first 2 months he later arrived at the high school.

[Michael Wojcik]

does anything in IT really ever die?

Has anything you've ever done ended up outlasting your time at a firm?

I expect most here have something like that.

There's one commercial product I started working on in 1988 which is still in use at some customers sites. That was developed by a company that no longer exists, though my current employer now owns the technology, so it's debatable whether it meets the "outlasting your time" criterion.

In 1989-1990 I worked on XGKS at IBM, and that's still available for download from Sourceforge (and I was pleased to find my name still in the README). There's a decent chance it's still in use somewhere, though the last update to the source was in 2004. Still, that's long after I left IBM (in 1991).