Yes, I did just crash that critical app. And you should thank me for having done so Welcome once again, dear reader, to the well-trodden path that is Who, Me? – The Register's weekly column dedicated to the tales and travails of readers who strayed, but found their way back. This week's traveller is someone we'll Regomize as "Harper" who many years ago was developing a Unix-based messaging system for a large …
It's how I knew that the gateway to the super secure, must be absolutely kept up to date platform that I built and then was 'managed' by someone after I took all the risks to build it (during which everyone was keeping out of the political blast radius in case it went wrong) was not updated for some & years after I built it ...
Reminds me of when I was a nipper working on a helldesk which didn't have a dedicated email address (this was back in the 90s). There was just me and one other guy working on it at the time and users would email us directly for help.
I thought it would be a jolly jape to set up a forward so that every email that came to me got forwarded to him for him to deal with. Once he realised what I'd done, he thought it would be even jollier if he reciprocated and sent them right back to me. It only took a few minutes for the mail server to fall over after we created our loop. In our defence we realised straight away that it was our fault, removed the forwards and once the dust had settled we owned up. Our long suffering boss took it in her stride and just told us not to do it again. Lesson learned.
Sounds like one of my former bosses. He gave me admin access to our production system (because I needed it), along with a few pointers:
- try not to be stupid
- if in doubt, check with somebody more experienced
- trust your own experience - we hired you because you do know stuff, no need to get on my bloody nerves for everything
- if you mess up, own up
- we have backups that we can actually recover from (thankfully we never needed to).
- I will cover my team's actions
And that's exactly what I tell my young colleagues.
I got admin access to the reporting system of a multi-national company but no one told me it was a live production system. Thing is this wasn't just a reporting for finance system it did everything. I also got very few details about the tables and data not from lack of trying. This wasn't my first rodeo without information and with a little bit a trial and error I get what I want out of them eventually. Off I went running my queries. Now this system was slow anyway and happened to have regular slow downs (I think I knew why as I wasn't the only one running reports). I'm on the phone to someone in finance trying to get some information and test the data I was running when all of a sudden I can hear in the back ground that the system stopped responding in the whole office. Whoops. I stopped my query and it popped back up. Lesson learnt and I was extra careful to ensure it didn't happen again and it didn't. I still for the life of me even today do not understand how that came to pass. I like your pointers but sadly sometimes you end up in these situations even without you knowing you're in them. I was going to warn them but I was a contractor that had only been there a couple of days also a bit younger than I am now I thought I would leave it. I doubt it would have made any difference because a few months later after automating a report used for key global business decisions I spotted a quite serious balancing error which meant every single monthly report they had ever done was wrong. I offered to fix it but was told "If it's always been like that leave it as it is" which the person I am wound me up no end.
This is why we have the EXPLAIN option in SQL queries, so the query optimizer can show us some stats on how many records it has had to scan, how it uses indexes etc. So you can see if it has to look at 2 million records to match a field value, adding an index on that field might be a good idea.
This reminds me of wondering aloud what would happen on my father's old telephone switchboard if somebody set multiple phones to redirect in a loop and called one of them from another phone.
Boringly the answer was that if a phone diverted to a phone that the call had already diverted from then it rang instead of diverting.
I had similar years ago (back when external email was nearly all SMTP) with Out of Office emails. Neither the original sender nor the recipient had checked the "Only send once" box.
As we were using an ISDN link from a black-box email concentrator to the provider we only had limited access to administer the system and the link was very quickly swamped. I did manage to track it down, shut down the user's PC and then clear the message queues.
I recently told this story about a former colleague who managed to create a mail flood between his inbox and mine, but here it goes again.
Whenever I or the other senior tech on my team went on vacation we'd set up a regular OoO response and, for a select number of addresses, a supplemental forwarding rule to the other one of us who was there, just to make sure requests from higher ups were properly followed up. One day I went on vacation and set the rules and, during that period, my colleague had to take a day off so did the usual. Unfortunately the forwarding rules weren't set to just forward mail from those contacts, but also when they were CC'd... so when the first email form someone on the list was sent to his address (or mine, really don't remember who first triggered the glitch), mails started being forwarded from one account to the other until it froze the Exchange server). He got a bit of an earful the next day, but nothing of consequence, and learned not to do it again (which is always good :) )
We had similar years ago with our helpdesk system. It was setup to auto-reply to any new emails (eg not replies to existing tickets) with a confirmation email, letting the user know their ticket reference etc.
All emails to us tended to go via the helpdesk, not to us individually, and that obviously included the 3rd parties for some customers.
On one occasionally a 3rd party IT company emailed our helpdesk with a request, so the helpdesk as normal replied... except the 3rd party had emailed us from THEIR helpdesk, and it didn't recognise our reply as such (think theirs identified replies based on Subject line, rather than the References header), so generated a new ticket at their end, and sent a new automatic email (not a REPLY to our email) to us. And so it continued until one of us could kill the stream of emails. Only saving grace was that our end had a delay of a couple of minutes between an email being generated and it being sent (a godsent when you accidentally hit send before you're actually ready), which helped reduce the damage and let us delete an email that had been generated, but not yet sent.
No need. Anti spam measures will do that for you.
We had a SOC connect into our platform - they pick up logs and analyse them for anomalies, then alert us if they find something dodgy.
During testing I got the emails, but the copied in helpdesk did not generate the associated tickets, so nobody but me saw the alerts.
Turns out Atlassian's Jira decided they were spam and thus neatly moved them out of the way before tickets could be generated..
One of our test architects spoke to the the product manager about ways of testing.
1) "If we are careful, and the wind is in the right direction, we can get our tests to pass"
2) Report to management "no matter what we did - we could not break it"
"Your product is 1). We need to get to 2)."
He was one of a group of people asked to write about computer testing for a prestigious magazine. They expected lots of complex graphs and so on.
Their article started.
Imaging we make cars. Your job is to test it.
1) Drive it at speed across a ploughed field. Whoops you've got a puncture - but no spare wheel - or jack ( 2 defects) and you've managed to scratch the paintwork - can they body shop repair the paintwork - no? another defect.
It was a great article, but it got turned down because it was "too obvious", and not technical enough
Part of the code I was working on had a basic keyboard interface which needed testing for bad data in the input fields. I spent some time thinking up random input to test with, and found a few validation errors that needed fixing. My colleague's approach was to dump his head on the middle of the keyboard and then roll it from side to side. He found a lot more errors.
I once had to come up with tests for a file transfer system which had to move files around in such a way that after a file was transferred it was removed from the source system. i.e. The intact file should only exist in one place.
Test 1) Start the transfer of a large file. Part way through, disconnect the network cable at one end . Check that after the failure is detected the file is still on the source system and the partial file has been deleted from the destination.
Test 2). Start the transfer of a large file. Part way through, disconnect the POWER cable on one side, then the network cable. After repowering, check that the file is still on the source system and the partial file has been deleted from the destination. Repeat the test powering off the other side.
The project manager would not authorise the second test. "Something might break."
A long time ago, I, along with a couple of friends, were learning NT system admin. A colleague had learned about setting up log in/out scripts on NT. Then, one day, I got a message sent to my PC. I didn't think too much about it, but a couple of days later, I started getting messages repeatedly. After getting the same message about 30 times in two hours, I asked my friend about it. He denied all knowledge. But I'd seen the machine the message was coming from in his office..
I told him this, but he still denied it. The messages kept coming and were affecting my ability to work, so I reported it to their line manager (who was also indirectly, mine). They still denied the message but it stopped happening.
I asked about it later, and they claimed they were messing around on the machine, and had written a script that when run on log out, amongst other things, sent a message to my machine. It was meant to be a debugging thing and one of my friends was actually developing the script on my machine, and testing it remotely. Apparently, he forgot he'd set the script to message my machine.
I'm sure I've told this one before.
We had a very annoying director who put read receipts on all e-mail messages. There was one message that I put, "unread" (aren't reading preview panes great?) into the Deleted Items which, of course, was set to delete on exit. Which, in turn, triggered an "unread" message to said Director. Next time I started Outlook, I'd go into my Deleted Items, select Recover Deleted Items and get that particular message back. Which was deleted the next time I logged out of Outlook. And so on and so on.
It lasted a good few months until she brought it to my attention when I feigned ignorance and said I'd investigate. Eventually (!) I reported to her that I'd "found a glitch with that particular e-mail" and had, I think, corrected it.
Pointless and petty and immature? Absolutely. But she deserved it (her office nickname was "The Bitch").
"Be as dasdardly as possible" This is the heart of QA. The plaintive cry "But you're not supposed to do that!" is its touchstone - we don't do it for the thanks...
For my sins, I graduated to that role when fellow coders eventually, grudgingly, would shove their 'finished' product to me first, because "That c*** can break anything"
It started out more in annoyance that my colleagues were churning out code for every one piece of mine, and I'd be reprimanded for being slow, when I thought I was being thorough - I came to the conclusion one of us was shit at coding. Could it be me? Surely not, so I started breaking things deliberately, in what was possibly anger.
I started trading code 'You check mine I'll check yours'. I'd break everything I was given, often in seconds ("The lazy b******s I bet they didn't...") They seldom broke mine - not because it didn't have errors, it often did and I knew it did - they just weren't being dasdardly enough.
It turns out having some kind of antisocial, destructive personality is the ideal characteristic for a QA tester - one must take inordinate and smug delight in breaking things, and revel in the tears of your fellows.
Much like a prior incarnation in my youth selling construction equipment - Sales reps would insist on presenting products with "Unbreakable, Indestructible, or 'Toughest' labels - my response was "That just gives the boys on site a challenge on their lunch break, and I'll have a grinning line at the counter the next day demanding a replacement"
"Hate your job, hate your life, revel in schadenfreude? Consider a career in QA"
*Sadly, I am now a grumpy old dick about almost everything "What asshole designed this, and what bigger asshole passed it?" I have pills for it - they don't work.
Quote
"It turns out having some kind of antisocial, destructive personality is the ideal characteristic for a QA tester - one must take inordinate and smug delight in breaking things, and revel in the tears of your fellows.
"Hate your job, hate your life, revel in schadenfreude? Consider a career in QA"
*Sadly, I am now a grumpy old dick about almost everything "What asshole designed this, and what bigger asshole passed it?" I have pills for it - they don't work."
You are our QA inspector and I claim my £5
<<wanders off muttering darkly about plating allowances and paperwork stamped "Fail"
I must disagree. Being Dasdardly is not enough. You ever hear this one?
QA walks into a bar. Orders a beer. Orders two beers. Orders 0 beers. Orders -1 beers. Orders 1/5 beers. Orders 2^0.5 beers. Orders pi beers. Orders i beers.
The customer walks in, orders a scotch. The bar burns down.
Proper QA isn't merely about being evil (or insane). You have to be completely thorough.
That's also how you get into a PhD program in mathematics.
My calculus students hated me as a TA.
If I knew then what I knew now, I would have been much tougher.
Sales reps would insist on presenting products with "Unbreakable, Indestructible, or 'Toughest' labels
Alternatively, put them in front of children.
When Magna opened in Rotherham, one of the exhibits was a set of JCBs for visitors to use. These were the genuine article, well, at least the arm and bucket and control system. (the link seems to show they are still there, but the controls look somewhat different)
The company responsible for said control system - which was supposed to be able to set limits on where the thing could swing, how fast etc. - were up every couple of weeks in the early days as one or other of the JCBs destroyed another piece of setwork, or suddenly forgot how to raise or something. More than once we heard the engineer bemoan the fact that these things were in site use around the planet but that the majority of their problems seemed to originate in the basement of an ex-steelworks in Rotherham.
"Kids", we told them.
M.
I do like to test to destruction of applications.
Create a test plan with a bunch of scenarios including things like loss of server, loss of database server, loss of database, loss of table, application falls over, bad data, stress testing,
Developers are fond of saying "but that will never happen" to which the response is "what if it does?" "But it won't", "well lets see....."
One app had to have about 14 months of work done because there were so many issues that it wasn't ready to see the light of day and certainly not customer facing! could write a 10,000 essay on that!
Also, got kicked off of a college prime server after the "email wars" between 2 sets of colleges IT students back in the 80s! Fun times!
My first employer used VAX mail. Apparently head of IT had been in a relationship with his PA. The poor lass tried to send him a pleading email after they broke up but put the whole text of the email into the "To:" field. The text of course contained the word "All" and the email system did what it was told...
If you really want to test anything with a interface for humans, take it to a primary school. It's the teachers who will (in all innocence) break it in the most bizarre and unbelievable ways. I think it's the constant exposure to vertically challenged proto-people that does it to them.
Haar!
@Will Godfrey: “If you really want to test anything with a interface for humans, take it to a primary school. It's the teachers who will (in all innocence) break it in the most bizarre and unbelievable ways. I think it's the constant exposure to vertically challenged proto-people that does it to them.”
Just today I was having to teach a retired social worker how percentages work. The fundamental basics of "if you start with 100 and you add 2%, what do you get?" (gets piece of paper, does complicated 100 + 2*100/100 calculation.... reaches for calculator...) I was trying to explain why 12 months of 5% interest is 1.05^12, and she was complaining about the "1.05", and I was trying to start from first principles of "add 2% is identical to times 1.02" by example. 100. add 2. 102. How would you do that by multiplying? etc.
"This is fundamental basic school maths"
'Oh, we didn't do capital stuff like interest rates at school'
"Ok, this is basic school rabbit breeding problem. You're a non-capitalist communal shared-ownership co-operative subsitance farmer collective, and start with 100 rabbits, each month 5% of the rabbits breed another rabbit. This is PRIMARY SCHOOL STUFF!"
One quiet afternoon not long after we installed our fancy new 3Com NBX 100 IP phone system, I accidentally found out a few trivia items.
* the NBX 100 had Port 25 open to the world.
* sending a small amount of traffic to the port could crash the system.
* it stayed crashed until power cycled.
Just like todayxs story, I spun it to the boss as a good thing I was able to identify the vulnerability, and updated our firewall rules (IIRC, we alrwady blocked incoming port 25 from the Internet, but lateral moves were suddenly more interesting).
Reminds me of the joke of a young man working in a grocery establishment when a shopper walks in and asks the young man about buying a halve of a watermelon. The young man tells the shopper to wait and he will ask the manger about the price. Walking to the manager, the young man does not realize that the shopper is right behind him and says to the manager that some fool wants to buy a halve of a watermelon. He then realizes that the shopper is standing right behind him and quickly follows up with “and this fine gentleman wants to buy the other halve”.
I may have told this story before. But many moons ago I was working for a largish international company (offices in NZ, Australia, the UK and Canada). An engineering intern had been allowed near the internet and read about the latest computer virus spreading around the internet.
Armed with this dangerous knowledge the intern decided he needed to warn everyone in the company. Instead of using some of the distribution lists (the memory is lacking as to whether or not he had permissions), he added every single person in the company to the To field.
This made the email about 2MB in size with all names added. Which meant the mail servers got hit with about 2GB of emails. Which they did cope with.....until a number of helpful folk hit Reply All. After which the mail servers expelled excrement and fell over taking most of the network down as they tried to handle the load.
Way back before the turn of the century I was working for a really big computer company that sold business machines internationally...
Anyway, one an email came in from a vice president in the large systems storage division warning us about a new computer virus.
This guy was high enough up that had access to the corporate email list and this warning went out to everyone who worked for the company WORLDWIDE.
And he got taken in by a hoax, the virus wasn't real.
Shortly after that a VERY sternly worded warning came out from the corporate network security people NO ONE was to send out virus warnings except for them.
I expect that VP didn't get a very good review that year!
Raphael's intern was a newbie and had that as an excuse, that VP was anything but.
In late 1994 I was working as a developer for a major financial asset management operation. I was at that time in a team of two working on a particular application which had come back from user test with a number of issued to be fixed before going live the following Monday, and of course everything took longer than expected so we went in on the Saturday morning to tidy the loose ends and push it to production in the Isle of Man. As my colleague had coded the overall framework he took on the task of building the InstallShield package, and by lunchtime he emailed the installer to the sysadmin in the IoM (all 200MB and more of it) and we innocently headed home.
Thirty years ago the global email infrastructure for such an institution with a dozen or so offices around the world was cc:Mail, linked by gateways using 14.4kbps modems. Experienced sysadmins from that period can see where this is going already, can't they? Over the Sunday night almost every local admin was woken by their pager informing them that there was a global email failure as there had been no activity for twelve hours.
On Monday everyone logged on to a message from the head of IT Operations bearing the immortal words: "Email is not a file transfer mechanism"
This post has been deleted by its author
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
