New boss took charge of project code and sent two billion unwanted emails

Welcome to another instalment of On Call, The Register's weekly wander through your tales of tech support. This week, meet a reader we'll Regomize as "Nick" who in the late 2000s found himself working as a contractor at a London investment bank. "I had technical oversight of a project related to overnight valuations of the …

[hoofie2002]

I used to work for a legal publishers where we sent out approx 50,00 emails every night to paying subscribers in the early 2000s. To be fair customers paid a huge amount for service subscriptions and since everyone was a high-end lawyer, many were somewhat touchy, argumentative and convinced of their superiority over near mortals.

Every. Single. Morning.: The relevant Manager who was very blue blood and posh but not the sharpest tool in the box complained that some subscribers had not received their email.

Every. Single. Morning.: Me searching through the transmission logs to send him the entry of the outbound mail that had been sent to a specific address.

Every. Single. Morning.: Me explaining to him that email was NOT guaranteed to arrive in someone's inbox due to spam filters etc. etc. etc.

Every. Single. Morning: Repeat the same F***ing process as the day before.

He really was a bellend.

[Richard 12]

At least search is trivially automated.

[BartyFartsLast]

I've had similar, being hauled over the coals by an office manager who was convinced email delivery was instantaneous and that somebody not immediately responding to their latest bullshit missive meant the email server was "fucking down again" when, in fact, most people worked sensibly and replied to stuff when they had time (or were ignoring it because they were as completely sick of the wanker as I was)

[GlenP]

I had the same from an HR Manager, "There's something wrong with the email, the urgent message I sent to ... in the States three minutes ago hasn't been received!"

I checked the server traces, "It's been sent from this end, I can't control what happens after that!"

She assumed email was like Fax, it connected your computer to the one at the other end and then sent the message, I had to explain it was like the post (the clue is in the name) and there could be delays at any stage,

[J.G.Harston]

Most people think email is not like fax, but more like a direct connection to the recipient's brain and takes control of their cognative functions and forces them to respond. "I sent it three minutes ago, they haven't replied!!!!!" Well, maybe they're doing something else. Maybe they haven't read it yet. Maybe they aren't sat in front of their computer. Maybe their email client isn't running. Maybe their computer is turned off. Maybe they aren't even in the same room as their computer. Maybe they are, y'know, having a life and doing something else.

[theDeathOfRats]

And as I tell everyone (work, family, friends...) nowadays: Instant Messaging doesn't imply Instant Answering.

[PB90210]

"'But... But... But... They're on company time!!!"

[A.P. Veening]

And answering emails isn't their money producing job.

[Anonymous Coward]

So you may get to explain time zones.

[logicalextreme]

Oh man don't get me started. I've had to field the same repetitive email queries as above — and I can't say for sure, but I suspect that automated report emails are significantly more likely to fall afoul of filters/rules, regardless of whether the recipient is the one who asked for the report in the first place. Ended up scripting something to pull all the "server accepted mail" IDs for a given report just so I could wave the bastards away quicker.

But in addition to that at one place I had to spend every single BST changeover explaining to the finance department why there was an hour's worth of sales missing from the reports and trying to explain how 0100 becomes 0200 without any seismic shift in the spacetime continuum. Also had to deal with the head of commerce at the same place throwing a wobbly with me about the fact that years don't all have 364 days with the weeks neatly numbered from 1 to 52. A week 53 was "just not acceptable" but every week had to have exactly 7 days. Literally threw something on the desk near me to express how unhelpful she thought I was being by explaining all standard week-numbering systems to her.

[Stevie]

years don't all have 364 days with the weeks neatly numbered from 1 to 52.

On the other hand, when else would you have the chance to reply to "WHY" with the Blackadder* reply "It is the way of things".

* Kenneth Connor. Sadly missed comic genius.

[David Hicklin]

> there was an hour's worth of sales missing from the reports and trying to explain how 0100 becomes 0200 without any seismic shift in the spacetime continuum

How did they cope with it going the other way ????

[Sherrie Ludwig]

How did they cope with it going the other way ????

Wow! We doubled our sales in an hour! BONUS TIME!! Yes, some mothers do have 'em.

[Not Yb]

One year I was paying attention to the time on my 'smart' phone around the time change, and it went back 2 hours instead of just 1. Technology bites again!

[Anonymous Coward]

I had to go in to work to turn the clock back at 2 AM. So I set my phone's alarm for 1:10. It did go off... the SECOND instance of 1:10. (I put in a bug report with OnePlus, but never heard back.)

The next year, I set it for 12:55. It never went off. (Maybe something to do with the ability to snooze for 5 minutes.)

After that, I set it for 12:45 and my "dumb" digital watch for 1:00. Both worked.

Thankfully my co-workers and boss have seen the humor in this.

[Steve Foster]

There's no real reason we couldn't alter the calendar to accommodate this.

Switch to fixed 28 day months, have 13 of them, and then an "intermission" non-day to round off the year. Two in a leap year. Any leap seconds (plus or minus) could also be included on "intermission".

c.f. Dave Gorman's Modern Life is Goodish [S3.E7]. (IIRC the name suggested for the "new" month was "Gormanuary")

[Not Yb]

"There's no real reason we couldn't alter the calendar to accommodate this."

The real reason is tradition and history. Thinking those aren't real reasons is a sign of missing something huge about the rest of the world outside the tech bubbles.

[doublelayer]

What problem would that fix, exactly? We can't solve this problem because there is not an even multiple of seven days in a year. Even if we change the length of a week, we still couldn't do it because, on leap years, our new calendar of 73 5-day weeks would break too.

Unless we simply refuse to do anything on the intermission days, we'll need to have some timekeeping concept for them. People needing medical care aren't going to like the do nothing idea. Thus, calendar changes won't help and there's no benefit in doing them, even if the equal-length months would have been nice if that was done originally.

[collinsl]

Just switch back to the pre-made and perfectly cromulent French Republican calendar.

Every month has 30 days, every week has 10 days, plus we could go back to decimal time as well! Today for example would be day 10 of week 4 of Brumaire of year 234.

I can see absolutely no problems with this whatsoever...

[TSM]

One of our key SaaS products (though hopefully not for much longer), from a very major vendor, requires that if you set up a weekly calendar, every year must have exactly 52 weeks. It is not possible to add a 53rd week in a year.

So every four or five years our calendar has a week that is 14 days long.

[Petalium]

I always assume that emails are not something that requires my prompt response and can wait until Friday afternoon, if you are in a hurry call me it’s faster than texting back and forth.

[Doctor Syntax]

Maybe they are ... in a different time zone and it's the middle of the night there.

[StewartWhite]

No, your email is automatically deleted

One place I worked the head of another department would send massive screeds by email then phone two minutes later to complain that he hadn't had a response. If you answered (via group pickup) a call from him where he had dialled the number of somebody else in the team who he'd sent said rubbish to he would first spend five minutes complaining that he'd called somebody else so how dare you pick up their phone and no, he wasn't happy that they weren't sat at their desk.

It got so bad that my boss created an email rule to automatically delete anything he sent and he then told him this, much to the sender's disbelief.

[Yes Me]

Re: No, your email is automatically deleted

"email then phone two minutes later"

Why did he wait so long?

Apparently, when you tell people that their email is being automatically deleted, they get terribly annoyed and send many angry emails. Fortunately, this doesn't matter.

[Prst. V.Jeltz]

Every time I had that it turned out they had misspelled the email address , or were just guessing at it , and had no way of confirming it .

[Michael H.F. Wilkinson]

Not so much Tim Nice-But-Dim

More a case of Dick Not-Nice-and-Thick

[Eclectic Man]

Sadly I can only give one upvote. You will have to decide for yourself whether it is in sympathy for your working for a 'blue blood' manager, or for the superb term "near mortals".

[Doctor Syntax]

"Every. Single. Morning:"

Have an autoresponder saying "The answer is the same as it was yesterday. It will also be the same tomorrow."

[Not Yb]

Knowing this type of manager fairly well... Print out several copies of a page that has "The answer to that question is the same as it was yesterday." in large friendly letters. When asked the standard daily question, hand them one of the copies.

[GlenP]

Email Swamping...

A few jobs ago I'd inherited a setup where a few users had dial up modems and individual email accounts. Clearly this couldn't continue but this was prior to the likes of Exchange Server being readily available (and within budget) and we didn't have the resources to set up our own mail server so I opted for an ISDN-2 connected email concentrator, as they called it. This worked well enough and as it only connected every 15 minutes the call charges were within acceptable levels.

One afternoon however I started getting complaints and discovered the email was being swamped. We only had limited tools but I found the problem after some investigation - ping pong out of office messages! Neither our technical manager or the person at the other end had set the the "only send the message once" switch on their email client (why the heck that wasn't the default is another story). After turning OOF off at our end it took a couple of hours to delete the errant messages from the queue (I couldn't just empty it as there were valid messages in amongst the cr*p).

A strict instruction went out to the email users to *always* make sure the send once switch was set.

[Anonymous Coward]

Re: Email Swamping...

Had this several times. Even worse was the "This mailbox is full" or equivalent error messages sent to a mailbox with a OOF or other bounce on it. Before you know it, the mail queues are into the high thousands and nobody is sending or receiving anything...

Not quite as annoying as the HR person sending their new company wide 10-20MB email with lots of nice graphics and sent in a way that it copied it to each recipient mailbox. Took the Exchange servers down for most of a day. Then, the next week they do it again. And again....

[Martin-R]

Re: Email Swamping...

Nope, absolutely no idea what you're talking about here...

From memory, it was about 1,700 emails back and forth before the Notes server fell over (trying to kid myself there wasn't an extra 0 on that count... it was over 20 years ago)

[C.M.R...]

Re: Email Swamping...

If it was Notes/Domino you'd likely need to add two more zeros before it fell over. A much maligned but actually very well built product that just looked a bit meh to the end users.... Finally killed off by IBM's over zealous license monitoring tool that took much more time to maintain than Domino itself...

[Doctor Syntax]

Re: Email Swamping...

"Then, the next week they do it again."

There are times when the only action is to tell them that if they do it again their email account will be taken down for a week, doubling up for each further offence. Let them discover something about eponentials.

[druck]

Re: Email Swamping...

Just set the outgoing email size limit to 25K. With HTML email by default, they'll never get further than sending; What the f

[Anonymous Coward]

Re: Email Swamping...

HTML email nothing. These days, if they're using Outlook it's virtually guaranteed to go over 25K just on header glop alone...

Yes, I have personally, um, *experienced* a truly demented message sent by someone stuck with Outlook, which contained nearly *60K* of header. Yikes.

[Excused Boots]

Re: Email Swamping...

"Took the Exchange servers down for most of a day. Then, the next week they do it again. And again....”

Hmm, how many weeks did it take until you pitched up at their office and punched them in the head?

[Not Yb]

Re: Email Swamping...

A local cable-co's main web entry page had several megabytes of image on it. At the time, dial-up was still available, so anyone wanting to look at the beautiful (lol) marketing-generated front page got to wait quite a while for the two images with the menu-click-areas to load. Unintended Implication: "We're slow and marketing driven, want to subscribe?"

[Phil O'Sophical]

Re: Email Swamping...

And always set the "Junk" flag on automated replies, so that a server which receives them won't send an automated response back.

[chivo243]

Re: Email Swamping...

once had this too, out of office replies, people at the company belonging to the same list servs with out of office replies, I woke up to 30,000+ and counting that day. That messed up the journaling of the mail server some how, and that was never sorted before moving to another platform.

[sammystag]

Re: Email Swamping...

This email ping-ping reminds me of the unfortunate turning on of our system to take charitable donations by SMS in the early 2000s.

It had its debut at, I think, some dog show collecting money for guide dogs. The kindly user would send a text to our number entering a multiple choice quiz and we'd reply with thank you which cost maybe a pound to receive using mobile terminated (MT) billing.

What we didn't know was that, for certain operators, the SMS gateway would receive an acknowledgement that our thank you message was received. By the time that acknowledgement hit our code it just looked like an SMS. At this point we'd reply to the user that his or her entry was not understood (from the same expensive MT number), in response to which our gateway received another acknowledgement, and so on ..

I forget the exact details of the postmortem and clean up but several users had been charged hundreds of pounds.

[Korev]

If he worked in The City, shouldn't the Regomiser have christened him Rich?

[phy445]

Agreed. See icon for reward.

A while back I suggested that the Regomiser code be open-sourced. Had that happened then this egregious miss-naming might not have happened!

[Sparkypatrick]

Think 'Leeson'. Or at least, I did.

[Tim99]

"Loadsamoney" (Harry Enfield) - May not work for left pondians?

[Tim99]

I must stop posting to El Reg in my time zone ,when it is early in the morning BST. For a while I haven't been able to do/undo changes within the last 10 minutes for a post. The site only shows my stuff from (a few?) hours ago.

My Loadsamoney suggestion is probably not for the blue-blood types, as it seemed to be for one of Thatcher's new money Essex types? Having said that, I was a UK Scientific Civil Servant at the time, and most of the blue-bloods and Essex types that I met there were pretty bright.

[Filippo]

I don't do system administration, but shouldn't a good email server be able to automatically detect such situations and just start dropping incoming messages from the offending sender, rather than collapse outright?

[Claude Yeller]

Re: Good email server

I have heard rumors about the existence of such systems.

I understand from those who did administrate and manage email servers, that setting up such systems is "intricate" and has been described in terms reminding me of painting pentagrams on the floor, lighting candles, and reciting incantations in a forbidden language.

These experienced administrators opinioned that email can indeed be set up right, in theory. Practice told them otherwise.

BTW, if you setup such a system nowadays, the odds the big email providers, Gmail, MS, etc, will actually accept emails from your servers seem to be pretty low.

[jake]

Re: Good email server

Nah. Milters are easy to use, if you understand how email actually works.

I first used the concept just about the turn of the century, on Slackware 8.0ish, running Sendmail 8.12ish

Your last line is bassackwards ... Us independent email operators are far more likely to drop all of google and Microsoft than the other way around.

[Claude Yeller]

Re: Dropping email from the biggest providers

"Us independent email operators are far more likely to drop all of google and Microsoft than the other way around."

It seems most email clients do not like this approach as up to half of their relations use MS or Google hosted email. Most of the rest is hovered up by the big network ISPs. Small independent email providers are a rounding error I gather.

[Kevin Johnston]

Re: Good email server

You are correct in that a number of different mail systems do offer that but one of the biggest limitations is coping with the different ways other systems do their Out of Office responses. Then add on to that different languages if you are sending email internationally and you discover the good old 80/20-20/80 rule where 80% is easy to handle and only takes 20% of your time and effort, I'm sure you can guess the rest. This rule also comes in a 90/10-10/90 flavour

[rg287]

Re: Good email server

BTW, if you setup such a system nowadays, the odds the big email providers, Gmail, MS, etc, will actually accept emails from your servers seem to be pretty low.

This seems to be a common misconception. I have a few domains on non-big-boy providers. A couple of clubs/societies using email provided with web hosting on cPanel & Plesk, along with some other fully self-hosted bits. I talled up the members of one of those societies and it came out about 25% Google, 25% Microsoft, 25% BTInternet and 25% "other". Leave the anglosphere and you'll see lots of other ISP and local providers who Google and MS have to exchange mail with. It's very much not the duopoly that people would have you think.

It is true that if you stand up a mail server and start firing out mail, then you'll have a bad time. If you want mail delivered then you need:

* to check your provider actually allows SMTP traffic and hasn't got port 25 closed.

* PTR records

* SPF

* DKIM

* DMARC (weirdly the presence of a record is more important than whether it actually imposes a meaningful policy)

* Checking your IPs are not on the spam lists. Which they will be if you've just spun up a DO Droplet or cloud VM.

* Some sort of TLS - STARTTLS at a minimum. Ideally MTA-STS, but that calls for configuring a web server as well to serve the MTA-STS policy.

And you'll want to slow-loris your traffic until the big boys get to know you. Don't send a billion emails on day one. Whenever I've stood up a new mail domain, I always get a few people on gmail/outlook to send some emails to me. I also send some to them and have them reply to me (and mark the domain as a safe sender). This gets you off the ground and builds a little domain rep.

That all sounds rather onerous, but it's mostly setting DNS records, and overall those bits are all easier than getting your head around Postfix and Dovecot configs.

If you've got a web-facing server and you're capable of hardening it and managing the mail software, then the other bits are pretty easy. You just need to know to do them.

[FirstTangoInParis]

Re: Good email server

At least these days you don’t have to hand edit send mail.conf. Or sacrifice some poor apprentice. But yes, DKIM is just a minefield.

[Anonymous Coward]

Re: you don’t have to hand edit send mail.conf. Or sacrifice some poor apprentice.

Whaddya mean "or"?

[Doctor Syntax]

Re: Good email server

I understand from those who did administrate and manage email servers, that setting up such systems is "intricate"

Deleting the offender's account is quick and easy, however.

[jake]

Yes.

[Filippo]

I don't usually complain about downvotes, but I'd really be curious about what's objectionable with my post there. It's an honest question, I've never managed an email server (not at any scale). Is the answer so obvious that I should've known it? Am I missing something?

[phuzz]

I think just mentioning 'running your own email server', was traumatic enough to cause PTSD in those poor unfortunates who have done so before.

[doublelayer]

I don't know why someone did, but if I had to guess, it might be a philosophical clash about what an admin should be doing with systems they manage. While there are those, and one of them has replied, who will block on a whim with the "I don't like Google, so GMail will get dropped because I say so" approach, it's more common for an admin to decide that their job is getting as much valid mail delivered as possible and that it's the user's decision whether they want to exclude it. Filtering out messages that impersonate a different server or contain dangerous links or attachments would be that admin's problem, but deciding whether to drop messages legitimately sent from an authorized internal account to another runs the risk that it was supposed to be delivered and your decision to drop will anger someone or break something. Without explicit rules, it's common in my experience for the admins to take a hands-off approach where they will respond if and only if the activity is causing technical problems, and since they didn't know that was going to happen, they didn't preemptively take steps to handle it.

[xyz123]

I worked for a bank that did something similar. A Bank that was made of stone. But not Southern Eastern or Western stone......

over 200 million emails and alerts went out. I found out about 8months later the guy had done it on purpose as it effectively buried emails warning he was transferring 10s of millions out of the banks accounts to other banks in countries with no extradition treaty............

[ajadedcynicaloldfart]

Northern Rock

You're referencing Northern Rock perhaps? If so, you owe me a pint!!!

[abend0c4]

The Log4j plug-in had no place in a release system

Not the way to go about it, but you can't deny he might have had a point.

[Anonymous Coward]

Valuing credit derivative products

There's a feller stands in the middle of Westminster Bridge that invites people to invest in his three shell monty product /s

[MiguelC]

The only thing I find odd about this story

is a project manager doing the work himself....

[Naselus]

Re: The only thing I find odd about this story

Yeah, my suspension of disbelief was stretched to breaking point by the idea that a PM a) understood what Log4j is, and b) was capable of actually removing it from the solution by himself

[that one in the corner]

Re: The only thing I find odd about this story

> a) understood what Log4j

Who said he did? "Anything I don't understand is worthless and can go"

> b) was capable of actually removing it from the solution by himself

Even the most meagre of intellects, once they have figured out how to edit a file (and save the changed copy!) can usually work out what the delete and backspace keys.

[An_Old_Dog]

Re: The only thing I find odd about this story

... and able to work out the git command line?

Or, was it a "we don't do source control" type of place?

[cookieMonster]

Re: The only thing I find odd about this story

From experience these are the worse types. Not good enough to do the actual work professionally, but have enough information to think that they can.

[munnoch]

Re: The only thing I find odd about this story

Indeed, someone needs to explain to him why he was promoted to manager...

[Herby]

Re: The only thing I find odd about this story

Promoted to manager? Peter principle in action. Most people in manglement get there this way.

[Doctor Syntax]

Re: The only thing I find odd about this story

I also found it odd that he became a team player.

[oldstevo]

Re: The only thing I find odd about this story

Indeed.

A PM that

a) acknowledged he had screwed up,

b) learnt from the experience

c) became a team player

where did they find such a unicorn and did they have him bronzed????

[David Hicklin]

Re: The only thing I find odd about this story

> I also found it odd that he became a team player.

It was either that or the stairwell/window options

[Anonymous Coward]

Test by Project Manager

As a project manager for various engineering fields, I've long held that if a product needs testing, just let the PM try to use it. I've astonished many project teams over the years with the ability to find a bizarre and unexpected way to break the product they've just finished testing while simply trying to use it...

When you absolutely, positively, want to break something; just let the PM try using it.

Needless to say, these days if I try to go anywhere near the product there is usually an engineer screaming "STOP! Back away from the product!" ;-)

[Anonymous Coward]

Dot Not

About 15 years ago I was a developer on a team of 4 working on .Net stuff. We came in one Monday morning to find a message from our Dev Manager along the lines of "I've spent the weekend migrating all our code to $next-dotnet-version. It's pretty much there, just needs finishing off and testing. Can you guys take a look."

Of course it was not "pretty much there" and we had to spend days fixing it all up and making sure the system still worked. Whilst all the time still being chased by said Dev Manager as to why our normal work wasn't progressing. We got there in the end of course and to be fair it did prove more interesting solving all those bugs than all the usual shit we had to deal with. But I don't think said Dev Manager lasted much longer in the company.

[tip pc]

Project Manager doing code?

did the project manager do the code changes to remove log4j or did they instruct a team member to do so & got no feed back as to why that was a bad idea.

I don't think I've met a project manager that actually does the work, they typically ensure the project is managed to achieve its goals.

[Anonymous Coward]

Regomize as "Nick"

You wouldn't think any British bank would ever again employ a chap called Nick; would you ?

[Red Sceptic]

Re: Regomize as "Nick"

Not sure how many of us here are old enough to get that reference - but I did! Nice one!

(For anyone wondering, search Wikipedia for Nick Leeson.)

[Anonymous Coward]

Re: Regomize as "Nick"

I doubt many Robins get to work in banks, either...

[collinsl]

Re: Regomize as "Nick"

I remember seeing on the news a few years back a PC Rob Banks talking about the impact of budget cuts to the police.

[munnoch]

Re: Regomize as "Nick"

I had a front row seat for that one. Arrived at work in the Tokyo office to be told -- "that big swinging dick has blown up the bank!".

After Barings I moved to Merrill. who blew up, then Bear who blew up...

[mechgru2]

Re: Regomize as "Nick"

At the time of the Leeson affair I worked with Roger Leeson. Great bloke, he was forever having to deny he was related to Nick.

[PRR]

Re: Regomize as "Nick"

> ...old enough....Nick Leeson

Just today:

"‘Rogue Trader’ Nick Leeson Hired by Risk Technology Firm HedgX ----- Oct. 28, 2025 Bloomberg news

> Nick Leeson, the former derivatives trader who brought down Barings Bank, has joined risk technology company HedgX as an advisor.

Leeson, who gained global notoriety after racking up hidden losses of around $1.4 billion at the bank in the 1990s.... He will help advise the company on risk governance, oversight and business development, .....the latest attempt by Leeson to move on from a decades-old financial scandal that is still infamous ...

[andy the pessimist]

call the dB person

If there was 2billion sql errors the database was badly broken. Shouldn't the dB admin be called?

[ttlanhil]

Re: call the dB person

not necessarily - maybe their script does updates that fail in a way that doesn't abort further processing

Trying to update a transactions (the financial kind) table and throwing an error per row could easily do that

Or worse yet, a retry loop with no limit or delay...

[doublelayer]

Re: call the dB person

A bit of code that retrieves a record, checks it, tries to change it and fails, then continues on to the next record could generate that kind of error billions of times without breaking the database or even the data, which since the update statement failed would be exactly the same as it started. A good update script would detect the error and stop, and a person who wrote a good update script would probably have done different things.

[Anonymous Coward]

The immutable attribute can be profitably deployed against manglement hazards.

Those that know about Linux's chattr +i † and the mischief makers appear to be orthogonal.

I have certainly stopped a few Dunning·Kruger afflicted individuals in their tracks.

Why the so afflicted have an overriding compulsion to fix what is patently not broken is one of the Universe's greatest mysteries.

"Si non confectus, non reficiat." — Vetinari family motto.

_{† I vaguely recall HPUX 10.x also supported this on HFS file systems but not on VXFS.}

[G.Y.]

bedlam3

bedlam3 springs to mind

[agurney]

"This week, meet a reader we'll Regomize as "Nick" who in the late 2000s..

Oh come on, get real, we're only 25 years into the 2000s, the late 2000s won't be until 2900 :(

[David Hicklin]

Although in the context of where we are not I think most of us parsed that as 2000 - 2009

How would you present it ?

[agurney]

How would you present it ?

In the same way as I might refer to the late 1600s, late 1700s, late 1800s and late 1900s.

We are currently in the early 2000s .

[Michael Hoffmann]

I'm still scouring the article (and the comments section), trying to find out why a PM had the required level of access to be able to just remove a module/package/service?

OK, yes, I'm probably assuming waaaaay too much that they had peer-reviewed source control for infrastructure-as-code blablablabla, rather that "everybody has root, even the lawyers".

[Stevie]

Bah!

I worked for a large government agency which consolidated several smaller agencies into one email address book. Turns out, the woods were full of me. Us. Whatever.

We also had quite small allowances for our mail accounts.

One of the Other Mes had very difficult correspondents who would not check the address book properly and just send to the first me they pulled up.

Which was me. Still with me?

One such person commanded a team. She would send out emails with humungous collections of attachments on a Friday night. Bad enough.

But each member of her team was straight out of a Dilbert (pre-suck years) cartoon, and felt moved to reply with inane "Good catch" or "I agree" replies.

Did I say "replies"? Silly me. I meant "Replies to All". With all attachments, er, attached.

I would return to work on Monday to find eleventy billion "YOUR MAILBOX IS FULL" messages (though how that was helping matters still eludes me).

So I wrote a mailbot that would send "Wrong me, Ms Bloggs" message along with a bounced copy of her email c/w attachments. I would also bounce any "replies to all" from her team back to sender c/w attachments, and would send Ms Bloggs a copy of her team member's bounced reply c/w attachments.

So she would get something like eleven copies of her original boxburster greeting her when she took her next look.

It still took her three more mailstorms to get the message.

[OllieJones]

HR screwup hoses email server

In the 1980s this old guy worked at one of the workstation hardware companies.We were using an email system developed in-house, with the hope of beating some bugs out of it before we unleashed it on paying customers.

The HR department planned an outing to a local amusement park. You know the deal: roller coasters, fun-houses, a tolerable band playing music.

They got the bright icebreaker idea of randomly giving out tokens with a number on them. The idea was to find the other person on stafff who had the same number, and collect a prize. I knew a fellow on the email server team. He took one look at this and said "here comes our load test."

So, people started sending emails to "all" with their number. Other people started hitting "reply all" saying "stop doing this" and "hey, that's my number".

The experimental email system disappeared in a cloud of greasy black smoke.

The good news was, it stayed gone and the company adopted sendmail.

[Ian Johnston]

If the Log4j plugin generated emails from errors, and was removed, what then generated these two billion emails?

[doublelayer]

As I understand the article, there was one component which sent emails for errors and the Log4J component was responsible for parsing them and sending only those errors that were above a threshold. For example, that might have been responsible for filtering out warnings that were not considered worthy of an email or for turning a bunch of error statements into one error report. Without that filter, the raw log got sent to the email reporter without anything in between.

I've done this before. As a child, when I set up my first public server, I decided that receiving an email whenever someone attempted to log in too many times and got locked out would be interesting. I could see patterns in that data, find out what credentials opportunistic attackers might be interested in, and at the very least it could be useful information to help me learn security. I did not have an idea of the scale of automatic SSH spambots. That logging got changed after a few hours and thousands of emails. I didn't just disable it, though. I have a parser run over those logs and alert me to specific types of information, for example when an attacker gets a username right rather than just trying root, admin, pi, user, etc, or when there are longer-term patterns in the data. The way I implemented it, if that filter turns off, I would get no emails, but it would be very easy to configure it in the way the article suggests where the filter being turned off means I get flooded again.

[Ian Johnston]

Thanks. That makes sense.

[cookiecutter]

and that's it!

PMs should not be allowed to make technical decisions. The most successful projects i've been on had zero PMs,

they hold the diary. ask "is it done yet?" and that's it!

I know of too many projects kiboshed because a PM thought they were technical

[Anonymous Coward]

It's worse when you have a company owner who thinks they're technical, but in fact has no clue and won't listen when you try to explain.

[D-Coder]

OMG, I had one of those.

His code is the reason for the (new) old saying, "If all you have is a hammer with three heads, everything looks like three nails."

[Anonymous Coward]

Don't fear, they now have AI to help them implement their technical prowess!

[Tim Cockburn]

Similar Email storm

There was an oddity in the company email system whereby finger trouble could send a mail to everyone on the system with a name beginning with the letter you typed.

It got patched after a mail about a boyfriend's behaviour was sent to everybody whose name began with B. Fellow inhabitants of Bracknell may recall why.

[Anonymous Coward]

This column should be renamed "The thank God it wasn't me file".