Techie assumed boss knew what they were talking about and killed Exchange by deleting data

Monday mornings can be filled with trepidation and tension, which is why The Register kicks them off with a little light-hearted fun in the form of a new instalment of “Who, Me?" It’s the reader-contributed column in which you admit to your errors, and reveal the consequences. This week, meet a reader who asked to be Regomized …

[GlenP]

I learnt many years ago that you only ever, ever, delete older temporary and log files - never any created or updated since the last server reboot.

Please don't ask how I know! :)

[David 132]

I've told this tale around here before, but...

Many years ago, I had built a master image for some PCs that were going to be used for demos at a big consumer show (E3, CeBit, Computex... I can't remember which one). It had taken hours of configuring multiple applications, figuring out dependencies, and so on. My boss came into the lab, watched me run through everything, and was happy to sign it off for duplication.

So, still chatting away to him, I hooked the gold drive up to our duplication rig, along with a completely blank drive, and fired up Norton Ghost (at which point, dear Reader, you might be getting an inkling of where this is going).

I zipped through the Ghost options and kicked off the clone process, still deep in conversation with my boss. "Haha!", I said, "as long as I don't inadvertently clone the drives in the wrong direction, ahahah!"

A few moments later, I was sitting there in shell-shock contemplating two blank drives, and my boss, having briefly considered making sympathetic noises, was tactfully tip-toeing out of the lab to get out of the blast radius of my wrath...

That ended up being a late-night finish, oh my yes.

[Anonymous Coward]

Done similar, you have my sympathies!!

My "work around " was to try and two different sized drives

[Scene it all]

I have done exactly the same thing on a running production IBM 370 system, back in the days when such a thing filled a room and was attended by "computer operators". I was *intending* to make a backup of "PROCLIB", the library of operator-triggered procures. This is things like "run payroll". Except I got the names backwards and copied the empty disk onto the full one. The "operators" immediately noticed this and went off for a smoke or a snack while I recovered PROCLIB from a backup tape. At least it did not take too long and I kept my computer room access privileges.

[J.G.Harston]

I just cannot understand how people cannot understand that continuously and deeply dragging somebody's concentration away from what they are doing will drag their concentration away from what they are doing.

I get it with family members who pester me for something WHILE I'M IN THE MIDDLE OF CROSSING A ROAD TRYING TO AVOID BEING SQUASHED BY TRAFFIC!!!!

[Anonymous Coward]

Sounds like a story you should send to El Reg to have your name Regomized and the story shared on a Monday... Or you could always post it as an AC.

[Cliffwilliams44]

What he was looking at WERE NOT temporary files, they were database log files! The reason they were so large is they had not performed a proper backup in a long time! A simple VSS backup would have shrunk those files. Another method is to temporarily enable circular logging! I find it amusing that someone who thought they knew Exchange, obviously DID NOT!

[Anonymous Coward]

this

[Caver_Dave]

Instructions like that ALWAYS need CYA paperwork.

[simonlb]

Usually in the form of an incident ticket which is updated with exactly everything that was done so there is a clear audit trail. I once had a head of department berating me to do something to fix an issue and I point blank refused until they raised a ticket. Oh, the threats of losing my job etc. they threw at me to try and intimidate me into doing it, but I refused. Once they raised a ticket, I did the work then put a complaint in against them (there were plenty of witnesses), and they got a bollocking for it had to give me both a verbal and written apology.

Processes are there for a reason, and you don't fuck about in Production.

[Philo T Farnsworth]

Stories like this make me so very happy that I spent my career in university research and not the commercial world.

In research nobody knows what they're doing and often gleefully admit it.

[Anonymous Coward]

If you're doing any work with an exchange server you really should know that the log files aren't what anyone sensible would think is meant by the term 'log file'. It's a part of the database that holds all email, specifically it's all the bits that have been changed in the database. Yeah, don't mess about with it.

[Francis Boyle]

If you're doing any work with an ̶e̶x̶c̶h̶a̶n̶g̶e̶ ̶s̶e̶r̶v̶e̶r̶ [Anything MS related] you really should know that the ̶l̶o̶g̶ ̶f̶i̶l̶e̶s̶ ̶a̶r̶e̶n̶'̶t̶ [anything isn't] what anyone sensible would think is meant by the term

[Brewster's Angle Grinder]

In defence of Microsoft (and I hate to do this) but the term log has wider application. E.g. sqlite3 has write ahead logs which are an integral part of the database. Similarly there are the logs of operations in journaling file systems.

It's an unfortunate term. But not all logs are for human consumption.

[I ain't Spartacus]

You're absolutely fine to delete the Captain's Log though. It's all just irrelevant meanderings and nobody ever listens to it again.

[Excused Boots]

This is true, they are transaction logs, ie ‘what has changed’ and aren’t committed or merged into the (in Exchange’s case) the edb file until a backup is performed.

[Scene it all]

Transaction logs are also what gets shipped to remote computers in some implementations of "hot backup" or "replicated" configurations.

[Pascal Monett]

"took the praise with good grace"

And praise well deserved too :)

[Anonymous Coward]

Tight fisted

I once worked at a company who were very tight fisted with what they wanted to spend on PC's. Their AS400's - no problems, their citrix - no problems, but real servers which we needed for quite a lot of users - not a penny. Even worse if they were /spit/ Mac users.

One of the servers we used was basically cobbled together with whatever disks and disk boxes we could get and was a way of allowing Mac users to share data with those who still had PC's or were on Citrix.. It was reliable, but could often run out of space. In the end I created a small folder and left up to 10 x 500MB files in there, so in a scenario like this I knew what I could delete to keep the lights on while then pestering the users to sort the real issues out (the teams still working on the files instead of using that space as a copy/paste into and then immediate cut/paste out of,)

[simonlb]

Re: Tight fisted

One job doing out of hours support we were expected to use an eight year old Dell laptop with 256Mb of RAM and a trackball - yes, a trackball - to do the job. What should have been possible in half an hour took three hours because it was just so old and slow. I got the IT guy to install the relevant software onto a spare desktop machine with 2Gb RAM (XP days) and used that instead - it was just like being at my desk. Was frowned on by management but it was either that or find someone else to do the OOH support on that relic of a laptop; the money would have been great, but I would have completely lost my sanity.

[GlenP]

Re: Tight fisted

Years ago I had a similar issue - the only laptop we had was fairly ancient. We had an issue with connecting via a serial cable to a mobile (for terminal connections to the AS/400) - it would only work if the mobile was on charge at the time. The conclusion was that the laptop was using TTL logic chips and the mobile CMOS, they should have been compatible but the TTL chips were less tolerant of signal voltage. It worked though, and got me out of a few holes.

[Anonymous Coward]

Re: Tight fisted

I once (about 20 years ago) had to deal with some software which couldn't handle more than 1TB free space on the (networked) data drive. We discovered that when we upgraded to server to a 4TB RAID array.

I initially created a huge file to fill the excess space (being sure to not have that included in the backup) but as they added more data I realised the flaw in that plan and did as you did - several 500MB files which can be deleted as needed was far more convenient than a 3TB file which needed deleting and replacing with a 2.5TB file and repeating that process.

[Prst. V.Jeltz]

Re: Tight fisted

Thats pretty cunning

[PCScreenOnly]

Re: Tight fisted

Btw, I liked the macs and their users. The the bods with the purse strings didn't like them

We went through checks every so often to see if they could be moved to pcs. Mostly there were compatibility issues with older files (typically quark), but then to get an equivalent spec to their macs, the pc equivalent cost more

[Don Bannister]

Re: Tight fisted

Many moons ago (when storage was a lot smaller) I used to use the filler files trick too.

I kept ones called 1M, 5M, 10M etc to be deployed when needed. The fun bit was they were created with all the contents being the same byte or character. The Zip versions were therefore very small.

Customers couldn't understand how I was able to produce 5M & 10M files from a floppy disk !

[JamesMcP]

Re: Tight fisted

FYI, that trick won't work with modern SANs. Most of them have inherent data compression, so what you *thought* was reserving 500MB or 1GB of space is essentially a rounding error. Those you need to fill with noise*

By the same token, the "estimated file storage" metrics from those systems are utterly overpredictions when you start throwing files that are encrypted or already compressed. Always include the the actual disk usage metrics on any dashboards or status reports.

*You should more likely create some partitions or quotas you can destroy to expand the array, but then "shrinking" the partion/quota later may be more of a pain.

[l8gravely]

Re: Tight fisted

Been there! Had a netapp cluster which was supposed to replicate to the other coast over a T3. Users were always filling up disks and such, which meant sometimes having to blow away snapshots. Especially when I found two 800gb log files with the same line repeated endlessly in them. Once they were deleted... I still had to purge all the snapshots to get back the space. Say goodbye to snapmirror to the other side of the country. Sigh...

[Bebu sa Ware]

Re: Tight fisted

> created a small folder and left up to 10 x 500MB files in there, so in a scenario like this I knew what I could delete to keep the lights on

The users in a more "relaxed" Unix environment were sufficiently cluey to use df to find file systems that apparently had available space and to which they had write permission.

I had a small Q&D C program that took as arguments a path to a directory and a size in Mb, daemonised itself, created a file in the directory, unlinked it and proceeded to write the required megabytes to the file descriptor which it kept open. The df command showed the usage but obviously du or find couldn't find the file. Fuser or lsof were fortunately beyond their ken.

Having a couple of these running meant I could a free a quantum of space by terminating a process with a signal.

Most of the problems were due to huge output files being intrinsically sparse but textual with space or tab characters making up most of the file. Compress or gzip on the user's older output files usually solved the problem but you had to have enough free space run compress or gzip.

[Mr Dogshit]

Remember to regularly defrag your M: drive

[Evil Auditor]

“Exchange servers, especially onsite ones, were nothing ... to be worried about"

Wot?! Exchange servers, by definition, are cause for anxiety disorder.

[Anonymous Coward]

We used to run an in house Exchange server (now on 365), it was one of 3 Windows Server 2012R2 VMs (exchange server, DC and file & print server) running on the same physical Dell box.

One day our support company told us the latest patches from MS were OK to install on all 3 VMs after they had tested them, so we set the install for overnight.

Came in next day to no email. The Exchange server had decided that despite running on the same physical hardware, its timesync didn't match the DC.and the Exchange services refused to start.

Whatever was in the patches had somehow made it so the exchange services were attempting to start on that VM before the timesync service on the DC VM.

We ended up having to delay the Exchange VM starting up until 30 seconds after the DC VM to give the DC time to get itself sorted out first.

[Anonymous Coward]

Have you landed in trouble after following orders?

No.

When insisted upon in spite of my doubts/objections, I just asked for it to be in writing.

Duly signed*, of course.

* magic words

.

[Prst. V.Jeltz]

you gotta wonder if that I.T. Director

a) even knew about the weekend chaos

b) knew it was his fault

[Doctor Syntax]

I'm sure Phil made sure he got to know.

[doublelayer]

Why are you sure? The best case is that the manager is more careful in the future. The worst case is that the manager gets angry for being blamed and takes it out on Phil, in a situation where Phil already has the problem solved and praise for doing the work. In my experience, there are times when someone is angry enough that I must insist on telling everybody who is at fault and what they did, and there are times when the thing someone did was bad enough that it also must come out, but if everything ended well, I don't have enough reason to make people angry. Also in my experience, there are people who react very badly to being told they were at fault for something even if you're saying it sympathetically and making it clear that you aren't trying to punish or insult them.

[Doctor Syntax]

“I refrained from dropping the director in the mire"

Unless you need to do otherwise in self-defence that's the right thing to do. You know where one of the bodies is buried and he knows you know.

[Jonathan Richards 1]

One more time

It's necessary for the, um, leverage, for it to be the case that you know that he knows you know.

[Brave Coward]

Quite typical

So Phil, who wasn't on call for the weekend, spend the most part of it sorting out an error he was not responsible for.

And got 'a praise and a huge pat on the back' as a reward, and that was it.

So typical of corporate behavior.

[doublelayer]

Re: Quite typical

It is, but in the defense of that behavior, the people who gave him praise but nothing else didn't know how much he did, how bad the problem was, or whose fault it was. Had they known those things, the result might have been different but not necessarily better (getting blamed for not knowing that log files were crucial and shouldn't be deleted, for instance). Sometimes, companies are ungrateful for things they do understand, but sometimes, it's more that they don't know when you've done something extraordinary (XKCD).

[The Oncoming Scorn]

Re: Quite typical

"When you do things right, people won't be sure you've done anything at all."

[The other JJ]

Experienced Exchange Server admins know to keep many terabytes of empty files in the root of each drive so some space can be freed quickly to allow the failed database to be repaired, dismounted and remounted to merge the log files.

(I think that's how to do it ... it's been about twelve years since we drank the Kool-Aid and moved to 365.)

[JMcKeand]

Every Exchange Admin Knows

Yes, manually deleting Exchange log files is a huge no-no. Just get Exchange to do it by having Windows Backup do a backup (to a local file). If you are out of disk space, use an external drive. Once the Exchange db is backed up it will delete the log files as they are considered committed to the db. This all assume that the Information Store Service will even start.

[usbac]

Re: Every Exchange Admin Knows

Way back, I had to admin an exchange server (Exchange 2003, I think) and the company was too cheap to license the Enterprise version. If memory serves, I think we also needed the Enterprise Windows Server license to run it, and that added a lot of extra cost.

In those days the information store was limited to 16 GB total. We were constantly hitting this limit. Exchange will dismount the information store once you hit this limit. I had to frequently run the command line defrag and compact, hoping that there was just enough "free" space left to be able to mount the store. Then, have everyone clean up their F'ing mailboxes, and take down the store, and defrag it again.

I'm not a big cloud fan, but I was very happy to offload Exchange to O365.

[Sandtitz]

Re: Every Exchange Admin Knows

"If memory serves, I think we also needed the Enterprise Windows Server license to run it, and that added a lot of extra cost."

Exchange ran on Standard edition just fine. Enterprise allowed for more memory (PAE) and >4 CPU sockets (never saw one), but Exchange 2003 was 32-bit and couldn't use more than 4GB. If your company was too cheap for Exchange Enterprise, then likely you didn't have more than 4 sockets.

"In those days the information store was limited to 16 GB total."

18 GB was the initial limit. SP2 raised it to 75GB on all non-Enterprise versions.

[Doctor Syntax]

Re: Every Exchange Admin Knows

"Once the Exchange db is backed up it will delete the log files as they are considered committed to the db"

Does this mean that it's a database that doesn't do checkpoints except when it gets backed up?

Can we have a "Wanders off shaking head" icon?

[cageordie]

Never did it to anyone else

Early on a project, when I was still messing around with tree layout and such, I decided to "rm -rf *" to get rid of my latest experiment. Forgetting that abortive "cd " I had done. I had got as far as typing "cd /".

Then things stopped working.

No worries. I had recently come from a defense contractor and was doing religious level backups. The O/S was a lot smaller then too. About 90 minutes later I was back to where I had been that morning. Next step, create users. No more logging in as root.

[chivo243]

Never on a Friday!

See title, that is all...

[HollowMask]

"Tell Microsoft to FIX IT!"

Many, many years back (NT4/2K era) I had a boss tell me I needed to contact MS and get them to re-write how Exchange/Outlook handled deleted email.

The group CEO, and his cronies, were using the un-purged deleted email as "filed". Critical sales data, CRM, &c.

Once you had about 2GB of un-purged deleted mail, it thrashed the #@%$ out of the server drives searching un-indexed blobs. Searching in deleted mail took tens of minutes.

Yeah...

When I advised them that the deleted mail was kinda supposed to be deleted, his response was to tell MS that a pissant mini-Corp *required* them to modify Exchange to index the deleted content, "or else..."

Went as well as you'd probably expect.

[BartyFartsLast]

Re: "Tell Microsoft to FIX IT!"

Many years ago, SBS2000, regularly running out of disk space, enforced deletion of Outlook recycle bin after consultation with the relevant management, user notifications, requests for acknowledgement etc.

Very first support call after the first enforced deletion "where has all my in progress work gone?"

*Sigh*

[Cliffwilliams44]

Re: "Tell Microsoft to FIX IT!"

Yup had that conversation.

User: "Where is my file"

Me: "Where was it stored"

User: 'In deleted items"

Me: "We purge Deleted Items after 15 days, you were informed"

User: "But I need my files"

Me: 'But you DELETED them! Would you file your House Insurance policy in the trash bin?"

[Thereneverwasaplot]

Re: "Tell Microsoft to FIX IT!"

I've seen CEO's do this before. I have never understood why. The logic is unfathomable.

[FeRDNYC]

Some wisdom imparted by one of my first bosses...

"People don't remember companies that never make mistakes. They remember companies that fix their mistakes well."

Guess that goes for system/DB admins as well!

[The Organ Grinder's Monkey]

Re: Some wisdom imparted by one of my first bosses...

See also car manufacturers that see resolving problems (esp recalls) as an opportunity to give the customer a good interaction with all the downstream benefits that brings. Or do it the way that 99% of manufacturers do it these days...

[Excused Boots]

"Further inspection showed that the vast majority of this was due to a rather full log file directory.”

And the second I read that line, I thought, yep see where this is heading!

[LessWileyCoyote]

Manglement by name, manglement by nature.

[Tim Chuma]

"Too many users in the inbox"

Imagine making $1billion per year from issuing passports and "not having enough money" to have enough users on the exchange server!

Also having a "fax" inbox that can only be accessed from the one inbox!

Not me being forgotten for a year being in inbox jockey and finishing my work in an hour each day.