KCL staff offered emotional support, clergy chat to help get over data loss Staff at King's College London are now being offered counseling and prayers to help them get over the data loss suffered during October's catastrophic IT failure. A month ago, departments across the university suffered “irretrievable data loss” when a 3PAR's one-fault-tolerant RAID Array kicked the bucket. Almost every system …
I wonder if the Chaplain says: "Do not worry my son - it is undoubtedly all part of God's plan" - or "That is God's punishment for your sins".
The former for social sciences data, the latter for real science data. Sadly the arts faculties are unlikely to have data to lose.
Now, ignoring the mechanics of the IT department's ineptitude, people will be searching for a reason, and I think the Chaplaincy WILL be able to offer a reason: God is angry with KCL.
Religion isn't to blame for people being dicks, they manage that all by themselves.
- It does seem to be a primary source of permission for being dicks, just ahead of football, politics and sometimes race/community*.
* I grew up in Northern Ireland, were all of the above were often in alignment.
Would make for a surreal scene in a monster movie:
Angry crowd with torches and pitch forks storm Frankenstein's castle because his monster has wreaked havoc.
Baron Frankenstein then appears on the parapet and announces:
"Good News! I've arranged anger management counseling classes for all!"
I can't claim any credit for this, I saved it from Usenet a long time ago:
Yesterday,
All those backups seemed a waste of pay.
Now my database has gone away.
Oh I believe in yesterday.
Suddenly,
There's not half the files there used to be,
And there's a milestone hanging over me
The system crashed so suddenly.
I pushed something wrong
What it was I could not say.
Now all my data's gone
and I long for yesterday-ay-ay-ay.
Yesterday,
The need for back-ups seemed so far away.
I knew my data was all here to stay
Now I believe in yesterday.
How does this Staffers were informed: “It is now safe to save new work on all drives.” work?
Does it include individuals using their own USB connected hard disks? If why not, because that should give individuals the security of their data they need.
As an aside, did the head of IT get his P45 for the monumental cockup?
"The Chaplaincy has arranged two special drop-in sessions where people can either speak to a Chaplain 1:1, or share experiences with other people affected".
I read that as "Chaplain 1.1" and wondered if "Chaplain 2.0" was a bot on social networking ! Or even...
"Computer, activate Emergency Theolgical Hologram !"
"Please state the nature of the Ecumenical emergency, my son"
(I think we need an "I need to get out more often" icon for such misunderstandings...)
IT Services has pressed ahead with the new back-up system which was going to replace the old one — and which in the actual act of replacing it, managed to destroy the backups — using “next generation technology”.
So the new tech wiped out the data from the old tech???? Or is it the new tech can't read the old tech's data? I hope there's a serious flogging in store for someone then for not testing compatibility.
RAID is not backup - it's redundancy against some hardware component failure
Disk/Volume Mirroring is not backup - it's resiliency against loss of source disk, but will still happily replicate corruptions and deletions of data
The only way to do this is make a regular, complete copy of your data sets, logically separate to the source data, preferably in an offsite location, that cannot be overwritten (unless it's expired and replaced by other copies elsewhere by design/policy).
Backup products are with big tape silos and disk silos and de-duplication tech are not cheap, and not always 100% reliable, but they are very necessary insurance policy over disk failure, data corruption and user error.
Can we take this opportunity to discuss our own domestic data backup arrangements and weigh them up? I'll start:
Main resource is a dlink NAS with 2x 2Tb drives in RAID , once weekly a vbscript/robocopy copies this data alternately to
1) Another dlink NAS with a single 2tb drive in
2) A 2tb drive stuck in the back of one of my computers,
So i have 4 x 2tb drives with the same data on . Am i safe? ...must move that 2nd NAS to the garage really for off site storage
Seems like you're covered for everything except a house fire, burglary, electrical surge and virus that attacks D-Link NAS boxes.
Unless your data is almost worthless, which doesn't sound like the case here, it's worth paying for an offsite backup service. The one I use costs £4 a month and definitely works, having had my laptop's SSD fail shortly after my NAS was misplaced in a house move. I don't think Mrs Coward would ever have forgiven me if I'd lost 400GB of photos.
> Can we take this opportunity to discuss our own domestic data backup arrangements and weigh them up?
Sure thing.
Main resource is a 4U rackmount server, running freeBSD and ZFS (24TB RAM, 6 core AMD), and 4 * 3TB raidz2 for main storage, 3x 1TB raid0 for scratch, with a UPS. Main file server, and runs quite a few VMs and background processing.
Backup is:
- daily snapshot && backup to a 6TB external drive
- monthly full zfs backup to another 6TB external drive
- weekly rsync's of core data to my friends server in France (and in return he syncs his backups to my machine, bit of a virtual "disk swap" backup on each others boxes going on).
So far have suffered lightning strikes, floods, two massive array failures and an over active squirrel without losing data. ZFS in particular is amazing, as it can restrict failures to block level, so even when array was trashed, 70% of the data was still accessible because different parts of different disks had failed. So far have had to restore from external disk twice.
It seems like a very resilient setup, and everything is scripted so I have nothing to do (except put the external backup drive in the slot). The two external drives are key for quick restores, the offsite backup is just to fill in anything that may be missing locally after the restore.
Likewise when my friends server bit the bullet, he was able to get his core data back from my machine, so it really helps to have an off site backup. This setup is a like-for-like swap, and it implies a certain level of trust (otherwise you can just encrypt everything before you send it across).
Sure, we can do the "my backup is better than yours..."
So let's start... at home..
Servers are Replicated to Secondary Servers offsite - with hourly snapshots - handy for some basic recovery stuff.
Servers are backed up locally to Hard Discs (in an array or 3, every 15 minutes)
All data (documents, music files, photos, that kind of stuff) backed up to 2 different off-site services (one I control, one I don't directly control) - real time as it happens
Daily Image sync/updates and backups to another offsite location.
And yes that's at home. Because I'm paranoid. So multiple places, types and shapes of backup imaging and recovery options.
I don't want to be THAT guy at work who preaches about backups to everyone else - staff, customers and so on, and then be the one who has to admit they lost stuff. It also serves as a handy way to check how good or bad this stuff is in reality.
I have one major work laptop to backup.
I have a 2x1TB NAS in separate building. Nightly differential backups of main data directories for 30 days. Weekly differential backups of entire hard drives
Approx monthly, complete backup to external hard drive that is then unplugged and put in a safe drawer.
Also a lot of 'live' code is available on production servers as well (which are also backed up to AWS by me, as I don't entirely trust the hosting providers backups)
Main backups largely cover accidental deletion or corruption (can go back to yesterday/last week), also total disk failure. Also loss or theft of laptop. Having NAS in separate building provides safety from fire and a thorough burglar. If one of our occasional overflying RAF jets ploughs into the office and destroys everything I'll probably be beyonf caring.
The external hard drive backups give an extra level of protection against nasty ransomware that encrypts the hard drive.
I'm sure I've missed something...
This post has been deleted by its author
I AM my own IT manager and I don't trust my backups. I make additional personal ones and tell users to additionally back up everything as well as having it on their laptop AND the server.
You can't be too careful. Fire, Flood, plague, locusts, rats, mice, rooks etc. I also backup "off site".
I've spent my entire career in IT doing my own backups. Reasons are fourfold;
1/ I don't trust an organisation-wide backup to restore something in both good order and good time.
2/ When I mess up, I can fix it myself without having to go bothering others.
3/ Having to redo a week's worth of odd-jobs, if you can remember them all, is the stuff of nightmares.
4/ If you want a job done right, do it yourself.
Yes, and yes.They have lost research data, admin data, student data. They did not have proper tape backups, they were relying on 1st and 2nd level backups stored on the same 3par as the live data. Unadulterated management incompetence, these idiots should not be allowed to flip burgers.
Backups for data in theological organisations strike me as about as logical as lightning conductors on church steeples. Surely if you place your belief in a God, then under what circumstances would your deity of choice decide to smite the building in which you worship, or destroy your data-set?
Unless, of course...there's...
Watch me...
Now, if it's just "business information", I'm happy to abide by the rules. But, if it's data on which the grant of my degree depends, I, not some IT director, will decide how to keep that information safe and available. Because, if you want a job done right...and it's your ass in the sling if they screw up (and they will, sooner or later).
Data security issues? Pull the other one. The risk of my losing the backup drive on public transit pales in comparison to the damage that could be done should some IT numpty manage to destroy the data.
Effin A.
Nobody is going to tell me I can't back up my own data, and nobody is going to keep me from doing so. Anybody with a brain at KCL should have a stack of optical discs at the ready, because completely losing data when you have an IT department with dedicated staff and adequate funding is just proof of total incompetence from the top down.
Fortunately my uni only backs up the standard Windows system, which only poor students and other losers use. I wouldn't let them anywhere near the data on my data. So I take the stuff home each evening. It's against the data protection laws? Catch me! Prosecute me! Lol!
Nice to see KCL getting back to the vision of their founders, but perhaps a little more emphasis on the other half of their motto is required: Sancte et Sapienter "With Holiness and Wisdom". The backups are certainly holey, but where's the wisdom?
Disclosure: I'm a UCL Grad.
A new job advert from King's IT:
https://www.hirewire.co.uk/HE/1061247/MS_JobDetails.aspx?JobID=74209
"The post holder will be responsible act [sic] as a bridge between stakeholders in the organisation and will perform a range of activities which help ensure the operational effectiveness and excellence of the organisation through identifying, defining and validating internal and external changes."
Poor English aside, King's IT are still doubling-down on appointing people to these kinds of pseudo-ITIL non-jobs and outsourcing to Cornwall (!) while the in-house IT rots. They've learnt nothing. Disgraceful.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
