
Been through this myself.
They will experience a drop in service level, increased costs and longer after-hours response time. This is what happens when the CFO is in charge of IT. Never put an accountant in charge.
I give it three years.
The University of Nottingham has announced it will outsource some of its IT operations in a long-awaited shakeup of the department. Last week, the body told staff it was aiming for a new operating model, with a blend model of in-house and partnered provision. According to the University of Nottingham University and College …
Three years would be optimistic. The last company I worked for started off-shoring IT support and Software Dev to an office in India - the experiment was deemed a complete and utter failure after just 12 months.
The CEO actually had to resign, such was the scale of the disaster.
But did it ever recover? The main problem with outsourcing is you get rid of the inhouse expertise and they dont come back so you are left with outsourcing as the only solution - that's why they seem cheap to start with. You'd think they'd teach this shit in accountancy 101 but, like maths its not part of the course.
People, cheifly those in managerial roles, and the CxO's more still than others, tend to forget that all we have now are computer businesses. Yes, theysometimes have some other machinery attached to the computers (like, baking ovens in the computer businesses formerly called bakeries, or big lorries in the computer businesses formerly called transportation companies). But forgetting this fact when you are a manager or worse the CEO herself, I see no other way out than it costing you your job.
Similar experience. Employer outsourced everything having to do with data centers with most of the support farmed out to contractors in S. America and India. Within a couple of years, the V.P. of Shared Services retired and the Dir. or N. American data centers and CIO were escorted out of the building by security. Everything wound up costing more and fewer services were provided. Not to mention the loss of institutional knowledge about how the business was using IT services. Such a deal!
Nottingham has a reasonable amount of research going on. Just wait until they try to tackle THAT hot potato.
"Yeah, I need 500Gbps sustained with no packet loss, 4PB of object storage, a 200TB NVMe cache, 3,000 processing cores and a 15,000 GPU cores with ECC DRAM of course... no, that's just for my research group. How much is that, please?"
It's usually more prosaically annoying than that.
IT charge 250quid and take 2-4 weeks to provision a network port, so labs become a nest of cheap of cables connecting cheap supermarket 4port desktop switches to the one official port in the corridor outside.
Or you need to keep an old windowsXP machine on the net because it has a card to talk to some 10year old experiment. But IT only allow a standard Windows10 with all ports locked down and no admin so your experiment now has a grad student making a homebrew HID emulator on a Pi to 'type' in the results
I assure you that all the comments about a loss of flexibility are already the case. Information Services rocks from system failure to system failure (e.g. no email for a week earlier this year) and absolutely no accountability (it’s always everyone else’s fault).
This post has been deleted by its author
I've just visited a lab where there are two pieces of equipment being run by Evesham Micro PCs. There's another lab here running the first commercial offering of a confocal microscope on a machine which is run by a 386 using DOS6.2. Trying to get THAT to talk to the new network was a chore.
"Yeah, I need 500Gbps sustained with no packet loss, 4PB of object storage, a 200TB NVMe cache, 3,000 processing cores and a 15,000 GPU cores with ECC DRAM of course... no, that's just for my research group. How much is that, please?"
No problem! we'll spin that up on Azure - that's the benefit of cloud, you'll just pay for what you actually use etc etc... What!? you need it to perform...
And lack of accountability, so you can cruise through the middle and end of your career, doing a terrible bare minimum job, taking full advantage of flexible working arrangements to hardly ever actually have the inconvenience of showing your face, and never be fired... allegedly...
Given Notts uni has a campus in China (lots of UK unis making a presence there) if the decision does fall in favour of in house expect the top bosses to reverse ferret and start emphasising that the China campus is part of Notts Uni "family" and so outsourcing to them is keeping it in house.
Oh for the days when the people in charge of universities ere academics, not people parachuted in from outside on excessive salaries who have no idea how academic institutions work. Notts uni has a long history of solid computing research, first large scale deployment of transputers for parallel programming I ever played with were at Notts Uni in the early nineties (cutting edge stuff back then).
Universities are now a PITA for "proper" academics, lots of pointless box ticking exercises wasting time better spent on research & teaching preparation.
Worst bit for academics is the slipping of standards, students who would have been filed or maybe scraped a third are now getting decent passes as top brass are more worried about low student satisfaction scores than keeping up standards so huge pressure to grade inflate.
"Anybody paying you 9grand a year for a degree is entitled to a nice shiny one"
The sentiment may sound reasonable but, depending on the subject, the shiny degree *actually* costs a fuck of a sight more than that. The difference, however, is buried in HMG's fraudulent accounting and will hopefully go unnoticed for a few decades, by which time the benefiaries will be the politicians with the job of sorting it all out.
The world is full of people spending hours copy-and-pasting and re-typing data that could have been processed automatically with less than one line of awk or a tiny bit of SQL etc.
I was sat next to someone on a train recently doing this: he spent at least two hours copying and pasting to change the layout of a question-and-answer type of document to move the answers from the end of the document to inline after each question. A typical Reg reader would have been literally a hundred times more productive.
Well, given Notty's student body size, and the need in these modern times for nearly every student to have some kind of computer access, I'd be fascinated to hear about how the annual intake, probably about 8,000 in undergraduates alone, gets such access in a matter of days, with or without the involvement of dodgy outsourcers. Self-registration may be on option, of course, but that carries its own risks.
"I'd be fascinated to hear about how the annual intake [...] gets such access in a matter of days"
Having recently put a son into a rival university, I can tell you exactly how that works. Firstly, a fair proportion of that annual intake probably had unconditional offers (either the dodgy kind or the kind that result from taking a gap year) and so have been "known" to the system for about 11 months already. For those people, it is sufficient to notice that they've actually turned up and then switch on their access. For those with conditional offers, they too are probably already partly known through their UCAS application details and when the results come out in August it may be possible to get most of the access set up by some imaginative scripting. (Fixing the mistakes is what Freshers Week is for, right?) For those who only become known during Clearing, you have about a month to process them. Historically, they've been small in number.
I'm told that increasing numbers of students are deferring any kind of choice until Clearing (after their results come out) which is presumably making life hard for universities (in every respect, not just IT), which might be why "people" are finally talking about changing the system in the UK so that no-one applies before their results are known. (That is, results in August, apply in the autumn, start courses in January.) If you are reading this from outside the UK and are not familiar with our crappy system: yes, we do most of our university entrance decision-making based on the *predicted* grades given to us by school teachers. Always have done. It's part of what makes Britain Great.
Speaking as a ex HE IT bod...
Students are registered on to a central student database as part of there application process - this maintains a state, as soon as a place is confirmed (via UCAS, directly etc...) the state is updated.
When the student arrives on campus (or starts getting setup remotely if they never come to campus) there is generally a self registration proces - where by the student enters a few key details they already have (student number, dob, name, etc.. etc...) and then if it matches - they are allowed to create or be issued a login to the University IT system(s), all of which are created by said registration system. It's not too disimiliar to setting up a online banking access.
Nothing manual to it from the IT department side of things, and it's generally surprisingly very reliable given the amount of string tying all the various different systems together!
When the student database changes the state to indicate the student has finished, left, suspended, disapeared or whatever, the process then deletes/suspends/tidies up.
As it happens, I'm an expat who is old enough to remember things like "UCCA" and "Polytechnics". Oh that paper document that you had to write on verrry carefulllly in black ink only using caps and not cursive. And those poor artsie friends who got ridiculous offers like "A & 2 B's" to study their subjects. Ironically one of those friends who missed her grades and got a polytechnic degree instead is now an emeritus prof at my old Uni, where I stunk my way through Chem. As you may know, science swots then generally got a trivial offer which you'd have to be really thick not to reach. If you impressed at the interview you were basically in. Curse those bastards in Leicester who rejected me outright.
Mass-creation of accounts isn't too difficult. I used to run the IT systems in an uni department that had its own network a good few years back. Once our new batch of students was confirmed, a list would be sent over to me detailing their forenames, surnames, central IT usernames etc. which I'd export to a CSV file.
I then had a script I created which worked through the list, created a new AD user account for each student on our own system, generated a random password for each student, configured their home directory/permissions, then spat all the details back out into a second CSV file. From that, it was a simple process to run a mail-merge so that an "account details" sheet could be printed for each student with their name, username and temporary password.
At the first session, the students show some ID and collect their account details - sorted.
No manual account creation in sight. I'd imagine most universities have automated methods for bulk-creation of accounts once the students have registered and their details have been recorded.
The descriptions from Dave K and others match my own experiences in HE. For a university with 20,000 "local" students (full time UGs and part timers, NOT distance learners), there will be about 4,000 staff (everything from porters to professors, including post graduate researchers). Staff turnover is less easy to automate than students who register at known times (although medics and teaching students register before other UGs, and some taught PG courses start at different dates). Staff registration has to be fudged quite a bit because some people are also students or have more than one job. PhD students and the like who are "writing up" without payment from a research council etc have to be retained within various systems.
In the 1990s, most universities wrote their own account creation systems. Nowadays there is a move to off the shelf identity management systems, primarily fed with data from UCAS or HR systems. In the early days of Active Directory, administrators specifically stated that the AD was for authentication to a Windows domain. Feature creep now means that AD authentication is used for library systems, inter university wifi (eduroam), anything that does LDAP -- make your own mind up whether this is always wise.
From a distance, thankfully, I have watched the implosion of an HE identity management project. The external consultants were splendid knowledgeable people, but when it came to things that mattered, in house staff were better informed. Management, largely drawn from outside HE, did not have the institutional background to understand that the project was going wrong and failed to listen to their own employees.
Clueless end user here.
At one of the places I teach in with around 8k teenagers and 5k other enrolments (apprentices, adults, employees on training &c) the workflow is as follows...
1) student fills in basic information on screen
2) after interview with teacher/initial skills check teacher sets up a 'pending enrolment' to course
3) student toddles off to physical desk to show ID and sign forms
4) enrolment becomes active and IT account works within minutes
5) single sign on to most basic systems plus teacher created accounts to a few commercially provided subscription services depending on course
6) Library offer IT induction courses including how to print, e-safety and what that acceptable use policy thingy you clicked on when you logged in actually means
Mostly seems to work.
They get their student ID information before they turn up, then they self-register. Uni doesn't just have to deal with their IT access. Their bigger effort is allocating rooms to freshers, giving them building access passes and individual keys. They do this by setting up big rooms or marquees and deploying hundreds of student volunteers with specially printed t shirts and hoodies.
@Daedalus "creating accounts for thousands of students every October"
I worked for a Uni and had to do exactly this. I administered a bunch of Unix boxes used by the engineering faculty, and got an extract from central IT who managed the VAXCluster (yes, that long ago), so I had to convert the extract into a Unix friendly format for creating the accounts, and tabulate the results so I could print the usernames and passwords off and guillotine them into little slips to give the students. I then got paranoid that my random password generator would generate profanity, as I was creating thousands of accounts it could have happened, and even mixing alphanumeric and special characters could still have created bUg63ry! or somesuch, so I had to eyeball all the results, just to be on the safe side.
Unlikely.
Most staff will have been in post for many years, having graduated from Nottingham and never seen the real world. Redundancy in HE tends to be quite generous, and the work - such as it is - isn't actually that unpleasant. They'll be sitting it out waiting for their package.
When the only upwards path is into management you end up with management by people who weren't recruited for managerial talent but could do the technical job and the technical job being done by people who weren't good enough to be promoted or too inexperienced to show whether they're good enough or not.
"When the only upwards path is into management you end up with management by people who weren't recruited for managerial talent but could do the technical job"
Sometimes, but usually I find that if that's the case, the people who could do the technical job got promoted into management, and because they either weren't good at that job or just didn't like it, they left. So management is made up of some random people who actually like but aren't necessarily any good at management, while all the technical people who were good at the technical work left because they wanted to keep doing technical work. Meanwhile, anyone that gets hired and can do the technical job well will do so, but they see what's coming and they're just biding their time until a different job comes along.
Korev: "It also means there's no career path for junior and mid-level staff. Eventually, this means that the senior people will be harder to find as there is no obvious path to get there."
Systems such as ITIL or Hay Job (D)Evaluation mean that many staff are up the creek already. Any mechanism that determines the value of a job based on its functional elements rather than the contributions of the job holder reduces career opportunities.
And the best bit...because you keep the expensive part of the IT organisation, outsourcing to a cheaper part of the world is typically only cost effective for 5 years if you are lucky. By which time the outsourcing country is starting to pay more for the "experienced staff" you require and any cost advantage disappears unless they can find an even cheaper country.
So, as a best case, you end up spending the same amount of money on a worse service. Almost as if the organisation is getting rid of the part of the organisation that works and keeping the bit that is failing.
"Knowledge Transfer"
A company I worked for was bought by their US based competitor, and after about a month of telling us how great the future would be, they reduced the UK operation to basically Sales and Marketing.
Our R&D and support divisions were "merged with" their US equivalents, and we were given a four week handover period to "bring them up to speed".
As the deadline approached, we were still trying to solve fundamental issues by email as the new owner was not prepared to fly anyone over, and the 8 hour time difference meant working hours didn't overlap. Out of hours working and overtime were simply forbidden.
Most of these issues came about as we used Unix/Linux for development and Macs for the CAD stuff, while they were entirely a Microsoft shop.
On the day we finally turned out the lights, we received a huge email stating the "most urgent" issues that remained unresolved.
Presumably they are still unresolved.
Well Manchester University has close relationships with both the Central Manchester and Salford NHS Trusts and it didn't stop a shedload of people "leaving". 68 was the official figure but it was higher because many others just walked. And some of them were jointly employed with the afore-mentioned trusts.....
One of the medical schools I am involved with maintains completely different systems for university and NHS work. All teaching and research staff have access to the university system, but not the other way round (it would require honorary NHS contracts to access that system - quite sensible in its own way). However, due to some historical stuff, it means that we have two separate virtual learning environments (I hate the term!) - one that all teaching and research staff can access, and one that only NHS staff can access. Guess which one is used most? (I'll give you a clue - NHS computers can't access the university system, and most of the teaching staff are clinicians...)
The brutal truth of outsourcing is that you will get exactly what you specified in the contract and nothing more. Everything else carries a fee and unless you have robust discipline and have planned really well (which few organisations do, especially in the first year) you will frequently find yourself dipping into contingency money just to carry on normal operations.
The reason is that same as why when requiring people to keep timesheets you often experience productivity drops; there were all those little extras people were doing. Someone who was staying on that extra half hour suddenly doesn't when watching the clock. Sadly this is a lesson which is mostly taught by experience.
More common in my experience is where a well-written contract with good service and support coverage isn't delivered because none of the bidders priced it sustainably (to win the business) and can't afford to provide the service levels specified and still cover their costs. You still select one, since you need the service, but when the Terminal Inability To Swifly Unf*ck Problems happens you then have to deal with the shoddy performance as well as the fight to exit cleanly and try again with another vendor.
Quote:
The reason is that same as why when requiring people to keep timesheets you often experience productivity drops; there were all those little extras people were doing. Someone who was staying on that extra half hour suddenly doesn't when watching the clock. Sadly this is a lesson which is mostly taught by experience.
My employer is starting to go through that experience....
Because he changed the clocking system for the hours worked to clocking everyone out at midday for lunch, then everyone clocked back in at 12.30.
Trouble comes ... previously if I was setting a machine up, I'd work into my official lunchbreak time in order that the machine is ready to go when the operators get back at 12.30, then take my 1/2 hr lunch, now everything stops at 12.00 (and indeed slightly before that as I dont want to be buried in changing grippers over)... so the machine spends a longer time stopped than it should.... anywhere upto 30 mins... and the time is charged at £60/hr and we have 8 setters with 30 machines to look after...
Hey ho... guess its worth losing 200 odd quid a week and employee goodwill just so the bean counters have the time spreadsheets looking neat and tidy....
Actually you get what some other fuck wit defined in the contract, which bears no resemblance to what is actually required in order to maintain any semblance of service - shitty or otherwise. That in turn leads to all the stuff missing from said contract becoming chargeable work.
Outsourcers both loath and love shitty contracts, depending on who the best salesman was when the contracts were drawn up and signed. I've seen the effects from all sides, and it leaves me feeling in need of a shower.
Its the other techies and end-users I ultimately feel sorry for. The PHBs and sales driods can go fuck themselves (literally and figuratively). Outsourcing only benefits their pockets.
And even when you get the contract almost right, you'll get the cheapest possible implementation of the literal text, if you're lucky. Sometimes you get a malicious literal genie trying to make you pay for a variance. If it is a terrible solution that no sane person would choose, hard luck.
> The brutal truth of outsourcing is that you will get exactly what you specified in the contract and nothing more
Yeps.
I used to work for a company which decided to go down the outsourcing route to cut costs.
First, they tupe'd a bunch of DBAs over to a service company. So while we were working with the same people, all of a sudden, things which they used to be happy to do (e.g. add indexes, review poorly performing queries, etc) were no longer possible, as it wasn't covered by their contract.
To be fair, I'm not sure how much of that was due to the contract, and how much was because they were cheesed off at the TUPE move. Either way, their willingness to do things in goodwill and/or go the extra mile went straight out the window.
Secondly, they brought in offshore contractors to rationalise the system stack. Only to discover that the contracted company was quite happy to class things like spreadsheets as a "system", since it counted towards their stats. Letter of the contract, don't ya know.
And alongside this, they offered generous redundancy terms, so a lot of people with large chunks of institutional knowledge cheerfully cashed their chips in and left for the south of France.
Traditionally, it had always been a company where a large number of good and talented people did their best to make things work, despite the heavily bureaucratic nature of the company. But between all of the above, it became harder and harder to achieve anything...
This post has been deleted by its author
It seems shortsighted that a university would consider outsourcing IT. In the past, this has been an excellent way for the school to provide on the job training / real life experience for students in their chosen field.
A lot of innovation has been driven by students and grads in just those roles.
Follow the money
The real problem with the cost of IT support is that there is little genuine incentive for the people who produce the IT systems in the first place to invest in reducing actual TCO (as opposed to the fake TCO peddled by salesdroids). Equipment is bought out of capital budgets, support from operational budgets and never the twain shall be summed.
Some IT support costs are inevitable - those related to changes to personnel and requirements, hardware failure, and resilience. A lot of the rest is fixing stuff on behalf of the people who took your money for it in the first place.
IT support overhead is unreasonably large, but the way to deal with it is to start pushing back on suppliers, particularly software suppliers, to deliver stuff that's more robust in the real world.
If it takes 10 people to do a job it will always take 10 people to do the job and one person to manage them. If you out source you will need one person outside to manage the 11 people and one person inside to manage the relationship with and one person to manage the relationship externally, so to do the work of ten people used to take 11 people now you need 13.
Someone needs to be cheap here and when you do IT cheap it invariably goes wrong.
Outsourcing never works, even contractors are rarely worth it, as they are only there to deliver the contract and really don't get what it will look like in 1 or even 3 years time like those that in for teh long haul.
Then you have the outsourcer's profit of 3 head count, plus whatever they can get from maximising head count utilisation across customers, 3 for the client to justify the outsource.
So they're underqualified/experienced, overworked/slower to respond, less familiar with the IT estate and usually remote. All to save the equivalent of 25-40%*. It's a great deal, if you only look at the costs. People buy the hype that they can get the same level of service.
*Not a guarantee. Performance and costs may vary over time. Historic charges are not an indication of future expenditures.
The only time I have seen using outsourced IT as a positive is when the organisation is too small to have the people to do the work.
So a small company with 10 people can get by with a few hours a month from a MSP and have a good IT setup. The other option would be getting some random person there to try and do the IT as well as their current full time role. This can work for a while in a simple setup, but tends to go wrong as soon as anything complicate happens.
4 years ago:
Operations Team: "We really need to replace this aging infrastructure as it's putting data at risk. Can we get a project together with some budget to get it replaced?"
Senior Management: "Don't worry, it'll all go to cloud!"
3 years ago:
Operations: "We do really need budget to replace the old hardware as it's getting increasingly unreliable and it's running out of space"
Senior Management: "Don't worry, it'll go to cloud, we won't need to spend money on it and people will soon realise the benefits of O365 and move data themselves!"
2 years ago:
Operations: "Seriously, we've had to get stuff out of the bin in order to keep the service running, let's get it replaced"
Senior Management: "Cloud!"
1 year ago: "All of the hardware is no longer supported by the manufacturer. All that stuff is at really high risk! Let's move all that data off to new infrastructure including cloud if you want to!"
Senior Management: "Cloud! Cloud! Cloud!"
Today:
Senior Management: "We're going to outsource the entire Operations team partly because we REALLY need to get rid of our aged and legacy infrastructure"
Operations: !!!!!!
"Seriously, we've had to get stuff out of the bin in order to keep the service running, let's get it replaced"
Anyone reading this, thinking this statement is hyperbole, would be wrong. A good chunk of your research data sits on hardware that was (reluctantly) taken out of a bin. FUN TIMES!!
> "Seriously, we've had to get stuff out of the bin in order to keep the service running, let's get it replaced"
An ISP used to have an email-to-fax gateway, which was literally something like a P200 with an ISA fax card plugged into it. As time passed and usage dwindled, the operations team kept asking if they could decommission this increasingly aging hunk of junk, and pointing out that if it went kaboom, there would be no way to replace it, short of going on an Ebay hunt.
Sadly, upper management insisted on keeping all legacy products active, so long as they still had customers, regardless of the cost and/or risks associated with keeping that product active. Gotta keep those numbers up!
Let me tell you, that one is an international phenomenon.
And I'm sure my previous company is definitely enjoying the quintuple rise in costs a little birdie told me about after we went through exactly all that in slow motion over about 7 years and eventually got outsourced.
Also, anyone who thinks cloud is always cheaper for provisioning steady-state infrastructure is an utter moron. Scale-out and dynamic loads, maybe, depending on if you do your sums and plan it properly.
Otherwise, if you don't have the capacity/real estate to locate your servers on-site, renting space in a bit-barn near your premises is generally much more cost-effective.
I'm hoping that at some point those shiny MBA courses start to implement a little snippet of knowledge on how consultancies work: especially the big ones are always focused on the NEXT sell. This means they'll never completely solve a problem, or are happy help you talking yourself into trouble because that pretty much guarantees the next assignment to then half-fix the problem they knew that was coming.
So no, not surprised. Just feeling very sorry for the Uni that it is taking that route :(.
Please do not believe the spin being presented. As a member of IS who witnesses the 'the belly of the beast' on a daily basis I assure you that the failings of IS over the past 10 years have been down to an insidious creep of corporatism, cronyism and bureaucracy within IS. We are too 'top heavy', too many people in suits talking about problems rather than fixing them.
Technical staff are excluded from conversations because they present the reality of the situation to the 'pipedreamers', reality isn't fun or sexy and doesn't look good on a C.V. so they disregard the good advice. Look at the turnover rate for the 'suited types', they stay for 3 or 4 years ... fail catastrophically and then move on to another well-paid job.
It's always the long-term staff who have to bear the burden and fix the failures of these opportunists.The most insulting thing is that the very same people that 'go the extra mile', the people that fight back against the rot within IS are the very same people that will be let go. Cut out the healthy flesh and leave the festering tumour. Senior Management need to be warned that staff in IS who genuinely CARE about education, who CARE about our students and CARE about our academic staff ... those IS staff will fight this.
If outsourcing provided the University with a better IT service at a lower cost then .... OUTSOURCE US. In reality the problem festers much deeper, I've witnessed outsourcing and it fails in complex businesses like ours.
Too many cooks ... you spend more time explaining the problem to your workers vs their ability to fix the problem.
Outsourcing does NOT work for complex organisations. Cut the rot ...