back to article British Library pushes the cloud button, says legacy IT estate cause of hefty rebuild

The British Library says legacy IT is the overwhelming factor delaying efforts to recover from the Rhysida Windows ransomware attack in late 2023. Rhysida broke into the British Library in October last year, making off with 600GB worth of data and, crucially, destroying many of its servers which are now in the process of being …

  1. t245t Silver badge
    Joke

    Legacy IT overwhelming factor delaying ransom recover efforts :o

    > The British Library says legacy IT is the overwhelming factor delaying efforts to recover from the Rhysida ransomware attack in late 2023.

    Wha', has the software atrophied (worn out) from being stored too long in the janitors cupboard.

    1. Anonymous Coward
      Anonymous Coward

      Re: Legacy IT overwhelming factor delaying ransom recover efforts :o

      Exactly, if is out of support it was out of support before the attack, if they had insecure ETL and an overall badly designed environment then the goons who did that are unlikely to do better in the cloud... the only thing which will change is that it will now be more expensive :-)

  2. Anonymous Coward
    Anonymous Coward

    Somebody else's computer

    So now they're following it up with more stupidity.

    There is nothing dumber than putting your critical infrastructure under the control of somebody else. You're now a paypig, exploitable for as much money as your new corporate overlord wants to extract from you.

    I'm gonna laugh so hard when one of the 'cloud' providers gets taken out by the same ransomware gangs.

    1. Brewster's Angle Grinder Silver badge

      The cloud: a defence against bean counters.

      "...many of its systems can't be restored due to their age. They will either no longer work on the fresh infrastructure or they simply can't get any vendor support after going end of life (EOL)."

      That, however, won't be allowed to happen in the cloud. They won't be allowed to leave systems rotting ("because, funds...") and then find they can't support them when they need to restore.That problem will have to be dealt with when the vendor or cloud provider pulls support.

    2. Mike 137 Silver badge

      Re: Somebody else's computer

      "when one of the 'cloud' providers gets taken out"

      More likely, the cloud client gets taken out via a wide open browser on its user base (actually, very likely indeed -- remember "somebody clicked on a link"?). The general assumption that the cloud is "more secure" is a gross misunderstanding. The cloud provider's infrastructure may be well secured, but a client's security effectively remains its own responsibility. There may be some support tools to help, but they (and other controls) have to be deployed with understanding to achieve adequate security. The big snag is that once you've sacked your IT staff because you've gone into the cloud, you've got nobody left who can optimise those controls. (Yes, that's an extreme scenario, but it does exemplify a real trend).

      1. Martin M

        Re: Somebody else's computer

        “ The big snag is that once you've sacked your IT staff because you've gone into the cloud, you've got nobody left who can optimise those controls”

        Agreed, but the skillsets required to run a legacy server estate and those to run a client estate are rather different. Doing the latter doesn’t mean you have to do the former.

      2. the future is back!

        Re: Somebody else's computer not?

        Personally I am going NAS and our household has a mere repository of ~2K books and many photos and vids. Cloud is convenient but a raid NAS looks better than that.

    3. Martin M

      Re: Somebody else's computer

      “ You're now a paypig”

      The British Library seems the poster child for an organisation that would have been much better off on a cloud provider,. Yes, there’s lock-in, but better than the alternative.

      “ I'm gonna laugh so hard when one of the 'cloud' providers gets taken out by the same ransomware gangs.”

      None of us are going to be laughing if a major cloud gets taken out, because quite large chunks of many large societies will no longer be working. Reckon it it would take more than the average ransomware gang though.

      Which would no doubt be met by a serious response, which makes it less likely. Hey, it worked for nukes, even if was a bit dicey.

  3. Anonymous Coward
    Anonymous Coward

    So now they're paying two ransoms? One from the ransomware gang and another (legitimate) ransom from a cloud (feudal lord) provider. Surely having ones own server is still cheaper because regardless someone has to maintain the server physical or virtual.

    I'll leave this link here.

    https://world.hey.com/dhh/the-big-cloud-exit-faq-20274010

    1. Dave Schofield

      >Surely having ones own server is still cheaper because regardless someone has to maintain the server physical or virtual.

      That *depends*. A lot.

      An organization buying say 100, or 1000+, servers is a big CapEx outlay in one go - even if you can buy them in sufficient quantities - then you have to rack, power, and get the other stuff sorted around backups. Whereas, provisioning the same number of cloud servers is a smaller OpEx outlay and the other services are already sorted. You just provision and pay.

      Over time, the OpEx total will probably be higher than the CapEx but its often easier to get approval for a lower operational spend than it is to buy the servers - and much quicker.

      1. Anonymous Coward
        Anonymous Coward

        You make a good point however you missed some key costs mainly related to staff. You're going to need someone to secure that cloud, properly cost that cloud, determine which cloud is best (instant access/archived), decide on the optimum connection, upgrade all your software to use said cloud, test the software. These costs aren't free and sure you will have existing staff that could potentially do it but you rather risk that or bring in experts for some of them? As it stands there are 3 main cloud players with no incentive in the long term to keep it cheap as once people move to a cloud it's nearly impossible cost wise to move to another especially for a business like this with this amount of data. Even coming out of the cloud can be extortionate.

        With all that in mind I would counter that on premises in this case would be the wiser option. I'm not saying you do but I have no idea why some people have this idea that you can just cloud it and it all just magically works and is secure.

      2. Roland6 Silver badge

        But cloud is just pushing that CapEx expense onto someone else, who will charge the cost of money plus some on top of everything else for the customers OpEx subscription price.

        Just having this problem with a client, because they failed to purchase new kit the last two plus years they now have a much larger CapEx to pay out, ie. They are looking at having to purchase 400 new servers this year rather than 100…

        1. Doctor Syntax Silver badge

          HMG prefers OpEx. CapEx means a big number that appears in public borrowing figures, gets looked at by the PAC and features in headlines in the Daily Wail. OpEx goes under the radar. The fact that it ends up costing more doesn't really matter to them because most of it it falls on the next government and the one after that and the one after that... They'll not all be the same party and the ministers who set it up will have moved on anyway. See also PFI.

      3. theblackhand

        Re:Cloud OpEx versus onprem CapEx

        The story hints that there has been significant underinvestment in the past and that there is a skills gap for cloud services in the current IT teams.

        That translates into CapEx being difficult to obtain but once it is obtained it will likely be eked out over an extended operational lifetime.

        And on the cloud side, a lack of skills often results in a failure to update and decommission older systems which results in spiralling OpEx costs without ever addressing the funding gaps in skills and legacy systems.

        So I'm not confident that we would see any medium to long term benefit or difference for either approach or a hybrid of the two if the underlying issues are not addressed

    2. Roland6 Silver badge

      > So now they're paying two ransoms?

      Three, software licensing for cloud verses on-prem licensing…

  4. shazapont
    Pint

    Legacy is here to stay... Live with it...

    "Legacy software" or more generall "Legacy IT" is usually discussed as if it were some anomalous hangover from the bad ol' days, from old, undesirable and badly regarded projects, or from when things "weren't done properly."

    Well, that's clearly wrong... Legacy is and always will be a fact of every project. If not now, then sooner or later.

    It's unfortunate that Legacy software is most often mentioned in the same statements that tells us about the bright new future ahead, with no more Legacy software to hold us back, to drag down performance and to remove the need for software maintenance.

    Let's help inject some realism into these discussions and remind everyone that focussing on the good news and the bright future that lies ahead is no way to run projects that we know will have problems. We know that. That's life...

    I hope they don't imagine a problem-free follow-up and then forget any lessons learned... we don't learn from the good news, so relish the bad news...

    1. Pigeon Post

      Re: Legacy is here to stay... Live with it...

      This.

      A Gartner researcher I once knew cheerfully reminded people that projects currently in delivery are 'legacy' by definition. He followed that up by saying "Another definition of legacy software: the programmer is dead or should be".

      Because of 'agendas', Manglement conveniently forget the 'ilities' - usability, security, scalability, availability/reliability, manageability/maintainability, recoverability, and so on. All significant parts of total cost of ownership - ignore at your peril.

      1. Vometia has insomnia. Again. Silver badge

        Re: Legacy is here to stay... Live with it...

        Securility, surely.

        1. Pigeon Post

          Re: Legacy is here to stay... Live with it...

          I bear you no il my friend. :)

          And btw this pigeon's name's not Shirley...

      2. Doctor Syntax Silver badge

        Re: Legacy is here to stay... Live with it...

        projects currently in delivery are 'legacy' by definition

        The legacy stuff is that which is running the business that brings in the money that pays for all the new shiny being developed which might or might not get delivered.

        That "might not" bit is worth reflecting on. The legacy was put together by folk who were able to make it work. Not all projects achieve that. Will the new shiny work out well enough to become legacy?

  5. Stuart Castle Silver badge

    The cloud may enable them to simplify their systems, which can help security. Guess what? So does investing in new on prem systems. The Cloud isn't inherently more secure than on prem (on the contrary, it may be less secure). What enables you to secure it is the systems analysis you should do when moving to any new system, on prem or cloud.

  6. may_i

    First question to ask their new cloud overlord:

    When you make backups of our data, are those backups immutable?

    Lest they end up losing everything a second time. Yes, TietoEvry, I'm looking at you!

  7. 43300 Silver badge

    "destroying many of its servers which are now in the process of being replaced."

    How does ransomware 'destroy' a server such that it needs to be replaced?

    1. devin3782

      Sometimes malware/ransomware can write itself into firmwares of the devices in the server or the boot loader or indeed the bios, we have Sony were the first to do this with their rootkits on music CD's so we really have them to thank, although i'm sure others would have come up with it sooner or later. As far as I know there's no way to tell for sure especially as a lot of firmware isn't available so you can compare and re-flash as needed or indeed fix it.

      1. 43300 Silver badge

        It ought to be possible to over-write the firmware with a known-good version, though!

        1. John Brown (no body) Silver badge

          Sometimes, maybe, possibly. Unless it's socketed (highly unlikely these days), or you can de-solder it and then reprogramme it, odds are you need to power it on to update the firmware, at which point it may be too late in the boot process to be sure your f/w update did it's job or just appeared to do it's job. The Intel Management Engine and similar are basically a whole other computer on the system board that's already running before any other boot device or on board BIOS/UEFI environment. ANd the age of the device will affect how much time and money can be put into recovering it before a new replacement is cheaper. Who even knows what might be happening in the firmware of other devices inside the case. I saw a Youtube video of someone installing and running Linux on an HDD PCB. If you can run an entire OS on the circuit board of a spinning disk, what can do with an SSD?

          And then, in this case at least, probably in most orgs IT system, it's an "organic" system that has grown over the years with various h/w and s/w of various vintages, some of which can only interact because of custom s/w, h/w and or bodges leading to "legacy workflows" that can't happen on replacement version of h/w & s/w, much of which may be precisely the reason the entire setup was vulnerable in the first place. But getting the budget to rip everything out and replace it with current kit is an immensely hard sell until something like a ransomware high-jacking happens. If the system is running ok, telling the Board that it needs to be replaced due to nebulous potential threats is an uphill battle.

      2. WaveSynthBeep

        An unlamented cloud provider did that by reflashing thousands of elderly servers with a management firmware image that failed to boot. It might have been fixable by individually desoldering and rewriting the flashes, but a big task to do that when you've just bricked thousands of machines. Probably cheaper to scrap them and start again.

      3. Dan 55 Silver badge

        I would expect most servers to use VMs, even in datacenters.

        1. Anonymous Coward
          Anonymous Coward

          VMs don't matter

          if they're someone elses hosts, it's not your problem, but your hosting company is gonna shut you off and be in full on defensive mode. If they're yours, and there's any route out to them through guest services or a route to the management vlan or other they'll be assumed compromised, and probably are.

      4. Doctor Syntax Silver badge

        It may well be the case but there's something seriously wrong with a vendor that lets malware overwrite the firmware but doesn't enable the owner to fix it, whether by failure to provide good images or any other reason.

    2. Paul Crawford Silver badge

      I suspect what really happened is old servers had their OS wiped and now you can't get a new image to boot (due to lack of working CD drive, no usable USB boot support, etc) or that new OS just don't support the hardware. So not "destroyed" in a real sense, but "rendered unusable for re-use" in practice.

      We have got rid of old Dell servers as it was too much convoluted steps to try and boot a new USB stick with Linux on it, not to mention your work might be worthless if the hardware actually fails in 1-2 years or less.

  8. Nate Amsden

    where are they going to get the money

    Redoing all of that is going to cost some serious cash and seems like they were/are very short of funds. Guessing that so much old stuff was cheap to run as you don't have support and updates anymore. 3rd party hardware support on servers is probably 95% cheaper than full OEM hardware/software support. Legacy also likely means not many changes are happening as well perhaps out of fear of breaking stuff. Eventually it'll all fail of course (the legacy stuff). So those savings don't last forever.

    Of course many on el reg have seen that in most cases cloud costs more. Certainly are situations where it makes sense. But it doesn't sound like they have a plan(or wasn't in the article) where the money will come from.(Or how much they expect to need/how long it will take). Seems like someone who doesn't know anything just says let's go cloud! Have dealt with several people like that over the years that don't know anything. Then are caught unprepared for the reality.

    1. 43300 Silver badge

      Re: where are they going to get the money

      "Seems like someone who doesn't know anything just says let's go cloud! Have dealt with several people like that over the years that don't know anything."

      The answer is "The Cloud"!

      When asked what problem they are trying to solve and have they considered the pros and cons of the various options, a blank look will probably be the result...

    2. Doctor Syntax Silver badge

      Re: where are they going to get the money

      "seems like they were/are very short of funds"

      There's never enough funds to do things properly the first time. There's always funds to pick up the pieces. The first is "nice to have" (translates as "you're not having it"), the second is "have to have" (translates as "get it ASAP").

    3. John Brown (no body) Silver badge

      Re: where are they going to get the money

      "But it doesn't sound like they have a plan(or wasn't in the article) where the money will come from.(Or how much they expect to need/how long it will take)."

      While technically a charity, the vast majority of the funding is via Government grant and it has legal status as a "deposit library", not to mention it's national and international status. They will have to go cap in hand, with a business plan, and ask for more/emergency funding. But as a "national asset", they will get it. They may have to fight for what they want as opposed to what they are offered, but they will get it. It may even be part of the reason for going "cloud", since the up front costs are lower than re-building an on-prem data centre.

  9. Tubz Silver badge

    They need funds, divert some from overseas aid that we know gets into the hands of dictators and terrorist groups!

    Whatever your personal feelings over Trump, his America First, is now being echoed by a lot of other countries population wishing to put home-grown issues at the top of priorities and rightly so!

    1. Anonymous Coward
      Anonymous Coward

      No you tax the mega riches passive incomes amoungst other measures.

  10. Anonymous Coward
    Anonymous Coward

    Shocked I tell ya !

    "Now, there is currently a belief that the team may not be sufficient in size to meet the demands of the rebuild program, and the report alludes to a potential issue with the way the Library pays for its talent."

    Really ? Years of staff reduction since everything was running great and kit was literally rotting, thus 0 upgrade, and now there is a big refresh, post attack, there is a need of competent and paid staff ?

    Colour me shocked !

    1. theOtherJT Silver badge

      Re: Shocked I tell ya !

      ...and I'm certain that going to the cloud will help with that.

      On prem we ran our estate with 4 people. In the cloud we now need 11, and we have had to dump a ton of responsibility onto individual businesses units to manage their own infra and those new cloud people are getting paid more than the old sysadmins were.

      Managing cloud is different. Sometimes better, sometimes worse, but almost always different and paying for people who know how to do it isn't cheap.

      1. 43300 Silver badge

        Re: Shocked I tell ya !

        I've also noticed that in consultancies, where it used to be the case that the senior consultants would turn their hand to lots of things, these days there's far more specialism - there will be one team for on-prem server setups, one for Azure, one for M365, etc. Inevitable really given how convoluted all the cloudy stuff is, and the fact that it changes so rapidly that if you aren't looking at it regularly then knowledge will rapidly get out of date.

  11. Anonymous Coward
    Anonymous Coward

    This will be unpopular but that is wrong with the cloud

    Can someone please explain why all the hate for cloud? apart from the fact its someone else's computers not some magical place that some people think it is why the hate?

    The British Library is not an IT organisation so why shouldn't it use someone else's computes, network, colocation abilities.... if they do it all themselves they need two separate offsite locations and all the various connections, power, cooling.. set up with redundancies for both sites, and they need the machines for both sites. Then they need a team of people to set up both sites configure replication, security patching, and then they need another team to physically secure both sites and maintain them. After they have done all that then can then set up what they actually need to enable them to do what they do. and in 3, maybe 5, years they'll need to replace all the machines, connections, cooling....

    If were being honest (I'll say it again the British Library is not an IT company), if they did it all on premises they will be lucky to get one and a half on site locations with just enough power and cooling with no, or very little redundancy, some overly complex and expensive replication software, with support fees so high they're in orbit, and is that's never quite set up right so it doesn't actually work but it will any day now. Security that's just about ok, except for Ruth as she can't see the need for all this MFA stuff and anyway she's so good she'll never be hacked (ok cloud wont do much for this but it does allow IT to say its clouds fault you must have MFA). Patching that just needs to be held off for some important reason, like Steve must be able to use the VBA script he's got that the new patch will break, and he hasn't got time to change the full stop to a comer, and lets not forget the group of servers under Angelo's desk that have been left out of the patching schedule as only Angelo knows they exist. Oh and the replacement of equipment will always be put off as the budget is needed for something else so nothing is ever updated and if something does fail the software is so old it cant run on newer machines

    Physical IT has become like a powerplant its essential to modem life, but you don't need to buy, setup, maintain, and run your own. You buy the output of the powerplant and only buy what you need and if you need more you buy more if you need less you buy less. The powerplant owner has all the expense of setting up, running and maintaining it you don't so charges you more for the power than it costs them to produce. You could run your own powerplant and get power a bit cheaper but most of us don't know how to or can afford to buy, set up, maintain, and run a one, and what do we do with any excess power, or if we don't have enough.

    1. theOtherJT Silver badge

      Re: This will be unpopular but that is wrong with the cloud

      Simple answer, from my perspective: it's outsourcing. Outsourcing always looks cheaper to start with, but ends up costing more long term. Ultimately you either own or rent, and renting is never, ever, cheaper in the long run.

      1. 43300 Silver badge

        Re: This will be unpopular but that is wrong with the cloud

        Usually seems to be at least twice the cost versus on-prem, taking all aspects into account over a five year period (the expected lifetime of a set of on-prem servers).

        1. Anonymous Coward
          Anonymous Coward

          Re: This will be unpopular but that is wrong with the cloud

          My rule of thumb is whatever an on-prem server costs to buy outright, including 5 years' hardware maintenance charges bundled, is approximately the cost of provisioning a similar spec server (IaaS) in public cloud for 1 year. To be fair this is before you start "turning it off when you're not using it", and doing reserved instances and all that fun. If your organisation is capable of that level of operational discipline. Ours sure isn't!

    2. Richard 12 Silver badge

      Re: This will be unpopular but that is wrong with the cloud

      The Library is for archive and retrieval. That's its entire purpose.

      If they put the archive "in the cloud", they've outsourced their core function. Their sole reason for being.

      You never, ever outsource your core functions, because doing so very rapidly means the organisation cannot perform its core functions.

      If a business making widgets outsources the widget making, it doesn't take long before there's no widget makers in the business. Then it's not long before there's no business.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like