back to article Google cloud glitch hits at Beer O'Clock Friday, fix coming Monday

Google's got a problem with its Cloud SQL service – about seven per cent of instances using the service's first-generation code aren't backing up properly. The problem started at 16:54 on Friday afternoon and by 17:34 the company announced it was “forcing” backups “as short-term mitigation.” There's no suggestion data is in …

  1. veti Silver badge

    The point of SLAs is to define exactly what they promise to deliver, and what compensation I get if they don't.

    Provided they can deliver what they promised, why do I care how they do it? They could be writing the whole thing out on vellum with quill pens for all I know. Or care.

    1. Anonymous Coward
      Anonymous Coward

      Agree. This seems like a non-story. I understand why Google would want to, and will, automate the process, but the whole idea is that Google owns the SLA and they can do it however they want to do it. If there is an issue, there is an issue... and financial compensation. If there is no issue, then it is really not anyone's problem or concern how they choose to meet the SLA.

      This is gen1 Cloud SQL, which is, frankly, out of date and only around for people who have not yet moved to gen2, so probably a non issue in any case.

      1. This post has been deleted by its author

      2. Adam 52 Silver badge

        Have you read the standard Google T&C and the SLA? If you're expecting any meaningful compensation for any sort of outage, data breach etc. then you'll be sorely disappointed.

        Supposing Google lose your Cloud SQL backup and you get fined €10 million by the regulator. Google will compensate you about $3000, and it'll cost you thousands in Californian lawyers to get it.

    2. Doctor Syntax Silver badge

      "The point of SLAs is to define exactly what they promise to deliver, and what compensation I get if they don't."

      This is the MBA version. The reality could be that a failed backup leads to a failed business.

      If you're a DBA or sysadmin working for the company that owns the database you know that a failure could lead to the loss of your jobs and those of your colleagues. If a problem happens on a Friday you work over the weekend to fix it, you don't leave it until Monday.

      If you work for a 3rd party provider you know the worst is that it cost your employer whatever the SLA says and nothing else. What's more your employer has already taken the decision to carry the risk themselves or to insure against it and if they don't care enough to tell you to work the weekend why should you care any more?

      1. Anonymous Coward
        Anonymous Coward

        "If you work for a 3rd party provider you know the worst is that it cost your employer whatever the SLA says and nothing else."

        That's not true. If there is some major issue, I doubt the only issue a provider will have is whatever the "SLA says and nothing else." They will probably lose the business, account. The SLA is a minimum. It's not like Google said "we're not doing any back ups until further notice, have a good weekend"... they have a fix/solution in place, they're doing back ups, it's working. It just isn't automated... and, in theory, manual DBA intervention is slightly more risky than automation. It is also what pretty much everyone does on prem, manual or partially manual DBA back ups.

        What the cloud providers do is a step beyond what any on prem provider would do... if Oracle or MSFT doesn't patch something in a timely fashion (just in theory :) and something goes wrong as a result, you are owed nothing and Oracle or MSFT will issue the patch when they are good and ready. They don't own *any* SLA... they are just dropping off code and, if there is an issue, they will fix it when they feel like it.

        That is the part I don't get about people's concerns about cloud SLAs. They are concerned because now you are in the cloud providers' hands and their competency is paramount... but that is the case with on prem today. It's not like people are making their own servers and writing their own databases from the ground up. If there is an issue anywhere, you are basically on the phone with Oracle, IBM, Cisco, EMC, MSFT, etc waiting for them to get a fix out. Same situation as on prem as in cloud, but at least in cloud the provider owns the overall SLA and it is on them.

        1. Anonymous Coward
          Anonymous Coward

          Here's the scenario -

          Customer is concerned that cloud is not 100% automated and manual intervention is required (only on back ups, until next week when it will be automated, on one legacy service). Ergo, they need to keep workload on prem where nearly everything is manual with little automation and bolted together from 10 different providers who all hate each other, patch levels are rarely the most current rev because they often require outages, etc.

          It is like a person who has been drinking all night at a bar reading about how much better self driving cars are in terms of safety than professional car service drivers... so they decide to drive home after eight drinks to avoid the risk of a professional car service driver as a self driving car would be optimal.

  2. ratfox

    Pushing an emergency fix on a Friday evening sounds like a good recipe for disaster. Doing manual backups over the week-end might entail a bit of risk, but probably much less.

    1. Doctor Syntax Silver badge

      "Pushing an emergency fix on a Friday evening sounds like a good recipe for disaster."

      Pushing an emergency fix on a Friday evening and waiting until Monday morning aren't the only alternatives.

    2. Anonymous Coward
      Anonymous Coward

      An emergency fix on any day is a recipe for disaster. Perhaps the fix itself takes just 15 minutes to code, but they still need to do QA, and finally let it propagate to their many DCs. If they are promising (?) the fix to be finished on Monday then it still sounds a bit of a rush job unless the error was an obvious one and can't have an adverse effect on other system parts.

      Google can very well afford a couple well paid techs to identify the root cause and start on the fix even on a Friday evening.

      But as they didn't - you can draw your own conclusions where Google's priorities are.

      1. Anonymous Coward
        Anonymous Coward

        They did get a fix out... nothing was ever broken. They just haven't automated back ups for DBs for a legacy service, gen1 Cloud SQL. The SQL services were never down and always working, back ups were never down and always working, they have not yet automated back ups.

        I would love to see your reaction if your boss asked you to work through the weekend to get DB back up automation working so zero manual intervention is required... even though the DBs are performing perfectly, the back ups are running, this is a nice to have automation piece.

        In other words, Google's legacy SQL environment is degraded to the point where it is working exactly like your on prem DBs which you would consider to be working perfectly, without issue.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like

  • IBM CEO explains why he offloaded Watson Health: Not enough domain expertise
    And not enough customers, Shirley?

    IBM chairman and CEO Arvind Krishna says it offloaded Watson Health this year because it doesn't have the requisite vertical expertise in the healthcare sector.

    Talking at stock market analyst Bernstein's 38th Annual Strategic Decisions Conference, the big boss was asked to outline the context for selling the healthcare data and analytics assets of the business to private equity provider Francisco Partners for $1 billion in January.

    "Watson Health's divestment has got nothing to do with our commitment to AI and tor the Watson Brand," he told the audience. The "Watson brand will be our carrier for AI."

    Continue reading
  • TomTom to chop 10% of workforce, blames automation tech
    Improvements in mapmaking platform to cost 500 employees their future at geolocation tech provider

    TomTom says it is laying off 10 percent of its global workforce due to advances in automation technology and greater use of digital techniques in its mapmaking process.

    The planned cuts will equate to about 500 employees at the Netherlands-based geolocation tech specialist, which was hit hard by the pandemic and remains in recovery mode.

    "Higher levels of automation and the integration of a variety of digital sources will result in fresher and richer maps, with wider coverage," said CEO Harold Goddijn. "These better maps will improve our product offerings and allow us to address a broader market, both in the Automotive and Enterprise businesses."

    Continue reading
  • Fully automated AI networks less than 5 years away, reckons Juniper CEO
    You robot kids, get off my LAN

    AI will completely automate the network within five years, Juniper CEO Rami Rahim boasted during the company’s Global Summit this week.

    “I truly believe that just as there is this need today for a self-driving automobile, the future is around a self-driving network where humans literally have to do nothing,” he said. “It's probably weird for people to hear the CEO of a networking company say that… but that's exactly what we should be wishing for.”

    Rahim believes AI-driven automation is the latest phase in computer networking’s evolution, which began with the rise of TCP/IP and the internet, was accelerated by faster and more efficient silicon, and then made manageable by advances in software.

    Continue reading
  • Worried about being replaced by a robot? Become a physicist
    Scientists develop algorithm that decides if your job can just be an algorithm

    An online index of nearly a thousand jobs may useful in cluing in folks to the automation risk their field of employment faces.

    At the bottom of the list – meaning they're the most likely to be replaced – are meat packers and slaughterhouse workers, while the least likely to see their jobs automated are physicists. In general, those in food services, grounds and maintenance, and the construction industries are most at risk. Those in education, social services, and management least so – but not entirely.

    Scientists at École Polytechnique Fédérale de Lausanne and the University of Lausanne in Switzerland collaborated to draw up the index as they contemplated the potential social impacts of automation. While their list is targeted at individuals, the team said there's other uses for the methodology they developed to create the automation risk scores featured in the index and map career transitions for displaced workers. 

    Continue reading
  • Qlik moves to boost position in crowded automation space during IPO year
    Initial boom in data vizualization, but CEO Mike Capone says it needs to 'play nice' to win in new segment

    Interview It's never a zero-sum game, according to Mike Capone, CEO of data visualization software specialist Qlik. For the sake of the company, he'd better be right, as Qlik is betting its success on a foray into a market with way more competition than the one in which it earned its stripes.

    In January this year, Qlik confidentially filed paperwork with US regulators for an initial public offering return to public markets, roughly six years after it was purchased by private equity investor Thoma Bravo.

    To win over investors it will need to convince them it can make it in the market for automation software, which tech analyst IDC predicts will grow 17 percent each year between 2020 and 2025.

    Continue reading
  • Insteon's vanishing act explained: Smart home biz insolvent, sells off assets
    Acknowledgement of shutdown almost a week after backend powered off

    Smartlabs, Inc, parent of vanished internet of things vendor Insteon, is unable to meet its financial obligations and has assigned its assets to a financial services firm to be sold.

    After recently shutting down the servers supporting its smart home hub app and saying nothing to its customers or partners, the California-based firm on Wednesday evening finally got around to publishing an update to its website.

    The notice explains that in 2017, Smartlabs had financial problems and obtained additional capital and new management to improve its situation. But the emergence of the global pandemic in 2020 and the ensuing supply chain disruption proved too much, prompting the company to seek a buyer in November 2021.

    Continue reading
  • California suggests taking aim at AI-powered hiring software
    Automated HR in the cross-hairs over discrimination law

    A newly proposed amendment to California's hiring discrimination laws would make AI-powered employment decision-making software a source of legal liability. 

    The proposal would make it illegal for businesses and employment agencies to use automated-decision systems to screen out applicants who are considered a protected class by the California Department of Fair Employment and Housing. Broad language, however, means the law could be easily applied to "applications or systems that may only be tangentially related to employment decisions," lawyers Brent Hamilton and Jeffrey Bosley of Davis Wright Tremaine wrote.

    Automated-decision systems and algorithms, both fundamental to the law, are broadly defined in the draft, Hamilton and Bosley said. The lack of specificity means that technologies designed to aid human decision-making in small, subtle ways could end up being lumped together with hiring software, as could third-party vendors who provide the code.

    Continue reading
  • Celonis buys German process miner with Power BI links in $100m deal
    Move follows Microsoft's own tools to find out how things are really done

    Celonis has bought Process Analytics Factory (PAF), a firm that integrates its technology with Microsoft's Power BI platform, in a deal said to be worth $100m.

    The two German companies claim the merger will help users on Microsoft's popular Power Platform benefit from output data from Celonis's process mining, automation, and collaboration tech.

    Process mining is the analysis of application logs to figure out how people actually run processes across multiple applications – as opposed to how management think they run or how they were designed to run. The argument goes that a detailed understanding of processes is a vital element of application strategy and underpins modernization and workarounds like robotic process automation.

    Continue reading
  • ServiceNow jumps into RPA with imminent 'San Diego' release
    Will bring bots to workflow users, perhaps not with the same precision RPA specialists muster

    Workflow specialist ServiceNow has announced a heavy emphasis on robotic process automation (RPA) in the next release of its platform, to get aging applications working with each other in a more user-friendly fashion.

    One analyst said users already invested in workflow would welcome the release, but RPA specialists may have a technical edge.

    ServiceNow has promised the "San Diego" release of its Now Platform will include more modern visual design as well as the RPA capabilities. The latter sits within the Automation Engine, which combines Integration Hub with a new RPA Hub that provides centralized command and a control center to monitor, manage, and deploy digital robots that automate repetitive manual tasks.

    Continue reading
  • Boston Dynamics' latest robot is a warehouse workhorse
    When does this thing get to unionize?

    Robotics company Boston Dynamics is making one of its latest robots more generally commercially available: a mobile, autonomous arm called Stretch.

    Stretch is outfitted with a vacuum gripping arm able to move a wide variety of box types and sizes, up to 50 pounds (≈22.7kg). Its footprint is about that of a warehouse pallet, and it can move around on its own, which Boston Dynamics said makes it a good fit for companies trying to automate without building a whole new factory.

    "Stretch offers logistics providers an easier path to automation by working within existing warehouse spaces and operations, without requiring costly reconfiguration or investments in new fixed infrastructure," Boston Dynamics said this week.

    Continue reading

Biting the hand that feeds IT © 1998–2022