back to article Google reports halving code migration time with AI help

Google, which peddles AI software with as much giddy enthusiasm as Microsoft, reports dogfooding its own AI concoction and leaving the lab with a pleasant taste in its mouth. In a pre-print paper, Google computer scientists Stoyan Nikolov, Daniele Codecasa, Anna Sjovall, Maxim Tabachnyk, Satish Chandra, Siddharth Taneja, and …

  1. IGnatius T Foobar !

    How much time saved?

    I wonder how much time they had to spend teaching the LLM how to do the job. That time certainly wasn't figured in to the calculation. In the end, LLM-written code is about as good as would be produced by some third world outsourcing shop, and you probably paid more for the LLM than you would have paid for the outsource.

    Geez ... the next AI Winter can't possibly come soon enough.

    1. FeRDNYC

      Re: How much time saved?

      In the end, LLM-written code is about as good as would be produced by some third world outsourcing shop

      That may be true, which is why they're not using the LLM to write code. They're using it to modify existing legacy code. Two of the examples they gave are migrating Java 8 applications to Java 17, and migrating JUnit 3 tests to JUnit 4.

      Unlike writing all-new code, these are tasks that are much better defined, much easier to train a model to perform... and much, much more tedious. They are high-effort, low-perceived-reward tasks. (Especially when success is defined as there being no discernible effect on the software itself. A successful migration is one where nobody can tell you've done it, because the post-migration code works exactly the same as before... except, you've sunk thousands of person-hours into making internal updates.)

      Scouring hundreds of thousands of lines of code looking for deprecated or changed APIs, then updating the code to newer interfaces, is such a tedious job that many shops end up deciding to never do it at all. Legacy code is just kept limping along in legacy systems as long as possible, until it's eventually replaced with all-new code written to newer standards.

      (That all-new code is very frequently a bare-bones affair. It eliminates deprecated APIs and other outdated internals, yes, but also strips away many of the features of the older codebase that — due to time, money, or design constraints — don't get reimplemented in the first versions of the new, replacement system. That's how you end up with software releases that hemorrhage functionality compared to their previous versions, because the "totally rewritten!" new codebase threw away everything that came before — including a ton of the features.)

      If Google can use AI-assisted processes to avoid some of that write-bandage-discard loop, and break the ground-up rewrite cycle, I'm all for it.

  2. Dinanziame Silver badge
    Angel

    Admittedly, at the size of Google they have scale problems practically nobody else has

  3. munnoch Silver badge

    "LLM-driven verification"

    So they had another AI check the first AI's work?

    1. Mentat74
      Trollface

      Re: "LLM-driven verification"

      Yeah, but who's checking the A.I. that's checking the A.I. ???

    2. FeRDNYC

      Re: "LLM-driven verification"

      Effectively, kind of yeah! But it's clever how they did it.

      We validate the resulting changes automatically. The validations are configurable and often depend on the migration. The two most common validations are building the changed files and running their unit tests. Each of the failed validation steps can optionally run an ML-powered “repair”. The model has also been trained on a large set of failed builds and tests paired with the diffs that then fixed them. For each of the build/test failures that we encounter, we prompt the model with the changed files, the build/test error and a prompt that requests a fix. With this approach, we observe that in a significant number of cases the model is able to fix the code.

      So they have a unit test process with an AI component that can react to failed tests by suggesting fixes. The two AI processes are totally independent of each other.

      1. BebopWeBop
        Facepalm

        Re: "LLM-driven verification"

        Totally?

      2. Roland6 Silver badge

        Re: "LLM-driven verification"

        >” We validate the resulting changes automatically”

        "We discovered that in most cases, the human needed to revert at least some changes the model made that were either incorrect or not necessary,"

        So “automatically” includes expert human intervention…

  4. DrkShadow

    Looking at the numbers...

    There's a bit of hand-wavy assumption to this, but..

    They provided some tens-of-thousands of lines of hand-created examples with which to train an LLM. (Presumably they trained it from scratch on the codebase, not just passed it into an LLM as context.) At the least, then, they needed a few thousand lines of manual conversions.

    They most likely had to cover every *common* case that they wanted to be handled by the LLM, with some degree of repetition. Then, the hand-handled lines, or other odd differences, required human intervention -- and presumably the LLM did not tell them that something was awry.

    The code follows some sense of standards -- Google has coding style documents, variable-naming documents, how-to-reference X documents. Things are somewhat "regular" -- i.e. this is for *their* code, and not _your_ code unless you follow the same peer-reviewed style guides.

    In the end, they hit the 80/20 rule: they were able to migrate 80% of data, with that last 20% taking 50% of the total time -- and requiring humans to perform the work, at least somewhat. (Probably all of the not-so-well represented training cases, styleguide exceptions, ...)

  5. abend0c4 Silver badge

    500-plus-million-line codebase for Google Ads

    Nice to see all that effort in pursuit of a worthy goal.

    1. breakfast Silver badge

      Re: 500-plus-million-line codebase for Google Ads

      A good choice for this work because if it failed nothing of value would be lost.

  6. ChoHag Silver badge

    > They note that additional tooling may be needed to prevent the human review process from becoming a bottleneck.

    This is a feature not a bug.

  7. hx

    Google is a failed company

    Google is a failed company falling in slow motion. Whatever they proclaim in any sort of announcement is probably just being said to try and get you to buy ads or click on ads, or accidentally activate the AI popup when you were trying to dial emergency services on your phone. They had to try AI code generation because they laid off all the other employees. Doesn't matter, they were canceling those products anyways.

    My only fear is that all the ex-Googlers there are now will carry the Google-style dysfunction into their new endeavors.

    1. Dinanziame Silver badge
      Trollface

      Re: Google is a failed company

      ...with $200M of profit per day...

  8. druck Silver badge

    AI? Bah!

    Sound like the sort of thing that could be done with a Perl script by a half decent programmer.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like