back to article AI drug algorithms can be flipped to invent bioweapons

AI algorithms designed to generate therapeutic drugs can be easily repurposed to invent lethal biochemical weapons, a US startup has warned. Experts have sounded alarm bells over the potential for machine-learning systems to be used for good and bad. Computer-vision tools can create digital art or deepfakes. Language models …

  1. The Man Who Fell To Earth Silver badge

    Why is anyone surprised by this?

    Seems pretty obvious. In Physics, we call such things when they happen accidentally a "Sign Error". When on purpose, a Parity Change.

    1. MrMerrymaker

      Re: Why is anyone surprised by this?

      People love a good Skynet story.

      Like AI in the full meaning even exists. True autonomy doesn't exist in man made systems.

      Humans could come along and make this a bad deal.

      But then humans can make cars killing machines. Yet again, it's alllllll about us.

      1. William Towle

        Re: Why is anyone surprised by this?

        > People love a good Skynet story.


        It's not like data-driven backdoors that can interfere with the processing pipeline exist.

        "...Aw c**p"

  2. Arthur the cat Silver badge

    One good thing

    is that, like making a Plutonium based nuke, trying to synthesize new nerve agents for terrorism is probably going to prove more lethal to the perpetrators than the intended victims most of the time, and state level actors don't gain anything above what they have already.

    1. Captain Hogwash

      Re: state level actors don't gain anything above what they have already.

      Yet development continues.

    2. Paul Hovnanian Silver badge

      Re: One good thing

      "state level actors don't gain anything above what they have already."

      So we should leave this in the hands of private enterprise? Maybe in a startup in someone's garage?

      1. SCP

        Re: One good thing

        "So we should leave this in the hands of private enterprise?"

        Could you clarify what you mean by "this".

        I would be more than happy for TPTB to take legal action against those making credible attempts to develop and produce [in this case] toxic agents with the intent to cause harm. This would seem to be covered by the sort of anti-terrorism legislation widely adopted by many countries.

        I would be unhappy for TPTB to seek to ban the use of AI technology from general access.

        Unfortunately far too many government officials seem keen on the latter sort of approach.

        The original article seems more like the result of something intended to generate publicity for the outfit behind the "research" than any genuine concerns for a new security threat or a breakthrough development in the use of AI.

        "Experts have sounded ..." pah! More like "3rd rate self-publicists". Sheesh - the Vulture definitely playing to its red-top banner here (or is the whole article a troll - in which case 'Well played Katyanna')! Perhaps write an article on the deady poisons that can be obtained from common plants.

  3. This post has been deleted by its author

  4. MrMerrymaker


    Don't do this

    Don't use em?

    1. Anonymous Coward
      Anonymous Coward


      You cannot do that. It's like an arms race, the country that has higher morals and says no will be left with a disadvantage. You always need to be deadlyer so that deterrence can work

      1. Tom 7 Silver badge

        Re: No

        Thats really working at the moment!

    2. Anonymous Coward
      Anonymous Coward


      You never want to do this. Countries with higher morals in R&D will always be left with a disadvantage to those who don't unfortunately. You always need to be more deadly (but don't use). Only that way deterrance works.

      1. W.S.Gosset Silver badge

        Re: No


        "We should BAN nuclear weapons!!"

        "What a good idea! Hey, while we're at it, we should ban murder, too. That'll stop it once and for all."

      2. ian 22

        Re: No

        As that old imperialist Winston Churchill said, "Nations have no friends, only interests." Substitute 'morals' for 'friends', and that quote will still be right.

  5. Ken Hagan Gold badge

    The world is not (and never has been) short of ways for people to kill other people. The risk is only if a new technology makes it easier for people who want to kill but were previously unable to do so (at least, on the scale they desired). Complex chemical synthesis sounds like a fair barrier to entry, given that most of the population struggle with home cooking.

  6. Bitsminer Bronze badge

    "I want [to synthesize] something that does not use [anything] on the watch list"

    That is the more frightening scenario.

    There are any number of molecules-as-a-service shops that have automated the synthesis of substances to order.

    They are all well aware of precursor watch lists. They may not have the (computational) capacity to detect a workaround in a customer's order.


    1. W.S.Gosset Silver badge

      Re: "I want [to synthesize] something that does not use [anything] on the watch list"

      I've read newspaper reports that the (illegal recreational) drugs-trade chemists have been doing this for some time. Bit of an arms race going on, between the chemists and the regulators playing catch-up.

      1. Anonymous Coward
        Anonymous Coward

        Re: "I want [to synthesize] something that does not use [anything] on the watch list"

        Like the movie Formula 51?

      2. Korev Silver badge

        Re: "I want [to synthesize] something that does not use [anything] on the watch list"

        I've read newspaper reports that the (illegal recreational) drugs-trade chemists have been doing this for some time. Bit of an arms race going on, between the chemists and the regulators playing catch-up.

        This is why countries now make entire classes of drugs illegal rather than specific molecules. Drug companies even buy software to make sure their chemists don't synthesise anything naughty.

        It'd be fun to see it contested in court as I'm pretty sure judges and juries have very little understanding of chemical fingerprints, Tanimoto distances etc.

    2. bombastic bob Silver badge

      Re: "I want [to synthesize] something that does not use [anything] on the watch list"

      wait until the AI spits out a recipe for making the molecules, and/or the equivalent of an organic chemistry C&C machine.

      Then someone accidentally leaves it on "auto" and the next thing you know...

      Another thing that came to mind while reading the article:

      D O _ N O _ E V I L

      (that goes double ++ for the medical and pharmaceutical professions)

  7. Anonymous Coward
    Anonymous Coward

    Don't let an AI control the synthesis of predicted-deadly neurotoxins :<

  8. W.S.Gosset Silver badge

    Genetic bioweapons too

    The core tools for doing this with DNA/RNA have been around for a while, too. Just need to drop an AI on top to run them.

    Potentially related item of interest: coupla years ago, a major vaccine researcher (Petrovsky in South Australia's Flinders U) ran the initial variant of Covid-19 through the standard tools to find out in the standard manner which animals were the most likely source of it. Viruses will bind most strongly to the originating species & related. Because that's how it evolved.

    It bound very strongly to humans, and only weakly to any other animal.

    I've recently read but haven't double-checked: according to the same tools, the virus China offered from its research database as its closest relative, ratg13, can't infect bats.

    1. W.S.Gosset Silver badge

      Re: Genetic bioweapons too

      This is worth a read for (A) an indication of the level of tools casually available, and (b) the implications for genetic bioweapon research. Author has published in AI magazines and is "a drug developer currently working on a rejuvenating gene therapy using the approach of partial reprogramming". eg:

      “Infectious clone technology” stands for creating live synthetic viral clones. Considering the heights of user friendliness and automation that genetic engineering tools have attained, creating a synthetic [virus] via the above methodology would be in reach of even a grad student.


      [COTS Commercial Off-The-Shelf virus-creator, in quote from Beijing study:]

      > Briefly, plasmid with the furin-S2′ site was generated using the Seamless Assembly kit (Invitrogen, Carlsbad, CA, USA) and transfected into CV-1 cells infected by vaccinia virus containing the genome of YN-ΔS-GPT.


      The pace of progress in genetic engineering is astounding. Here is a description of the above Seamless Assembly kit:

      > The GeneArt® Seamless Cloning and Assembly Kit enables the simultaneous and directional cloning of 1 to 4 PCR fragments, consisting of any sequence, into any linearized vector, in a single 30-minute room temperature reaction. The kit contains everything required for the assembly of DNA fragments, and their transformation into E. coli for selection and growth of recombinant vectors.

      > • Speed and Ease — Clone up to 4 DNA fragments, with sequence of your choice, simultaneously in a single vector (up to 13 Kb); no restriction digestion, ligation or recombination sites required

      > • Precision and Efficiency — Designed to let you clone what you want, where you want, in the orientation you want, and achieve up to 90% correct clones with no extra sequences left behind

      > • Vector Flexibility — Use our linear vector or a vector of your choice

      > • Free Tools — Design DNA oligos and more with our free web-based interface that walks you step-by-step through your project

      > • Diverse Applications — Streamline many synthetic biology and molecular biology techniques through the rapid combination, addition, deletion, or exchange of DNA segments

      Up to 4 DNA fragments can be joined in a desired orientation in about half an hour, without having to deal with restriction enzymes or ligation. Once you’re done, quickly “upload” your creation into E. coli to propagate the resulting design. Easy-peasy!

    2. low_resolution_foxxes Silver badge

      Re: Genetic bioweapons too

      Is this statement misleading, considering the binding nature of covid?

      It has been suggested that the covid binding receptor (ACE-2) has the appearance of being artificial. In the sense that, it is a non-native genetic design, with the theoretical and practical suggestion being that the ACE2 binding receptor was genetically modified.

      As such, it has been proposed that it was modified to become more human like, since the ACE2 receptor has been demonstrated elsewhere to be a horrendous human pathogen

      1. Richard 12 Silver badge

        Re: Genetic bioweapons too

        Long proven false. Conspiracy theory by those who don't understand the basics.

        It's exactly the same argument as claiming that octopus eyes are man-made.

        Roll the dice enough times and evolution finds a way. Viruses roll the dice very fast. Usually they come up snake-eyes, but occasionally - they don't.

        1. W.S.Gosset Silver badge

          Re: Genetic bioweapons too

          False. Assuming you live in the real world.

          It looks like you both mean the furin cleavage site.

          It does exist in nature in coronaviruses. But in a very distant branch.

          At the measured hence known mutation rates of both, it would take _at least_ 10,000 years to race full-speed across that gap, or rather, chasm. To create a furin site in _precisely_ the right place. All without having any interstitial form appearing anywhere on the planet ever.

          To put it another way: you're pushing Disinformation.

          1. W.S.Gosset Silver badge

            Re: Genetic bioweapons too

            A truly surreal aspect of the source of that disinformation campaign, which you're enthusiastically participating in, is that one of its participants actually documented it in his own book.

            Documented the people, their professional assessment that the virus was man-made, and their emergency discussion and decision to cover it up. Including his own role and participation in both the decision and its subsequent execution. In his own book, which he heavily publicised. Quite surreal.

            Chairman of the Wellcome Trust, btw. Kristian Andersen and Fauci were two of the others he documented, off the top of my head.

            1. W.S.Gosset Silver badge

              Re: Genetic bioweapons too

              I beg your pardon: Director, the actual head who actually runs it, not Chairman. Jeremy Farrar. (The Wellcome Trust's "chairman" is just a sinecure PR figurehead title.)

              And another of the discussion&decision bods was Professor Eddie Holmes

              --"a world-leading authority on viral evolution", per the Australian Institute of Science's official bio, who "has revealed the fundamental processes of microbial evolution, determined the origin and evolution of major human pathogens including influenza, dengue and the AIDS virus, and shown how genetic and epidemiological data can be combined to radically improve our understanding of disease evolution"

              -- who stated the virus was at least 80% certain to be man-made. Andersen said about the same IIRC.

              And --"interestingly"-- they were both panicking about it and doing the urgent ring-round. Farrar wrote that Andersen needed several drinks to try to calm down after he inspected the virus, Holmes said "This is bad", etc. I can't remember exactly how he described Fauci & the others but similar reaction.

              An observation:

              People who are uninvolved in a third party's illicit & dangerous activity have no reason to panic if it comes to light. Do YOU panic if your work colleague's caught making meth at home? Stupid of him, but that's for him to deal with. No skin off your nose -- it's not your problem.

              Panicking would only make sense if they believed they were responsible. Were involved.

              Likewise, conducting an aggressive steamrolling public campaign to promulgate a public belief the exact opposite of their documented private belief.

              Here's the book if people want to read it themselves: "Spike: The Virus vs The People" by Jeremy Farrar, Director, Wellcome Trust.

  9. Simian Surprise

    > having model APIs where you can cut off access if it looks like some bad actors are trying to use your toxicity models for these sorts of various purposes would be a step [towards harm reduction].

    Great idea! We can analyze the model usage to see if it looks like what someone would be using to create toxic compounds. All we need to do is train a machine-learning model...

    1. Danny 2 Silver badge

      >model APIs where you can cut off access

      Clippy - It looks like you are developing a nerve agent. Would you like some help?

      "We didn't do this but...We didn't want to go that extra step. But..."

      But they obviously seriously considered it!

      1. Richard 12 Silver badge

        Yes, of course they did.

        It's their job - what further steps would an evildoer take beyond the ones we're authorised to do?

  10. marcellothearcane

    The thought [of misuse] had never previously struck us


  11. AVR

    Chemicals are scary...

    ...but guns and explosives are far more efficient means of killing. Which is why armies use many more of those than chemical weapons, even if they're willing to invite outside fury.

    For private murder using an unknown chemical would probably point to the chemist who could synthesise new nerve agents, it'd cut down the suspect list by orders of magnitude.

    1. Tom 7 Silver badge

      Re: Chemicals are scary...

      Its like the screaming about thermobaric bombs because of the vacuum they create. Any bomb of that size does the same thing.

      1. A.P. Veening Silver badge

        Re: Chemicals are scary...

        Its like the screaming about thermobaric bombs because of the vacuum they create. Any bomb of that size does the same thing.

        True, but there are relatively few conventional bombs of that size

    2. Anonymous Coward
      Anonymous Coward

      Re: Chemicals are scary...

      Whilst comparing the effectiveness of different methods of killing can be seen as macabre it also serves as a means of placing into context the realities of different threats and so balancing the "fear factor".

      In this respect it can be worth comparing the Tokyo Subway Sarin attack and the Manchester Arena Attack - there are entries on Wikipedia.

      In both cases there were significant losses of life, severe injuries to many more, and the trauma caused to those whose loved ones were affected; all perpetrated as a result of deranged ideologies. Take a moment to remember these people and others who have suffered (and are facing) these and similar atrocities.

      But in seeking to continue living whilst taking sensible precautions it is worth taking note of the resources deployed to carry out the Tokyo attack compared with those of the Manchester Arena bomber (see the Wikipedia entries). Generally, mounting a serious chemical attack is not a simple undertaking and knowing what is deadly is not the same as having the means to carry out an attack. The fear of such an attack is much greater than its reality (state level actions excepted - more so for some states).

      1. Jellied Eel Silver badge

        Re: Chemicals are scary...

        There was a chemical weapon attack today. Well, an attack on a chemical plant that resulted in an ammonia leak. Or as you say, the attack in Japan, or attacks in Syria, Iraq etc. So evidence that terrorists are actively developing and using chemical weapons.

        Or there's the designer drugs trade, which has been busy synthesising plant food & bath salts to work around legislation or restrictions on precursors. Or there's novichok, which can only be created in Russian state labs. Apparently designed so it can be created from easily available chemicals. Plus there's some incentive to come up with new formula novichok because it it hasn't been very effective.

        Sadly, criminals do seem inclined to create and use CBW though.

        1. Anonymous Coward
          Anonymous Coward

          Re: Chemicals are scary...

          So not a chemical attack in common parlance, and in particular not relying on some novel AI generated toxin.

          Syria and Iraq were chiefly state level actions (hence the exception for states). The wikipedia write up on the Tokyo attack indicates the difficulties the sect had in developing and deploying an effective attack and they seemed highly motivated and resourced. Their cost-effective choice was Sarin. I would not be surprised if terrorists are looking at chemical (and biological) attacks - they invoke a lot of terror. But they are difficult to implement compared to "traditional" bombs.

          The drugs trade certainly has some talented chemists - but the balance for terrorists would seem to favour producing and marketing recreational drugs to fund "traditional" bombs or acquiring some high-tech guided munitions. The war in Ukraine is demonstrating the effectiveness of drones and man portable missile systems - and these have been a concern for security services for some time.

          Novichok is interesting - but would seem to present many of the same (if not greater) challenges as Sarin production did for the Tokyo sect. For now it looks like the only threat of Novichok comes from the state known to have it and has been seen to be willing to use it in unsubtle assassinations carried out in a NATO country. Other states have used VX to assassinate troublesome siblings of the dear leader.

          The fact that a well-resourced state actor with access to very powerful toxic agents and highly trained operatives thinks they are not effective enough suggests that there are considerable challenges in mounting an attack. Elsewhere mass attacks on civilian populations have largely relied on WWI level technology (Chlorine, Mustard Gas) - still very nasty stuff.

          I am not sure why you think criminals [as opposed to terrorists] would be interested in CBW - where is the value in it for them? As a commodity it seems higher risk than conventional gun-running and with a smaller market. As something for their own use it seems to serve no useful purpose, carries significant risk of self-inhumation, and is difficult to acquire/produce/store/deploy.

          The bio-weapon aspect is another area of concern - but again the Tokyo sect's attempts to use botulism illustrate this is not an easy attack to make - and there are plenty of rather nasty pathogens around.

          It is right that security services remain watchful on these things and continue their interdiction activities, but I think the AI angle covered by the original article is an irrelevance.

  12. Korev Silver badge

    he dual-use experiment was carried out for research purposes, and a paper on the matter was published in Nature this month.

    Please bare in mind that Nature and Nature Machine Intelligence are separate journals; the former is a highly prestigious general journal, the latter is a very specialist one.


    Can be used for good?

    I'm a farmer, and I can tell you we need some more toxic molecules for Waterhemp. Stuff is getting out of control, and the old chemistries aren't working anymore.

  14. John69

    Bioweapons or chemical weapons?

    Usually bioweapons means pathogens, this is making "ordinary" chemical weapons.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2022