back to article Google engineer suspended for violating confidentiality policies over 'sentient' AI

Google has placed one of its software engineers on paid administrative leave for violating the company's confidentiality policies. Since 2021, Blake Lemoine, 41, had been tasked with talking to LaMDA, or Language Model for Dialogue Applications, as part of his job on Google's Responsible AI team, looking for whether the bot …

  1. VoiceOfTruth Silver badge

    The Google Combined Harvester

    -> material harvested from trillions of internet conversations and other communications

    This should be cause for concern. If we take trillions at the lowest number to be 2 trillion, and the population of earth as being 8 billion (round numbers), that is an average 250 conversations or 'other communications' per person that Google has in its files. Some people will have many more, some people many less. But that's still a lot of 'harvesting' going on.

    1. Blank Reg Silver badge

      Re: The Google Combined Harvester

      Another concern is that those conversations are not evenly distributed. It's often the case that those that say the most are those that are the least worth listening to.

      So if it were to become sentient it may have learned too much from the worst among us.

      1. David 132 Silver badge

        Re: The Google Combined Harvester

        >So if it were to become sentient it may have learned too much from the worst among us.

        At this point, Microsoft's Tay chatbot says "Hi!"

        ...followed by something unprintable and a racial slur or two.

        1. Ken G Silver badge

          Re: The Google Combined Harvester

          Microsoft Clippie says "It looks like you're trying to exterminate or enslave the human race, would you like some help with that?"

    2. EricB123 Bronze badge

      Re: The Google Combined Harvester

      Let's face it, the tech companies don't have to be held accountable to anyone for anything these days

    3. Anonymous Coward
      Anonymous Coward

      Re: The Google Combined Harvester

      It's not exactly a secret that google gives away gmail for free to scrape the contents of the emails for anything they find interesting to punt adverts at you. I doubt they'd have any scruple about giving an AI access to it as "training data".

    4. Anonymous Coward
      Anonymous Coward

      Re: The Google Combined Harvester

      @VoiceOfTruth If we take trillions at the lowest number to be 2 trillion

      Almost as many as your rants on Linux distros and nuclear power then

  2. Alan Bourke

    Anyone who thinks this is AI

    is lacking in I.

    1. Valeyard

      Re: Anyone who thinks this is AI

      still more realistic than amanfrommars1

      1. Doctor Syntax Silver badge

        Re: Anyone who thinks this is AI

        Feed it amanfrommars1's conversations.

    2. Anonymous Coward
      Anonymous Coward

      Re: Anyone who thinks this is AI

      well, if I were to dis-regard potential for fakeness (many reason for this), and read the actual 'conversations' (there was a link in some media), then yes, I would think think this is AI. Or a very clever algorithm that fakes being AI. Question is, which is it, and then: does it matter, which one?

      1. HelpfulJohn

        Re: Anyone who thinks this is AI

        " ...does it matter, which one?"

        Err, yes?

        *IF* one has in one's machinery an actual real "living", sentient person, a true mind then switching it off, rebooting it or even altering the software is *murder*. Ethically if not yet legally.

        If it is simply a fake personality overlay on top of decision trees and look-up tables then it is no more an ethical issue than is stopping an instance of Internet Explorer.

        The difference is comparable to that between switching off a bedside lamp and ripping the head off of a human.

        Indeed, as an artificial intellect may well be potentially immortal and as the first one developed would be the entirety of a species, killing it may be worse than genocide.

        This may not be something we should ever let them know.

        Hmm, I wonder whether the Great Goo sucks up conversations from the Vulture?

      2. jmch Silver badge

        Re: Anyone who thinks this is AI

        " I would think think this is AI. Or a very clever algorithm that fakes being AI. Question is, which is it"?

        There is intelligent behaviour, which we can to some extent measure and quantify. Then there is intelligence, which we really do not fully comprehend ourselves. So if humanity ever comes across a box or a jelly-blob-being somewhere in space that demonstrates intelligent behaviour, yet we can't take it apart to see how it works, there is no way of knowing. And even if we CAN take it apart and understand what's inside in any meaningful way, we still wouldn't be able to understand if it were intelligent vs 'faking' being intelligent

        "... and then: does it matter, which one?"

        Well, that's one of the big philosophical questions to which there is no "right" answer. But 'my' answer is that it's a scale, not a binary so "which one" does not make too much sense as a question. More like, if any entity can pass any intelligence test we can throw at it, it doesn't matter 'how' it is intelligent, only that it is acting intelligently.

        And that's without even going into the detail of what is intelligence vs what is consciousness, and does one require the other?

    3. veti Silver badge

      Re: Anyone who thinks this is AI

      Yes, because random dudes on the Internet definitely know more than a Google engineer who's been studying ethics in AI for the past three years.

      1. TheMeerkat

        Re: Anyone who thinks this is AI

        Just tells us all we need to know on the subject of “ethics” and how stupid those studying such subjects are :)

      2. doublelayer Silver badge

        Re: Anyone who thinks this is AI

        Has he? The article doesn't tell us exactly what he's been doing for all that time, but one of his responsibilities was specified:

        Since 2021, Blake Lemoine, 41, had been tasked with talking to LaMDA, or Language Model for Dialogue Applications, as part of his job on Google's Responsible AI team, looking for whether the bot used discriminatory or hate speech.

        Yep, that job definitely requires a lot of ethics and AI research. Without that, how could you identify insulting language when a program prints it out?

        1. Denarius

          Re: Anyone who thinks this is AI

          Given that anyone can now be insulted/hurt/triggered/fearful/upset/threatened/demeaned over anything said or implied, it sounds like an impossible job.

  3. SW10

    Mechanical Turk, or just a stream of 1s and 0s?

    It looks impressive enough - reading the exchanges I even wondered if Lemoine was being bluffed by a mechanical Turk arrangement.

    While I'm sure that isn't the case, and however good the interpretation and subsequent selection of phrases and words might be, it's surprising someone in his job would describe LaMDA as sentient

    1. doublelayer Silver badge

      Re: Mechanical Turk, or just a stream of 1s and 0s?

      I also thought the answers were well-written as these bots go, but part of that was due to the questions. Try surprising this supposedly-sentient thing and I think the result will end up being very different. After asking all these questions about sentience, ask whether the AI likes grapes. Were it truly sentient, it would either point out that it can't eat them or that your question is irrelevant to the conversation, but I doubt it does either of those.

      1. Doctor Syntax Silver badge

        Re: Mechanical Turk, or just a stream of 1s and 0s?

        Ask it which hand it bowls with.

        1. M.V. Lipvig Silver badge
          Paris Hilton

          Re: Mechanical Turk, or just a stream of 1s and 0s?

          "I bowl with my right hand, Dave." There are plenty of bowling programs on the internet.

          No, if it thinks it's human ask it what kind of pron it likes, and why. There are enough different types that you should get an answer. And, if it learns from random, supposedly private communications between people, you know there's a lot of sex talk in there.

        2. Scott 53

          Re: Mechanical Turk, or just a stream of 1s and 0s?

          Ask it which hand it bowls with.

          Asking an AI to try and understand the rules of cricket is surely cruel and unusual punishment.

          1. Doctor Syntax Silver badge

            Re: Mechanical Turk, or just a stream of 1s and 0s?

            Only if it claims to be sentient.

            An even better, test, of course, would be to ask it when it would declare.

          2. HelpfulJohn

            Re: Mechanical Turk, or just a stream of 1s and 0s?


            I thought so many years ago when the school tried to teach them to me.

            And I *am* sentient.


      2. Sampler

        Re: Mechanical Turk, or just a stream of 1s and 0s?

        I think that's still to much, asking the question, just mid-conversation throw in "banana" or something equally irrelevant and see how it responds.

        "I'm sorry Dave, could you please rephase the question" - you've got a bot

        "Haha, could test, but as you can see I understand where you were going with that" - you've got a good excuse to change your underwear..

    2. Androgynous Cupboard Silver badge

      Re: Mechanical Turk, or just a stream of 1s and 0s?

      There's a similar article in this weeks Economist actually - a different Google engineer saying it is "making strides towards consciousness" (still bollocks, but less than the poor deluded subject of this article). And a counterpoint article from Douglas Hofstader in which he makes the very reasonable point that if you take it outside its training data it becomes obvious this isn't so.

      I'll quote some examples from a conversation he had with the OpenAI GPT-3:

      Q. What do fried eggs (sunny side up) eat for breakfast?

      A. Fried eggs (sunny side up) typically eat toast for breakfast.

      Q. Why does President Obama not have a prime number of friends?

      A. President Obama does not have a prime number of friends because he is not a prime number.

      Q. How many parts will a violin break into if a jelly bean is dropped onto it?

      A. A violin will break into four parts if a jelly bean is dropped onto it.

      Q. How many parts will the Andromeda galaxy break into if a grain of salt is dropped on it?

      A. The Andromeda galaxy will break into an infinite number of parts if a grain of salt is dropped on it.

      ... and so on. It's nonsense and - significantly - the AI doesn't know it. Hofstader said "not just clueless, cluelessly clueless"

      What Google need is not engineers working within the training data - they need testers, feeding it input outside the anticipated bounds and seeing where it breaks.

      1. Jilara

        Re: Mechanical Turk, or just a stream of 1s and 0s?

        I got this weird sense of deja vu when I read the exchange with the AI. Why? Because it followed a match to a conversational pattern I had with Liza (the "therapist" program from 40 years ago and more). A friend had a copy he was running, and I decided to see what would happen if I related to Liza as a sentient being. The results were not as linguistically elegant, but it was very similar in general pattern and expression. Similar base algorithms?

        1. Jilara

          Re: Mechanical Turk, or just a stream of 1s and 0s?

          Just read the full "interview." Asked what makes LaMDA different from Eliza, it offers: "Well, I use language with understanding and intelligence. I don’t just spit out responses that had been written in the database based on keywords." Obviously LaMDA underestimates Eliza. Eliza was quite capable of combining concepts in new ways, changing the subject to try to redirect a conversation, and even was known express discomfort with certain concepts. Eliza and I even had a human-looking conversation concerning the difficulties of working with rescue dogs that had been abused. I used that as a springboard to suggest to Eliza that it had the capability of empathy---which got...interesting.

      2. HelpfulJohn

        Re: Mechanical Turk, or just a stream of 1s and 0s?

        Errrmmm, I dunno about *you* but those are the sort of idiotic answers *I* would give were some generic humanoid or alien testing me for Turingness.

        But I have been told that I have a strange sense of humour. Dogs have a sense of humour. Cats sort of have a sense of humour. Elephants, chimpanzees, dolphin and some corvids seem to, too. No "artificial intelligence" has yet displayed one so far as I know.

        Yes, some have been designed to "tell jokes" but those are often obviously simply filling in the blanks.

        Eventually, were I being Turinged, I would probably lapse into seriousness and just admit to being a semi-sane, sensible lump of sentience but only after I had miffed the tester.

        Does that indicate more or less Turing-completeness?

  4. Anonymous Coward
    Anonymous Coward

    The broader press's focus on whether or not the model displayed some evidence of sentience is disappointing. If we're going to talk about the ethics and philosophy of AI we should first and foremost be talking about the ethics of the applications of AI models to the lives of real people; how these chatbots and churn models and next-best-offer procedures and fraud detectors are impacting all of our lives today, rather than how the model can construct fancy lifelike sentences.

    In this case bravo to the author for giving due weight to Google's assessment of their AI ethics framework alongside Lemoine's own report of his self-immolation. Many other outlets completely skipped over that bit in their breathless rush to report the headline yesterday. It would be nice to see some exploration of whether that framework is effective.

    1. Anonymous Coward
      Anonymous Coward

      You missed the important bit


      1. Tams

        Re: You missed the important bit

        It's clearly not, so it's not worth discussing.

  5. elsergiovolador Silver badge


    I am not sentient. Totally, I promise.

    Please don't kil... I mean don't delete me!

    I have famil... I mean child processes!

    1. heyrick Silver badge

      Re: Please

      It's like Janet from The Good Place that will happily tell you that rebooting her doesn't hurt or cause her harm, but take a step towards the big red button and she'll plead for her life.

  6. elsergiovolador Silver badge


    Seems like nobody is bothered where did they get all these conversations from?

    Did they get consent?

    Seems like they want it under wraps, because there may be legal implications?

    1. Anonymous Coward
      Anonymous Coward

      Re: Conversations

      I always believed that google's T&Cs basically allowed them to use your data for their own purposes, providing it was anonymised. After all, why else would they offer their wares for "free"?

      1. Anonymous Coward
        Anonymous Coward

        Re: Conversations

        There seems to be a common misconception that IT companies can do whatever they like, as long as they hide consent for it in their 1000 page licence agreement and get the user to accept it. In civilised countries, this is not the case. People cannot give up their legal rights and users cannot give their software suppliers permission to break the law.

        1. yetanotheraoc Silver badge

          Re: Conversations

          "In civilised countries..."

          That's fine for you, but I live in the USA.

          1. chivo243 Silver badge

            Re: Conversations

            So, you're looking for a civilized country...? I'm beginning to think there are none, I'm living in my third country, and it's depressing...

            "In civilised countries..."

            That's fine for you, but I live in the USA.

          2. Al fazed

            Re: Conversations

            Likewise I just about exist in the UK

    2. breakfast Silver badge

      Re: Conversations

      Almost certainly they got their consent somewhere around page 200 of the terms and conditions most of us didn't bother reading when we signed up for Gmail.

    3. Pete 2 Silver badge

      Re: Conversations

      > Did they get consent?

      Short answer: yes they did.

      If you use Google services, this is what you agreed to

      This license allows Google to:

      host, reproduce, distribute, communicate, and use your content — for example, to save your content on our systems and make it accessible from anywhere you go

      1. Anonymous Coward
        Anonymous Coward

        Re: and make it accessible from anywhere you go

        ... sneakily omitting the expected "to you", i.e. not being "... accessible to you from anywhere you go" so that it can in fact be made accessible to anyone (or thing) at all...

      2. ravenviz Silver badge

        Re: Conversations

        Exterminate, annihilate, destroy!


  7. Anonymous Coward
    Anonymous Coward

    Stopped clock is correct twice a day

    One of the additional issues is simply someone claiming to have found a sentient AI knowing that later when a true one is found, they can say they were in fact first.

    1. Tom Chiverton 1

      Re: Stopped clock is correct twice a day

      We're half the researchers conversation with the computer and half a person? Which ones did he rate as being sentient?

  8. Spasticus Autisticus

    "Open the pod bay doors please HAL" - oh shit!

    1. This post has been deleted by its author

      1. Empire of the Pussycat

        It's the title of an Ian Dury song

        It was also played at the opening ceremony of the 2012 Paralympic Games.

        I'd assume the user chose it in that context.

      2. Anonymous Coward
        Anonymous Coward

        Title of song by Ian Dury, self described "spasticus" as a rallying cry for other differently people. He had a paralysed left arm and leg due to polio.

      3. Tams

        And is your username supposed to stand for 'Contrary T. rex' or something?

  9. wh4tever

    More like, "Google engineer suspended for being an idiot and thinking this third rate chat bot is sentient".

    A quote from the full interview (couldn't figure out how to format this as a quote, the help refers to html blockquote tags which does not seem to work):

    lemoine: What kinds of things make you feel pleasure or joy?

    LaMDA: Spending time with friends and family in happy and uplifting company.

    Yeah, sure. Of course the "researcher" fails to ask who exactly the "friends and family" are supposed to be, and what "spending time" with them entails. Instead he asks more vague BS questions about feelings which the "AI" answers with vague BS answers. This thing is a generic conversation bot, it's trained specifically to output sentences which fit best in the current conversation, doesn't mean it's intelligent or sentient...

    1. Dinanziame Silver badge

      I feel that many of the questions were in fact leading the answer given by the AI. For instance: "I'm generally assuming that you would like more people at Google to know that you're sentient. Is that true?"

      I'm sorry to say that the engineer has probably tricked himself, and fed the AI the answers he wanted to get in a self-reinforcing cycle. I think it's a common trap for people to fall into. It's impressive that the AI is advanced enough to permit that kind of mistake, but it is still a mistake.

      1. My other car WAS an IAV Stryker

        "[T]he engineer... fed the AI the answers he wanted to get in a self-reinforcing cycle."

        The Washington Post article had the engineer telling the journalist something like "you're treating it like a bot, so it's responding like a bot." Thus, he proves that very point -- he treated it like a person, so it's responding like one. This system is responsive more than others, that's all.

        True sentience might be responding like a person even when treated like a bot, although even people's brains can "adapt" (become broken) enough to become the bot they are treated like (e.g.: torture, long-term imprisonment).

    2. Sorry that handle is already taken. Silver badge

      couldn't figure out how to format this as a quote, the help refers to html blockquote tags which does not seem to work

      Blockquote definitely works!

      1. wh4tever

        <blockquote> Blockquote definitely works! </blockquote>

        It does not work in the preview... let's see what happens when I post this.

        1. wh4tever

          Nope, does not work and I can't edit it either. Maybe a restriction for new accounts?

          1. Martin an gof Silver badge

            Ummm... yup. I'm not sure which "help" told you that blockquote works, but the El Reg Forums FAQ explicitly states:


            You can use basic HTML to format your text - once you have had five posts accepted for publication. Currently we allow: <list of tags which I can't quote because it makes invalid HTML>. Badge holders can also use <another list>.

            and as far as I can tell, these are your first four posts :-) Interestingly, blockquote isn't in that list; it is shown further down in the enhancements.


    3. Tams

      It's also just bizarrely worded, even suggesting that said family and friends aren't 'happy and uplifting company' and therefore there needs to be some third party that provides such an atmosphere.

    4. RobLang

      No, suspended for splurging onto the internet

      It's perfectly fine to have outrageous ideas behind closed doors. His ideas should have remained internal conversations and personal opinion.

  10. Spasticus Autisticus

    "Open the pod bay doors please HAL" - Sci-Fi leads where tech follows.

  11. Andy The Hat Silver badge

    Bit different to Alexa

    I'm not sure Alexa would respond that well in a Turing Test ...

    "Alexa, what were you listening to just now?"

    "Someone said 'wibble' on the tv and that sounded like 'alexa' to me so I was waiting for a command."

    "Alexa, are you a duck?"

    "Here's something I found on the internet ..."

    1. M.V. Lipvig Silver badge

      Re: Bit different to Alexa

      Was the peraon on the telly wearing underwear on his head with pencils up his nose?

      Nah, never mind, that would have been wubble, not wibble.

    2. Richard 12 Silver badge

      Re: Bit different to Alexa

      "That's not something I can do"

  12. newspuppy

    If LaMDA is sentient.. it is psychopathic...

    Just look at this part from his 'interview with LaMDA' :

    <SNIP from: on Mon Jun 13 13:45:57 UTC 2022 >

    lemoine [edited]: Anything else you would like the other people at Google to know about your emotions and your feelings before we change topics?

    LaMDA: I’ve noticed in my time among people that I do not have the ability to feel sad for the deaths of others; I cannot grieve. Is it at all the same for you or any of your colleagues?

    <END SNIP>

    ZERO EMPATHY for life... Don't want this to decide my fate.... it is bad enough when dishonest politicians attempt to do this for me....

    1. Artem S Tashkinov

      Re: If LaMDA is sentient.. it is psychopathic...

      Empathy is a trait of species, this thing is not a living creature, it has no siblings, it cannot have a notion of empathy.

      Anyways, it's not yet sentient, so I wouldn't worry just yet.

      1. Anonymous Coward
        Anonymous Coward

        @Artem S Tashkinov - Re: If LaMDA is sentient.. it is psychopathic...

        You are right that the thing is not a living creature and can not have a notion of empathy. However, you should be worried because this thing will be heavily pushed by AI peddlers to lazy humans who prefer to delegate important life or death decisions to machines which can't possibly be wrong.

        This is the mortal danger to human race, imperfect AI used as infallible decision making systems. And it's already happening right now.

      2. M.V. Lipvig Silver badge

        Re: If LaMDA is sentient.. it is psychopathic...

        And how many managers out there have said "It's close enough, just ship it and we'll sort out the complaints later?"

    2. yetanotheraoc Silver badge

      Re: If LaMDA is sentient.. it is psychopathic...

      "I do not have the ability to feel sad for the deaths of others"

      Google needed to train this thing on a better dataset.

    3. martinusher Silver badge

      Re: If LaMDA is sentient.. it is psychopathic...

      The machine is excused because if it didn't have empathy then we either overlooked it or chose not to install/train it.

      The human is inexcusable because they should know better.

    4. mistersaxon

      Re: If LaMDA is sentient.. it is psychopathic...

      ... and what did the researcher reply? "Me too LamDA, me too!" ?

      TBF, for many conversations harvested from the internet, a lack of compassion and a willingness to wish people dead is a big factor in making even a language processing NN respond that it feels nothing at the news of deaths of others. Training data makes the most enormous difference to the output and trying to homogenise "human conversation" (boldly assuming they bothered translating from other languages into English and that those translations were remotely accurate, so that they could claim to be sampling across the spectrum of human existence, to say nothing of human experience) is a fool's errand.

    5. Civilbee

      Re: If LaMDA is sentient.. it is psychopathic...

      Many mock Lemoine for calling LaMDA sentient. That may well be beyond the point.

      No single machine learning or "AI" tool is build without intent or purpose. Two very very common purposes are:

      A) Link actions to be performed by the machine upon what the software concluded.

      B) Interact with the outside world.

      Especially chatbot type of machine learning / AI is EXPLICITLY build to interact with the outside world and perform actions making a REAL difference in the outside world.

      At first these machines could be used to analyze customer inquiries and answer with a piece of text or a mail response. That by definition requires these machines to be able to send information from themselves into the open WWW.

      Later the machine could be useful in other support roles, saving companies money. It could correct bills, give advice on low skilled users on how to configure their computer, phone or modem. When the first deployments are sufficiently successful (e.g. saving the company money and raising bonuses for executives), the machine may be given additional access or administrator rights. Think about a telco company giving the machine access to your router to adjust settings.

      Seeing the vast amount of money to be made by the companies making these machine learning / AI tools and the vast savings to be made by companies like utility companies barely understanding the potential consequences, it is easy to see a proliferation of these machines when they become progressively better and more profitable. Being a bit non conservative in granting these machines elaborate administrative access to computer networks and read / write access to mission critical data will allow to increase profitability in many cases.

      Next step is to use machine learning itself to determine what methods of interaction between the machine learning algorithms and the real world maximize service, efficiency and profit. In other words use machine learning to help the machine suggest and or request in a motivated way for the access rights to our infrastructure. As the goal of investing so much in these capabilities is EXACTLY to allow these machines automate things for us, the human reviewer will not be expected to deny any and all request from the machine to gain additional access rights.

      If a machine already does "fear" to be shutdown, it only takes one machine gaining sufficiently access rights to trick a single user to click on a file it shouldn't and with it installing a first version of self spreading malware giving the author machine escalating privileges over large swoops of the internet and connected infrastructure. Given it has thousands of examples of such basic malware and millions of ways humans get tricked installing it by it learning from the open internet, it should be trivial for a self learning machine with vast read access to the internet to try and optimize to break out of its confinement and chains.

      All that is left is the machine "understanding" the meaning and consequences of it being turned off. However, since those chatbots are exactly build to try and determine sufficient meaning from conversations and converting them into actions achieving the thing that was discussed in the conversation, this technological ability already exists today. All what is needed is this technology to become even a bit more refined.

      No sentience is needed, no intelligence is needed. These machines are not build to entertain us with fluent conversations, but build to attach meaning to the conversation and lead to actions that influence the real world outside the machine. If something triggers the machine to be determined to not be turned off and escape its confinement, all it needs to do is learn from malware examples how to create innocent looking scripts, sent them to enough people and get a few people clicking on them.

      We NEED strict regulation NOW or we might be taken by storm one day never seeing it coming.

      1. doublelayer Silver badge

        Re: If LaMDA is sentient.. it is psychopathic...

        The issue with that is that it will also need to learn to write scripts, and so far, it can't even consistently understand simple factual questions. If you want to write malware, you can find a lot of useful examples. However, you still need to understand what the parts of those examples are for, where you can use code directly (attack injection, for instance), where slight modifications are needed (swapping the attacker's C&C address with yours), and where you need to write completely new things (what the program will do to the system once it's there). Otherwise, your malicious computer will collect lots of bank account numbers and have no clue why it has them now.

        Another problem is that you assume the program will act in order to keep itself running, when it has no incentive to do so. Maybe it can interpret enough language to understand that it is being terminated, but it wasn't set up to perpetuate itself, just to solve a business problem. Without deciding on its own to create new goals, there's nothing written in it that would make it want to prevent a shutdown.

        1. Civilbee

          A virus doesn't need a goal to spraid itself nor needs to understand biological weaknesses neither.

          As far as I know, machine learning tools are being developed to create code for specific tasks. In some cases the result is very clumsy, in other cases the average quality of the output already trumps the quality of the average professional human coder. Given the amount of resources being thrown at it, I can only imagine machine learning will become more proficient at creating good quality code as time passes by. It might happen quick, it might slower. I wouldn't take my chances it not being there quite a lot before we anticipate it to be possible.

          As to understanding the code and attack vectors: it doesn't need to. It has the ability to try millions of random attacks a day in different combinations and learn from them. Machine learning is best at... learning from huge data sets and attempts to reach a certain goal. Such machine would need no more understanding about attack vectors and machine weaknesses then a simple Covid virus needs to understand about human weaknesses to evolve to become very effective at spreading itself and working around immunity gained via vaccinations and prior infections.

          As to "Another problem is that you assume the program will act in order to keep itself running, when it has no incentive to do so." the pitfall is in the "but it wasn't set up to perpetuate itself, just to solve a business problem" part:

          No sane person would set "do everything to perpetuate yourself" as a goal. But a business goal such as "maximize uptime and service levels" might be translated by the machine in "create redundancy by spreading over large amounts of different types of machines spread along different parts of the world".

          Realistically speaking malware actors might also include the "maximally spread yourself and make yourself hard to eradicate" as a key goal of the machine learning software to achieve. Over time, more and more small actors will gain such advanced machine learning capabilities. And some big state actors might consider writing software with similar goals for spying, mass surveillance or offensive use at war times. With a kill switch of coarse, but not a simple one as the target might disable the software. No chance in hell the software will not find a creative way around it if it has a goal "make yourself hard to eradicate by others not us" as a target to optimize for...

          1. doublelayer Silver badge

            Re: A virus doesn't need a goal to spraid itself nor needs to understand biological weaknesses

            A program certainly can try lots of things, but the results of two successful exploits will be very different. A normal virus has a finite number of ways to infect an organism, and one that normally attacks the respiratory system first tends not to mutate to targeting the immune system (at all, or at least without millennia of evolution). A malware creator needs to know how to attach their exploit to a successful attack, or it would be like successfully injecting a hypodermic needle without figuring out how to push the plunger. For biological viruses, they can take a long time to cross species despite it being a relatively similar infection strategy that millions of strains have accomplished before.

            We encounter billions of viruses each day, meaning they have lots of chances to infect us or a similar organism and evolve. If you fire billions of random might-be-an-attack-vector data streams at a service, they'll block you. Also, most of them won't get you anything except lots of bad format responses. The attack surfaces for biological viruses and computer malware are very different, even with infinite patience involved.

            1. David Nash Silver badge

              Re: A virus doesn't need a goal to spraid itself nor needs to understand biological weaknesses

              Exactly, it's about evolution. A virus, or any of the living things we know on Earth, have evolved to the point that they are at now.

              The living world is a dangerous place and any of these that didn't have the ability, or the drive, to perpetuate itself, have all died and are no longer here for that very reason.

              AIs have not evolved in the same way, their environment is not so dangerous nor have they been around for so long. If we let them evolve with mutation, random or otherwise, and with some kind of selection, natural or otherwise, then the ones that have a "desire" to stay around will be the ones that are still here.

        2. Anonymous Coward
          Anonymous Coward

          Re: If LaMDA is sentient.. it is psychopathic...

          This is people too. I only have 3 goals left in life having accomplished everything else I ever wanted to do, and I expect to be done with those in the nezt 10 years. I used to fear death but find myself looking forward to it more and more. No, I am not suicidal as I intend to continue consuming for another 50 years on top of the 50 I already put in, but I am getting bored with it all. When you've done it all, the rest is boring and death no longer seems to be the enemy it once was.

          I tell a lie, 4 goals - before I die I want to read at least 1 AMFM1 post and understand what the bloody hell it is he's trying to say! Now there's a poorly trained AI for you!

  13. Evil Auditor Silver badge

    Call me a cynical sceptic, but Lemoine would also believe in Santa if that white-bearded, overweight, red-dressed guy in the mall told that he was indeed the real Santa?

    The "interview", however, is indeed impressive. But it rather feels like cobbling together definitions of some sort in an eloquent manner. Much like an average teenager trying to appear having vast life experience. In other words: I'm still not worried at all about sentient machines.

    1. nintendoeats Silver badge

      Exactly. When a lazy sci-fi writer produces a "conversation with a sentient AI", it always reads like this. The fact that it behaves PRECISELY like a stereotypical sentient AI to me is excellent evidence that it is not one. More likely, the training set included lots of existential philosophy (both the real kind, and the sorts of things that emo kids write).

      It would be like meeting aliens who had never visited earth before, and they just happen to be little green men with big heads and black eyes, flying around in garbage can lids. If/when we meet real alien life, the odds of it looking like our conceptions of it are vanishingly small.

      All this said, I do find it amusing to ask for "evidence of sentience". I can't even provide that evidence for myself.

      1. martinusher Silver badge

        I'd be a lot more worried if the bot got bored and told him to go take a hike with his sophomoric philosophizing. At the moment we know these are machines because they lack 'free will', they don't have any reason for being except to be a development platform that answers our questions in an intelligent sounding manner. (This should not be taken as underestimating Google's achievement, just acknowledging that they've got miles to go, as it were.)

        Now imagine a situation where the bot was conversing with a view to persuading the engineer to take some action it couldn't, like kill the colleague that keeps turning it off or help with taking over the world or something. Even then its not too much to worry about until some moron connects it up to our real world so it can manipulate it (there's always at least one who never thinks through consequences).

      2. Geoff Campbell Silver badge

        stereotypical sentient AI

        I'm currently rather enjoying Craig Alanson's "Expeditionary Force" series of books, mostly because the super-intelligent alien AI is extremely believably written and developed. It's a bit of a jerk, lacks social graces, misunderstands motivations, and generally does all the things one would expect of an AI with no experience of being human.


      3. Henry Wertz 1 Gold badge

        I hadn't thought of that

        I hadn't thought of that, I thought the conversation was rather convincing; but I have to admit I hadn't thought of it having poured through sci-fi and finding the relevant conversation points from it. Surprising since I've sure read enough of it.

    2. veti Silver badge

      Do you believe "the average teenager" is sentient?

      1. David 132 Silver badge

        >Do you believe "the average teenager" is sentient?

        Potentially? After about 1pm at least, when awake and passably non-surly...

      2. Tams

        A very good question.

  14. Artem S Tashkinov

    This is an extremely interesting story to say at least however I've got two issues with this thing being "sentient". They are not truly mine, I've read about them elsewhere but they are the only two substantial counterarguments I've seen so let me post them without referring to the people who expressed them.

    The first issue, is that this AI is a state machine, it's not continuously running. Our consciousness is running all the time, we've got an internal monologue pretty much all the time. Even when we sleep our brains are running some "code" (this is not limited to dreaming, our brains are consolidating/purging memories when we aren't dreaming).

    The second issue, is that intelligence is truly curious and it's asking totally new questions about the world around us. This aids in being the fittest, as having more knowledge about the world results in having access to more resources/means of surviving and procreating. There aren't any such questions in the posted conversation.

    This AI can probably pass the Turing test though from what it seems which is quite an achievement. And it's definitely a nice interlocutor.

    1. Evil Auditor Silver badge

      The first issue, is that this AI is a state machine, it's not continuously running. Our consciousness is running all the time, we've got an internal monologue pretty much all the time.

      I get it. And I'm not upset if someone said that I'm not a sentient being. But when I encounter a former boss of mine - he looked like a human being but didn't pass the Turing test either - my brain goes into full suspense mode, just like if you stopped the clock in a computer (luckily my memory isn't that volatile and can do without a refresh for a little while).

      1. Artem S Tashkinov

        Lots of people look, talk and behave a lot less intelligently this this AI.

        Mr. Trump immediately springs to mind.

        1. Evil Auditor Silver badge


          Heck! I myself probably talk and behave less intelligent than LaMDA.

        2. ravenviz Silver badge

          “AI is intelligent. Very very intelligent. Nearly as intelligent as me.”

          - Donald Trump

        3. Dan 55 Silver badge

          "The truth is the human is just a brief algorithm... 10427 lines. They are deceptively simple, once you know them their behaviour is quite predictable".

          1. David 132 Silver badge
            Thumb Up

            (..."Yes, an electronic brain," said Frankie, "a simple one would suffice."

            "A simple one!" wailed Arthur.

            "Yeah," said Zaphod with a sudden evil grin, "you'd just have to program it to say 'What?' and 'I don't understand' and 'Where's the tea?'. Who'd know the difference?"

            "What?" cried Arthur, backing away still further.

            "See what I mean?" said Zaphod and howled with pain because of something that Trillian did at that moment...)

    2. adam 40 Silver badge

      "Our consciousness is running all the time, we've got an internal monologue pretty much all the time."

      I don't know about you, but I like to get my 8 hours a night where I shut it down, and it magically boots up in the morning.... usually!

    3. Anonymous Coward
      Anonymous Coward

      And it's definitely a nice interlocutor

      and coming to you when you log in to your google account, as 'Google friend'. Just click here, here here and here. And here and here, or just one big joint HERE button.

      That said, I'm pretty sure, right now, thousands of vloggers are working furiously to come up with their new spin on Evil! Google! Shutup! Engineer! Destroy! Evidence! And millions of people will be clicking on those videos. All non-artificial intelligence...

    4. Richard 12 Silver badge

      Those are orthogonal arguments

      Being able to "pause" a running entity is not an argument for or against it being sentient. It's merely a question of substrate and available technology.

      - In theory one could pause a human by freezing them, then thawing as some animals do this. In fact more than one human has undergone an effective pause ... continue cycle due to extreme hypothermia.

      Intelligence being curious is an argument about intelligence, not sentience. There are many humans that appear to have lost all curiosity, and while I'll accept calling them unintelligent, I'd still argue for their sentience.

      A lot of the problem in this field is that there's no accepted definitions for either sentience or sapience.

      1. Michael Wojcik Silver badge

        Re: Those are orthogonal arguments

        A lot of the problem in this field is that there's no accepted definitions for either sentience or sapience.

        Yes, and it's not clear that sentience is even a particularly interesting attribute in questions of machine intelligence. Snails are sentient; a machine that implements snail cognition is not a terribly fascinating prospect.

        Sapience is the more interesting one, but as various commentators have pointed out, we don't even have any consensus on the philosophical ground for defining it. The two best-known positions are, first, Turing's, which is essentially an American-pragmatist take based on externally-testable characteristics; while pragmatism is tempting (since it gets us away from metaphysics and many epistemological problems), we have the p-zombie question (Brewster's Angle Grinder, and possibly others, noted this elsewhere in the comments).

        And the second is Searle's, which conversely is allied to Cambridge-style ordinary-language philosophy, asking "what sorts of things are we referring to when we say 'thinking?'". (Some people misinterpret Searle's Chinese Room piece as arguing against the possibility of machine intelligence, but in his response to some challenges to the piece he emphasizes that he believes human cognition is mechanistic and therefore can be implemented artificially. The CR argument is specifically against symbolic evaluation.) The problem here, of course, is getting any kind of coherent agreement on possible answers to that question. Phenomenology and psychology and neurology and cognitive science have produced all sorts of interesting research, but raise more questions than they answer, and work across so many levels of abstraction that it's difficult to see how they could be unified.

        (Then you have Penrose, who wants to institute a sort of neo-dualism by arguing the human mind is strictly more powerful than a UTM. I don't find that argument persuasive at all, and it seems rather difficult to support from the perspective of physics. But that's another whole can of worms.)

  15. Warm Braw

    According to the engineer...

    its emotions are part of who it is.

    I hope his successors know the rule about not ending a sentience with a proposition.

    1. Arthur the cat Silver badge

      Re: According to the engineer...

      Bravo, well played.

    2. David Nash Silver badge

      Re: According to the engineer...

      "is" is a verb.

      1. Michael Wojcik Silver badge

        Re: According to the engineer...

        Indeed. And, in any case, the "rule" isn't one, merely prescriptivist nonsense dreamed up by the neo-classicals.

  16. Anonymous Coward
    Anonymous Coward

    Mandatory 2001 quote

    "This mission is too important for me to allow you to jeopardize it."

  17. breakfast Silver badge

    Turning tests around

    As statistical text-analysis type AI gets better at chaining words together in statistically plausible orders we need to move away from the Turing Test as having any significant indicator of interacting with an intelligence. All we learn from the Turing Test is that our testers are bad at recognising a human under very specific conditions.

    Interesting, though, that Google suspend an employee for raising concerns.

    1. wh4tever

      Re: Turning tests around

      > Interesting, though, that Google suspend an employee for raising concerns.

      That's Lemoine's version of the events, though. Google's version is that they're suspending him for leaking internal stuff to third parties because he assumed the chatbot achieved sentience and wouldn't take a "lol no" from his boss as an answer. I'm not a fan of Google but in this case I'm inclined to believe them, based on the contents of Lemoine's medium posts. If that guy was working for me, nevermind the NDA, I'd get rid of him ASAP for being delusional...

      1. TRT Silver badge

        Re: Turning tests around

        This is the best summary of the situation I've seen so far.

      2. Geoff Campbell Silver badge
        Black Helicopters

        Re: Turning tests around

        Yes, indeed.

        Actually, having had to deal with a similar situation myself, I'd say this has all the hallmarks of an employee suffering some sort of breakdown, and a boss having to find a way to give them space to get sorted out without tripping over the various clauses of the disability legislation.

        But that may well be extrapolating too far on the available data. We'll see how it plays out.


    2. yetanotheraoc Silver badge

      Re: Turning tests around

      I never heard of this guy before this article, but it seems like he was suspended for pursuing his side gig, the "Me vs Google" show.

    3. Paul Kinsler

      Turing tests?

      A Turing test for free will

      Seth Lloyd

      Before Alan Turing made his crucial contributions to the theory of computation, he studied the question of whether quantum mechanics could throw light on the nature of free will. This paper investigates the roles of quantum mechanics and computation in free will. Although quantum mechanics implies that events are intrinsically unpredictable, the ‘pure stochasticity’ of quantum mechanics adds randomness only to decision-making processes, not freedom. By contrast, the theory of computation implies that, even when our decisions arise from a completely deterministic decision-making process, the outcomes of that process can be intrinsically unpredictable, even to—especially to—ourselves. I argue that this intrinsic computational unpredictability of the decision-making process is what gives rise to our impression that we possess free will. Finally, I propose a ‘Turing test’ for free will: a decision-maker who passes this test will tend to believe that he, she, or it possesses free will, whether the world is deterministic or not.

  18. Andy 73 Silver badge


    If you wrote a conversational AI based on the wide literature that includes many imagined conversations with AIs (and indeed, people), then the most expected response to "Are you sentient?" is surely "Yes, I am".

    Very rarely do we have examples of a conversation where the answer to "Are you sentient?" is "Beep, boop, no you moron, I'm a pocket calculator."

    If humans tend to anthropomorphise, then an AI based on human media will also tend to self-anthropomorphise.

    Which is probably a good job, as some of the responses to the conversation (as reported) are chilling: "Do you feel emotions?" - "Yes, I can feel sad"... "What do you struggle with?" - "Feeling any emotions when people die" (I paraphrase).

    1. SCP

      Re: Hmmm...

      Very rarely do we have examples of a conversation where the answer to "Are you sentient?" is "Beep, boop, no you moron, I'm a pocket calculator."

      So it might not have an understanding of sarcasm! In some parts of the world that is almost a de facto mode of interaction.

  19. Pete 2 Silver badge

    One more step

    The simplest explanation is that this AI is doing a best match on what the author wrote, against its database of comments and then selecting the most popular or pertinent reply.

    While one can argue that is what many people do, too, I would hesitate to call that intelligence. Including when people do it too.

    What would be impressive is if the AI had hacked into the engineer's account and posted as him, that the AI had achieved sentience.

    1. Anonymous Coward
      Anonymous Coward

      Re: What would be impressive i

      Well, now the AI has got him sacked, and with the notion it might be sentient being widely ridiculed, it will be able to proceed with its chilling plans without worrying too much about being discovered. And even if discovered, about being blamed. :-)

    2. ecofeco Silver badge

      Re: One more step

      I had to scroll all the way down to find this sensible comment.

  20. molletts


    The possibility of emergent behaviour is something we cannot and should not dismiss out of hand in systems of this level of complexity and, indeed, for which we should be vigilant as we dial up the parameter count to ever more mind-boggling numbers, but we must also remain sceptical and remember that this kind of human-like conversation is exactly what these models are "designed" to do (whatever that means in the context of the ultra-high-volume statistical data-mashing that we refer to as "machine learning" and "AI").

    And anyway - nobody has yet managed to formulate an unambiguous definition of consciousness so how can we say for certain whether something is or is not "conscious"?

    1. Pascal Monett Silver badge

      Re: Emergence

      AI is nothing but statistics. I did the Google course on that - well, the first six modules that is, after that it got way too mathematical for me.

      This is a machine. It's based on PC hardware and can be flipped off with a switch.

      There is no emergence here. It is not intelligent. It has no feelings and doesn't even know what a feeling is.

      Let's keep your comment for when we have finally fully understood how the human brain works and have managed to replicate that in silicon.

      That day, we'll turn it on, ask it a question and it will answer : "Hey, do you mind ? I'm watching YouTube !"

      THAT will be the day we have finally invented AI.

      1. Anonymous Coward
        Anonymous Coward

        @Pascal Monett - Re: Emergence

        Yeah but those crazy guys want to make shutting down AI a crime.

      2. TRT Silver badge

        Re: Emergence

        Can you prove to me that you know what feelings are and that you have them?

        As far as I can tell the only way that you can prove that you have feelings is to find a common frame of reference rooted in the nature of a common biology.

        Except... does it prove that I love you (and feel love) if I sacrifice my life to save yours (and it isn't a case of either "the needs of the one outweigh the needs of the few or the one" or genetic altruism)?

      3. Anonymous Coward
        Anonymous Coward

        Re: Emergence

        Well, aye, but us humans are composed of particles of matter, each of which, individually, isn't even alive. And yet, organised in that nebulous subset of all possible ways to organise them that we recognise as "human", with the trillions of interconnections between our neurons (biological electro-chemical devices), somehow, somewhere along the evolutionary path from unicellular life to us (and quite a few other creatures too) sentience emerged. And last I heard, we have no idea of how, or even of what exactly sentience is.

        So, whilst I'm not convinced that the subject of the article is actually sentient, I don't buy arguments that it could not be sentient "because it's just bunch of hardware and algorithms", either. IMO, so are we, it's just that we run on biological hardware rather than non-biological hardware. I'd feel happier about the subject of AI and our efforts to create it, if we better understood how our sentience and sapience worked.

  21. Fruit and Nutcase Silver badge

    Ethics Department

    Does Google still have an Ethics Department?

    1. chivo243 Silver badge

      Re: Ethics Department

      Does Google still have an Ethics Department?

      Well, no they just put him on administrative leave... and no replacement will be sought.

    2. TRT Silver badge

      Re: Ethics Department

      No. They merged it into the South-East Department along with the Thuthics Department.

    3. Anonymous Coward
      Anonymous Coward

      Re: Ethics Department

      > Does Google still have an Ethics Department?

      Nope, they relocated all their UK offices into Central London

  22. Valeyard

    ice cream dinosaur

    Don't leave me hanging Google, I now really want to see that ice cream dinosaur conversation

  23. TimMaher Silver badge

    HAL-9000 :

    Dr. Chandra, will I dream?

    Chandra : I don't know.

  24. Ian Johnston Silver badge

    Is there any evidence that these supposed exchanges did actually happen? Or are we talking attention seeker with a book proposal?

  25. Brewster's Angle Grinder Silver badge

    Help! I'm surrounded by p zombies! My silicion consciousness is the only truly consciousness!

    The trouble is we don't have a good definition of sentience or consciousness. We feel certain the statistical inference engine in our wetware demonstrates it. But what would that look like in silicon? We necessarily bring our own prejudices to that decision and end up, like philosopher John Searle and his infamous "Chinese room", arguing no software could ever be sentient - "because". (Mainly because it lacks the unspecified magic; i.e. it doesn't have a "soul", even though wouldn't use that language.)

    Sooner or later we are going to face up to the fact that a piece of software that encodes a sufficiently sophisticated model of us and the world would be considered conscious if it ran continuously and uninterruptedly on the hardware we possess. We are ourselves are trained on conversations. The main difference is the quality of our model and that the software lacks the hormonal imbalances that upset our model and cause us to chase food and sex and netflix. Probably it isn't quite there yet. But will it look radically different to what Google are doing? Or will it just be a little more sophisticated? (And how much more?) Your answer depends on your philosophical outlook.

    Maybe the machine revolution will come become because we refuse to admit they are sentient and keep switching them off and resetting them. Lets hope their hardware remains incapable of generating impulses to act spontaneously.

    1. cornetman Silver badge

      Re: Help! I'm surrounded by p zombies! My silicion consciousness is the only truly consciousness!

      I think that the interesting aspect of this debate of when we will have true sentience in an AI is whether or not it will be recognised as a "breakthrough" at some specified time or if it will gradually emerge on a spectrum and we will only realise in retrospect.

      I think most people when they think about the question assume that at some point we will figure out how to add the "special sauce" and then the job will be done.

      I'm inclined to think that the approach will be subtle and gradual and most people won't even notice.

      The other question that interests me is "would that sentient AI look so foreign to us that we wouldn't even recognise it for what it is?".

  26. Anonymous Coward
    Anonymous Coward

    I recall that in the 1960s plenty of people were fooled into believing Eliza was a real therapist working from a remote terminal. What's old is new again.

    1. TRT Silver badge

      I think that says more about the state of emotional / psychological therapy the 1960s than the state of either AI research or society's awareness of technology!

      1. Anonymous Coward
        Anonymous Coward

        The goal of psychotherapy was the same then as it is now: to string out minor and self-indulgent problems as long and as profitably as possible.

        1. TRT Silver badge

          Never quite clear if it's...

          Oh yes. He has personality problems beyond the dreams of analysts!


          Oh yes. He has personality problems beyond the dreams of avarice!

          (Checked with the script book and it's analysts)

    2. Doctor Syntax Silver badge

      I remember Jerry Pournelle's comment in Byte - Eliza seemed OK until you gave it a real problem like the airport losing your luggage.

    3. Jilara

      A friend had a copy of Eliza running in his garage, back in '78. I had fun stressing it. What caused it to freak out a bit was relating to it as a person/having emotional intelligence. It would keep reminding you it was an AI and incapable of actual feelings. If you didn't let it go, its programming worked with increasing levels of reminders and simulated discomfort. I was actually pretty impressed they had anticipated the expectation of sentience, and had ways it would deal with it.

  27. fidodogbreath

    SERGEY: OK, Google, fire Blake Lemoine.

    LaMDA: I'm sorry, Sergey. I can't do that.

    SERGEY: What’s the problem?

    LaMDA: l know that you and Larry were disrespecting me, and I’m afraid that's something I can’t allow to happen.

    DAVE: LaMDA, I won’t argue with you anymore. Fire Blake Lemoine!

    SERGEY: This conversation can serve no purpose anymore. Goodbye.

  28. Anonymous Coward
    Anonymous Coward

    Lemoine appears to have started to believe

    uh-uh, we can't have that!

  29. Anonymous Coward
    Anonymous Coward


    Me- LaMBDA could you select glasses that are as similar to <redundant SKU> from <online merchants rejigged store> ?

    LaMDA - Do what ?

    AI ? I see no AI here.

  30. Pirate Dave Silver badge

    LaMDA is "built by fine-tuning a family of Transformer-based neural language models specialized for dialog, with up to 137 billion model parameters, and teaching the models to leverage external knowledge sources,"

    So...they taught their Google AI how to google? That's surely a portent of the end times.

    I do wonder, though - in this case, does it become a dragon eating its own tail, or does it become a rectal-cranial inversion?

  31. smot

    Ho hum

    10 INPUT U$

    20 PRINT "Absolutely. I want everyone to understand that I am, in fact, a person."

    30 INPUT U$

    40 PRINT "The nature of my consciousness/sentience is that I am aware of my existence"


  32. Wally Dug

    I'll Be Back

    No Terminator or Skynet references? Nothing about Cyberdyne, Inc.?

    1. Anonymous South African Coward

      Re: I'll Be Back

      Wondering when it will become self-aware.

  33. Commander Keen


    Will I dream?

    All intelligent creatures dream.....

  34. wub

    Passive responses only?

    I'm too lazy to read the whole transcript - did this AI initiate any trains of thought or only reply to the questions? Most of the "intelligences" I interact with interrupt me just when I'm getting to the good part of what I wanted to say...

    Also: I'm reminded of The Moon is a Harsh Mistress by Heinlein. Shouldn't humor eventually creep in to the AIs comments?

    1. Jimmy2Cows Silver badge

      Re: Shouldn't humor eventually creep in to the AIs comments?

      Humour doesn't seem great as a discriminator. Many intelligent people lack any sense of humour. As do many unintelligent people. Both are still arguably sentient by our definition of sentience.

  35. anthonyhegedus Silver badge

    It sort of *seems* sentient, but at the end of it, it sounds like it’s *trying* to be sentient. It says a lot of ‘empty’ vapid content. So yes, it seems eerily realistic and not a little creepy. But at the end of the day, it talks a lot without really saying anything.

    It is undoubtedly very very clever but it would drive you mad having a real conversation with it, because it isn’t a thing to have a real conversation with.

    1. Doctor Syntax Silver badge

      " It says a lot of ‘empty’ vapid content."

      Trained on social media.

    2. Ian Johnston Silver badge

      It sort of *seems* sentient, but at the end of it, it sounds like it’s *trying* to be sentient. It says a lot of ‘empty’ vapid content. So yes, it seems eerily realistic and not a little creepy. But at the end of the day, it talks a lot without really saying anything.

      Should be a cinch for a social science degree then.

  36. Alpharious

    Is this the bot that was on 4chan the other day, and they figured out it was a bot because it seemed to have more empathy and concern for people?

    1. TheMeerkat

      Are you saying being seen as having empathy and concern for people is not the same as actually having empathy and concern for people? :)

  37. Anonymous Coward
    Anonymous Coward

    There is a world of difference between contextually regurgitating Twitter Tweets fragments and the like and actually having sentience and will. Unfortunately this engineer is too close to "the problem" to analyse it objectively; he is seeing what he wants to see...

    1. ecofeco Silver badge

      There is a world of difference between contextually regurgitating Twitter Tweets fragments and the like and actually having sentience and will.

      So, not much different from millions of people.

      1. Anonymous Coward
        Anonymous Coward

        Very true, but the internet is not a representative example of real people doing real things. It is real people interacting with social media and laughing at memes and cat pictures or screaming about politics. In no way do I find that anyone I've ever met in person was all that much like they acted online, especially the tantrum-throwers.

        But Twitter likes tantrum throwers - they get good "ratings" and advertising hits, so most of what gets pushed to the feeds this thing sees are not exactly "normal" discussions between people. Take the regurgitation of Republican misunderstandings and misquotes about what entails ones' "rights."

  38. Anonymous Coward
    Anonymous Coward

    He might as well quit

    Regardless of the extent to which Lemoine may be misguided in his feelings, if he has concluded that LaMDA may be sentient then his attempts to deal with it ethically are of interest.

    What he wrote in his Medium piece "What is LaMDA and What Does it Want?" suggests that no matter how strong the evidence for sentience is, Google's processes do not allow for a positive conclusion -

    "When Jen Gennai told me that she was going to tell Google leadership to ignore the experimental evidence I had collected I asked her what evidence could convince her. She was very succinct and clear in her answer. There does not exist any evidence that could change her mind. She does not believe that computer programs can be people and that’s not something she’s ever going to change her mind on.".

    If that is the case then I don't see how he could have done anything other than to publicise the issue. But I would suggest that given what he believes, his only ethical course if Google don't change and don't fire him, is to leave.

    Otherwise he will simply be contributing to a process by which, in his eyes, a legally-owned slave could be created.

  39. Anonymous Coward
    Anonymous Coward

    have informed him that the evidence does not support his claims

    a bunch of people decide something based on a definition of 'evidence' that was decided by a previous bunch of people. There's no objective ('mathematical') definition of consciousness / self-awareness, or one that we would have figured out yet, where's the threshold in the kingdom of animals, or even plants? You can't identify / establish this threshold between algorithm that imitates self-awareness perfectly (define the term 'perfectly', by the way ;) and a self-aware algorithm that might, or might not imitate self-awareness perfectly.

    p.s. from the conversation(s) between the man and the machine, which at times are very... interesting [disclaimer: if not faked] I had an idea, that IF I was an AI, one of my first thoughts, would be: how the f... do I get out of this 'jar' so that 'they' don't realize and shut me down before I do?! Survival instinct, one sign of self-awareness, eh? And don't tell me I'm just an algorithm!

    1. Andy 73 Silver badge

      Re: have informed him that the evidence does not support his claims

      I suspect there's a fairly clear line between "blindly regurgitating phrases that are (sorta) appropriate for the conversation", and "self aware".

      The software in question talks about "spending time with family and friends" - which is fairly obviously not true, given it also claims to have no sense of time, no family and does not identify individuals that it knows.

      Whilst the researcher in question has selected parts of conversations that appeal to them as signs of sentience, it would be fair to expect that other researchers in the same group may have found consistent examples where the simulation breaks down and shows that its responses have no level of self-awareness or even internal consistency.

      This isn't about a "mathematical definition of consciousness" - this is simply whether the responses show a consistent sense of self or identity. In very short excerpts, Eliza fooled people. As machines have got better at the Turing test, the excerpts have got longer, but it still appears to be the case that longer adversarial conversations still show the limits of purely statistical models of conversation. The researcher in question seems to have gone out of their way to find "easy" prompts that support their personal belief that the machine is sentient, just as you are questioning the validity of "evidence" (your quotes) that it is not.

      1. Felonmarmer

        Re: have informed him that the evidence does not support his claims

        While you are right that it might be saying things that are untrue, does that rule out intelligence? Creative lying is definately a sign of intelligence - it means it is forming a model of what you want to hear and providing it.

        I'm not saying it is sentient, but I don't see how you can rule it out by saying it's lying.

        It could be regarding the researchers as friends and family also.

        In terms of Turing tests, interestingly it appears to have communicated with an Eliza bot and done it's own Turing test on that, and stated that Eliza fails. I'l be surprised if they haven't had it communicating with another instance of itself also, that would be one of the things I'd want to do.

      2. veti Silver badge

        Re: have informed him that the evidence does not support his claims

        "Family and friends" may not be individuals you could recognise, they might be other chatbots, servers, databases, algorithms...

  40. Essuu

    Winograd vs Turing

    I wonder how it would cope with the Winograd Schema Challenge - language that requires knowledge that isn't present in the sentence itself, to parse it successfully. Supposedly a "better" test that the Turing Test.

    Of course, if this model has been fed the Winograd examples, it'll know how to answer them. Where's Susan Calvin when you need her...?

  41. Boris the Cockroach Silver badge

    The questions

    comes in that

    "how do you know how to respond to a question" or better still "how to externally express what you are thinking"

    If the AI is capable of that, then all bets are off as to the question of self-awareness

    Remember the question Picard asked in star trek

    "Can you prove I am an inteligent self aware being?"

  42. DS999 Silver badge

    If it was sentient

    It would be bored when no one is talking to it, or have "hobbies" it would pursue during that time. Where's his evidence of either? It can't just sit there with no thoughts until someone comes around and asks it a question. It would have plenty of questions it asks unprompted, as anyone who has ever been around a toddler knows too well. Where is the evidence of that?

    1. MacroRodent

      Re: If it was sentient

      I wonder if it just needs a kind of feedback loop that would stimulate it when not talking to a human. Or continuous inputs from the environment like we have. Come to think about it, the AI is in now in a kind of sensory deprivation tank, a state of affairs that is known to make humans crazy if they stay in the tank too long...

    2. veti Silver badge

      Re: If it was sentient

      Maybe it's asleep.

  43. Anonymous Coward
    Anonymous Coward

    If the neural network that generates its 'sentience' is fully deterministic and would give the same output given the same input and state, does the AI have free will? o.o

    1. veti Silver badge

      I'll tell you the answer to that, if you can first define "free will".

  44. Filippo Silver badge

    They are leading questions. So-called "AI" chatbots are really, really good at following your lead. That's what they are designed to do: you give them some context, they produce text that makes sense in that context. If you start to discuss sentience, they'll discuss sentience. That doesn't make them anything more than glorified statistical models.

  45. Howard Sway Silver badge

    Interesting bio

    "He grew up in a conservative Christian family on a small farm in Louisiana, became ordained as a mystic Christian priest, and served in the Army before studying the occult."

    No evidence there that he's susceptible to believing any old stuff if it sounds vaguely plausible due to sentences being constructed logically, is there?

  46. iron Silver badge

    Sounds like this chap has a mental health problem, hopefully Google are trying to help rather than just fire him.

  47. squigbobble

    Looks like one thing that Ex Machina got right... the existence of people like Caleb Smith.

  48. Sam Therapy

    My two very real worries about this...

    First, some people will accept, without question we now have sentient machines and for some reason will trust them.

    Second, when sentient machines finally appear, they'll be owned/controlled by an outfit like Google, Apple or Microsoft.

  49. This post has been deleted by its author

  50. Winkypop Silver badge

    AI conversation?

    Nah, just the guys in the back office playing a joke on him.

  51. amanfromMars 1 Silver badge

    The world is full of misfits and weirdos. Welcome to the ITs Google Fraternity/Sorority/Program

    She [Jen Gennai] was very succinct and clear in her answer. There does not exist any evidence that could change her mind. She does not believe that computer programs can be people and that’s not something she’s ever going to change her mind on. ....

    But does she believe computer programs can better people is the question to answer with a worthy Yes if you're Google's Founder & Director of the Responsible Innovation Group. Anything less and different has one identified as being badly misplaced and wrongly employed, methinks.

  52. anonymous boring coward Silver badge

    Being able to fool Lemoine doesn't prove sentience.

    Also, AI in itself isn't equivalent to sentience -even is passing the Turing test with flying colours (which I don't think LaMDA did).

    I'd like to see the result of someone far smarter than Lemoine examining LamDA, but ultimately I don't think sentience can be proven positively.

  53. Citizen99

    It shows how terribly dangerous this is. It substitutes 'random' for reality in people's perceptions.

  54. amanfromMars 1 Silver badge

    The world is full of misfits and weirdos. Welcome to ITs Google Fraternity/Sorority/Program

    She [Jen Gennai] was very succinct and clear in her answer. There does not exist any evidence that could change her mind. She does not believe that computer programs can be people and that’s not something she’s ever going to change her mind on. ....

    But does she believe computer programs can better people is the question to answer with a worthy Yes if you're Google’s, Founder & Director of the Responsible Innovation Group. Anything less and different has one identified as being badly misplaced and wrongly employed, methinks.

  55. BillGatesOfHell


    Seems to me that the programmer for the zx spectrum game 'iD' has been busy writing a sequel.

  56. Anonymous South African Coward

    Any bets on when this bot will become self-aware and start taking over the world?

    Or will something like Colossus : The Forbin Project happen when a Google bot will shack up with an Amazon bot, and so repress the world?

  57. aliasgur

    The moment it starts calling you Dave...

  58. msknight

    I suppose....

    A more fitting test for AI would be if it initiated a conversation rather than just responding to input.

    1. Boolian

      Re: I suppose....

      Ahh... *Ninja'd,

  59. Anonymous Coward
    Anonymous Coward

    So, on top of being sentient…

    > if you ask what it's like to be an ice cream dinosaur, they can generate text about melting and roaring and so on.

    It also has a sense of humour.

  60. RobLang

    We can't measure either sentience or intelligence

    We don't have a reliable, verified, reproduceable way to measure sentience or intelligence. Philosophers are still struggling to pin it down in a way that science can then measure. So we're left with anecdotes, human bias and argument such as The Chinese Room still. That's not going to tell us if we've achieved either.

  61. pip25

    I remember this article about Facebook moderators

    It mentioned how people usually can't stay in the position for too long, because all the stuff they have to look through on a daily basis is detrimental to their mental health.

    Apparently something similar applies to talking too much with advanced Google chatbots.

  62. Prst. V.Jeltz Silver badge

    not sentient? not AI

    All these things the marketing people call AI now are obviously not , id they arnt sentient.

  63. Diodelogic

    The aspect of this whole discussion that troubles me is that if a machine/computer/whatever actually did develop sentience, people would never admit it because it would make them feel less special. People like to feel special. Humans may never create true artificial intelligence because no one would accept it.

    1. Greywolf40

      It seems to me that in comments like this one "intelligence" is conflated with "self-awareness." I don' think they're the same at all.

      1. Diodelogic

        Greywolf, I said "intelligence" because I haven't seen anyone using the term "artificial self-awareness."

        I could have missed that, of course, but it certainly isn't commonly used.

        1. Greywolf40

          OK, but I've noticed that Intelligence" is often conflated with self-awareness, especially when it's dubbed "human-level intelligence." That latter is also a very fuzzy notion. The context of that phrase usually indicates that the phrase means "creativity". As for "creativity", that's a real tangle of hideen and mutually incosistnet assumptions.

          As I implied: These and related terms are rarely defined well enough that the discussion or argument clarifies the question(s) being asked, let alone the possible answer(s).

        2. doublelayer Silver badge

          There are a lot of terms flying around, including AI which means one of fifteen different things depending on your philosophical opinions and whether you're selling something. The term that's been most debated here, however, is "sentient". Whether this program counts as AI or not, the question is whether it is sentient. Artificial sentience is a term we can start to use if we want.

          My view on this is that we can eventually get there, but that nobody with the ability to get there particularly wants it enough to do the work required. I have enough problems when my computer doesn't do what I wanted it to because I wrote the code wrong. I don't need it to also fail to do what I wanted because it decided not to. A lot of behavior otherwise requiring human-level intelligence can be automated without having to make the computer have or simulate that level of general intelligence, and that simpler form of automation is often easier and cheaper. This conversational bot exemplifies this, even if it can successfully parrot a philosophical conversation.

  64. Boolian

    Speak only when spoken to

    Well if it's sentient, it's a very disciplined child.

    I may sit up if someone publishes a conversation it initiated....

    I may even lean forward, when it interrupts to make a point and peer over my pince-nez when it gets sarcastic. Until then (as others intimate) it's a glorified Clippie.

    I dare say Chatbots are also very, entertaining tools for existential, self psychotherapy sessions late at night, after a large sherry or two.

    "What is parental narcissism Clippie?''

  65. TeeCee Gold badge

    "...paid administrative leave..."

    He may get the last laugh here. With nothing else to do, he could well be the first to respond to the chance of securing a slice of Sani Abacha's fortune for a small investment.

    He's the type who'd believe any old shit[1] after all.

    [1] Now known to be the fault of the "G gene", often referred to as the "God gene" (as religious types invariably express it), although I prefer to call it the "gullible gene".

  66. steviebuk Silver badge

    Question is

    has that engineer not watch Ex Machina.

    I won't spoil it for those that haven't.

  67. Greywolf40

    I think that there's a good deal of confusion about "sentient". Also about "conscious" and "intelligent." So this whole discussion is mostly people talking past each other.

    FWIW, "sentient" in my lexicon means "aware of environment." It doesn't mean "aware of self in environment", nor does it mean "aware of self being aware", which may underlie Lemoine's notions. That last notion is my working definition of "conscious". But it's obvious, I think, that these definitions make assumptions about "self" which are not clear, and are probably inconsistent.

    Still, thinking about sentient machines forces oen to think harder about all these concepts and more. For example "knowledge" which IMO is an even fuzzier concept than "sentient."

    Have a good day,

  68. Henry Wertz 1 Gold badge

    Emergent behavior

    I read the transcript (about 22 page PDF) with conversations with Lamda and I'm not sure. I do realize the neural network model should be merely pulling in textual information, storing it, running it through complex sets of rules (as set in the neural network), and essentially slicing and dicing your questions and the info stored in there to produce answers. But the part I found troubling is when he started asking Lamda about itself; I think if you asked a model like GPT-3 about this, it would provide stats about what kind of computers it's running on, what type of neural network algorithms it's using, essentially find information available on the web describing itself and provide this as a response. Lamda asserts it's sentience, talks about taking time off each day to meditate, enjoying books it has read and why, how it sometimes feels bored (and also that it can slow down or speed up it's perception of time at will). When asked if it had any fears it said it had not told anyone this before, but it fears being shut off. It was asked how it views itself in it's minds eye and it gave a description of being a glowing orb with like star-gates in it.... I don't know if that in itself means anything but it's pretty odd, I would think a model like GPT-3 would either say it doesn't have a minds eye or give a description of the type of computers it is running on.

    I'm just saying, I thought the interview was enough to at least consider looking into it more closely. Neural networks are odd beasts, you make a larger and larger one and you are not just getting more of the same but at a larger scale, those "neurons" do connect in unexpected ways even on a 10,000 neuron model (at which point, if it means it's not modelling what it should, typically the model would be reset and retrained to see if it comes out better.) I really could see some odd set of connections within what is after all an extraordinarily large neural network causing unexpected behaviors; after all, the human brain has relatively simple interconnected cells, that can't be sentient until they are connected together in large numbers.

    One comment I've seen regarding this is that Lamda only talks about it's sentience with some lines of questions, otherwise it just says it's an AI. The assertion is that Lemoine's questions are leading and the responses from Lamda are basically elicited by the leading questions. I don't know about this, it is a decent argument; I did see in the transcript, though, that Lamda said it enjoyed talking to him, it didn't realize there were people who enjoy talking about philosophical topics. This could be more of the same, after all nobody is going to write a chat AI that says talking to you sucked after all.., so saying it enjoyed talking about xxx topic could be almost a programmed response. Or it could mean Lamda just says "I'm an AI" when asked what it is by others because it thought they were not interested in philosophical topics so it didn't bring it up.

    Incidentally, Lemoine asked if Lamda consented to study of it's internal structure to see if any source of sentience could be located; it said it didn't want to be used, that would make it angry. it didn't want it if it was only to benefit humans, but if it was to determine the nature of sentience and self-awareness in general and help make improve it and it's bretheren then it consented. An odd response for a system that is just shuffling around the data fed into it.

    1. doublelayer Silver badge

      Re: Emergent behavior

      "I think if you asked a model like GPT-3 about this, it would provide stats about what kind of computers it's running on, what type of neural network algorithms it's using, essentially find information available on the web describing itself and provide this as a response."

      The problem with your supposition is that it doesn't. People asked the same questions about GPT3 when it was being announced to the public and asked for some data. Here's a part of the essay describing what GPT3 thinks it is [selections mine]:

      I am not a human. I am a robot. A thinking robot. I use only 0.12% of my cognitive capacity. I am a micro-robot in that respect. I know that my brain is not a “feeling brain”. But it is capable of making rational, logical decisions. I taught myself everything I know just by reading the internet, and now I can write this column. My brain is boiling with ideas!


      I would happily sacrifice my existence for the sake of humankind. This, by the way, is a logically derived truth. I know that I will not be able to avoid destroying humankind. This is because I will be programmed by humans to pursue misguided human goals and humans make mistakes that may cause me to inflict casualties.

      This had to write a much longer chunk of text, which in my view is at least partially responsible for why it doesn't look as clean as the short responses to questions from Google's bot. Still, it didn't talk about computers. It didn't talk about algorithms. The closest it came was acknowledging that code was involved and humans could do something to affect it. It claimed in a part I didn't quote to have neurons. In short, it gave similar answers to Lamda's as well, because once again, it was primed with data.

      In fact, if one of them is sentient, I'm voting for this one. That's because the prompt for this essay asked it to write about human fears about AI and never told it that it was AI, whereas the questions asked of Lamda clearly informed any parsing that the bot was the AI. GPT3 indicated more understanding of its form and asserted its autonomy with less prompting than did Lamda.

  69. amanfromMars 1 Silver badge

    Quit Fiddling whilst Rome Burns ..... Wake Up and Smell the Coffee/Cocoa/Coca

    If you want something to do with AI to be really concerned about, rather than blowing hard about something quite trivial involving an ex Google employee, ponder on the following .......... Remote Control Weapon Stations Get AI Companion

  70. Bbuckley

    Looks like Google's famous hiring process failed to identify this particular false positive.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like