Re: Over-confidence
Because it's not reasoning an answer. It's guessing a likely response from information it's been trained on. Look up discussions of Atari 2600 chess and you'll find lots of comments about how it wasn't exactly a mastermind like:
The strenght is not half bad all things considered, and it certainly seems to be somewhat stronger than Microchess, but maybe not by much. But any decent chess player should be fully capable of beating this game across all levels.
[
source]
And find similar statements about modern chess engines like:
Even the most ordinary cellphone processor's can easily beat any human 100 times out of 100.
[...]
Short answer to your question, To avoid detection, cheaters play worse than the engine's full capacity and this can result in them losing the game.
[
source]
Then add in the prompt that Gemini receives which tells it that it is an AI, trained with neural networks, lots of data, etc, and compare it to this more modern page with excellent scores for many computer chess solvers. It's got lots of basis to assume that a modern computer will beat a basic system, it's got basis for saying it is a modern computer, those words get linked together with high likelihood and a confident answer pops out.
The same thing happens when it "changed its mind". It got told that something similar to it failed, and the prompt includes lots of data intended to get it to not stay confident in a wrong answer, so it put those together and the "actually, I'm crap" response pops out. That can happen in lots of scenarios. For example, it's not very hard to get an LLM to engage in a conversation that goes like this:
User: Asks a question.
LLM: Gives correct answer.
User: Are you sure?
LLM: No, I hallucinated that.
It's gotten enough prompting that responses challenging it are often accepted, whether or not it's wrong. There's a lot of fragility, because depending on how it's challenged, it will usually give either "Definitely, that was correct, and I can tell you more" or "Sorry, I got it wrong, AI has weaknesses, please try again", mostly unaffected by whether the answer being challenged was right or not.