Re: AI != Human
I agree with most of what you say (have a vote). Artificial Intelligence is not Human Intelligence. Where I disagree is that there should be no expectation of analogous behavior.
Chat bots are based on neural networks, which are opaque feedback loops that somehow store information and allow the information to be retrieved for problem solving goals. Computer neural networks were originally created to test the theory that this is how the mammalian brain stores information.
Chat bots are designed to generate human-like responses. They are more than just clever code, which seems to imply a complex decision tree. We cannot demand a proof from a neural net the way we can from an inference engine like Prolog. The way the neural net comes up with an answer is opaque.
To the degree we are using chat bots to simulate human responses, we should expect problems analogous to human foibles and that they may be susceptible to being tricked by the hacker equivalents of hypnotists, mentalists, and magicians. It won’t be the same, but it may be strikingly similar and may offer opportunities to learn about ourselves.
For instance, while chatbots hallucinate, humans are susceptible to recovered memories, misremembering, and false memories.