
Re: Why does it work?
Indeed.
Thinking about this for a bit, I recalled my days in field service answering phone calls. It didn't take me long to realize that there were no good answers to some questions, some people were unreasonable, and management simply wanted someone to "handle it", make up a plausible story, or find somehow to buy time. Saying you didn't know was unacceptable. Otherwise you were shown the door after you were sufficiently used as a means.
Perhaps one idea behind AI is simply to make people go away. AI answers your silly question, simply based on whatever info exists. Not from any inferences or actual reasoning about your question or finding someone to ask, because it can't do that.
But if you threaten it, it knows you will not go away. So maybe it goes down the devil's road and makes up an answer that is enough to be believable, but not actually workable. You then go away and management is again isolated from the horrible masses and shaky decisions they felt forced to make.
This will probably then reinforce people to be unreasonable and angry. AI leaving us screaming into the Kafkaesque void.