OK you have raised in my head the possibility of an entity being intelligent without being aware of it, and as I am not a philosopher any more than a mathematician I'll just have to bail.
BTW your description of an ML system (training and tuning, or lack of, by humans) does little to alter my opinion of "LLMs" as overblown "Bayesian" spam filters that for some reason seem to require power inputs that threaten the environment itself.
I am angry at Google (et.al.) because of their unrelenting encroachment on our very existence, but angrier at those who wilfully refuse to accept that it is a problem (i.e. the general public, not even the governments, as that is what governments do and is a separate issue!) and just keep lapping up the crap.
Hence my Lovecraft reference earlier . . . I should have just left it at that!