Re: Today’s discussion topic
The output looks good but you would be foolish to trust anything in it that resembles a fact.
Since this is also true of most humans, I'm not sure why you think it's an interesting thesis.
Like LeCun, I'm not particularly impressed by ChatGPT. I know enough to know it's not particularly exciting. In the realm of LLMs, I like the bidirectional BERT family more than the unidirectional GPTs, and even for the BERTs I feel a bit meh. Yeah, transformers are a bit more elegant than some of the previous work, but I want to see something more like the heterogeneous-competency architecture of EfficientZero in a language model. And something rather more resource-efficient. (Also, as always, what the public sees is already some way behind what's coming out at conferences and on arXiv...)
But I'm also tired of the ignorant hand-waving dismissals of LLMs and other big ML projects. Arguments like "it's just a statistical model" are irrelevant; a statistical model can be a non-deterministic UTM and so compute any computable function (assuming the classic CTT), so if you want that argument to count for something, show me the human CNS is strictly more powerful than a UTM. I'm not holding my breath. Or "it just regurgitates what it learned": show me that's not true for human beings.
I've studied and taught composition (writing) and rhetoric at the graduate level, and I've studied natural language processing and machine learning at the graduate level, and while I certainly don't know nearly as much about these areas as someone actively working in them (who could keep up otherwise?), I'm pretty confident that the vast majority of comments I see about LLMs and similar – whether fawning or dismissive – are just as much "shiny bullshit" as the output of those systems.
Biting the hand that feeds IT
