The amount of nonsense in this paper is remarkable
All quotes are from the actual paper.
- "If forced to type a random character sequence, their speed drops precipitously."
Someone will have to tell these people that keyboards have a certain lay out for a reason and that subsequently it's not physically possible to type random sequences as fast as sequences in the targeted language of the layout. Random character sequences are bound to be inefficient and hence slow you down.
- "If the guesser wins routinely, this suggests that the thinker can access about 2^20 ≈ 1 million possible items in the few seconds allotted. So the speed of thinking – with no constraints imposed –
corresponds to 20 bits of information over a few seconds: a rate of 10 bits per second or less."
This is of course ludicrous. In the most efficient storage method possible we only need 20 bits to enumerate 1 million items, so as you've supposedly only generated 20 bits of relevant data, the "speed of thinking" is 20 bits/s?!?
- "This dilemma is resolved if I give you a specific task to do, such as typing from a hand-written manuscript. Now we can distinguish actions that matter for the task from those that don’t. For example, two different keystrokes are clearly different actions. But if you strike the key in 91 ms vs 92 ms, that variation does not matter for the task. Most likely you didn’t intend to do that, it is irrelevant to your performance, and it is not what I want to capture as “richness of behavior”."
Probably one of the most blatant admissions of cherry picking possible. Why not just measure something where a lot of combinations of factors DO matter? E.g. tell me what the bitrate is for Chopin's prelude in F# minor (Opus 28 #8). How did they get to the point where they expected "fair" measurements of speed by crafting very biased tasks? Note: of course even in typing a lot more things do actually matter than just hitting a key. For example the amount of force required and how deep the key travels has quite a bit of impact on a satisfactory end result.
- "So the discussion of whether autonomous cars will achieve human level performance in traffic already seems quaint: roads, bridges, and intersections are all designed for creatures that process at 10 bits/s. When the last human driver finally retires, we can update the infrastructure for machines with cognition at kilobits/s. By that point, humans will be advised to stay out of those ecological niches, just as snails should avoid the highways."
Given how poorly cars perform in traffic, they might want to hold off from having cars perform in an environment that has entropy requiring for "human level" kilobits/s.
All current attempts at creating AI are clearly also failures. Cause we can for example measure an LLM by it's ability to produce tokens and translate that to bits/s. To do this it reads from memory at hundreds of GB/s and needs to do trillions of calculations but that's entirely not relevant based on how the paper measures things. And sure enough, if you throw enough at it, it can outperform a human in some tasks but that's before we consider efficiency. The estimate for the human brain is that it uses 20W a day or on average less than 1W an hour. Good luck getting anywhere near human performance at 1W an hour.
How did anyone read this paper in advance without suggesting a bit of a rethink before publication?
Biting the hand that feeds IT
