Re: "Whether we burn $500 million a year or $5 billion – or $50 billion a year – I don't care"
"Yes, exactly. Sorting them physically into piles, is a known strategy to cope with human well-known cognitive lack."
Which is why I referred to counting things in a microscope. Biologists are not picking up and sorting cells (or whatever it is they're counting).
"Wow, you’ve fallen right into the anthropocentric trap. Why is counting 3 items so much easier for a human? Because the human brain literally has specific circuits for counting in blocks of 2, 3, 4 and 5."
That's not the point. If there are eleven items, which is neither in that list nor an even multiple, it still won't take as long as 237. Quantity is the problem if the question is losing track, and you are the one who brought up the losing track prospect. Of course, a computer has hardware which is great at not losing track, so a program that was able to make use of that to perform cognitive tasks should be much more reliable than a human would. This is all irrelevant because the computer is not trying to count, as you said yourself:
"You just need to understand the tool you have; and want to succeed at the task in hand, rather than trying to catch it out."
Which I will extend "You just need to understand the tool you have; and what it can do (generate some text) and what it can't (guarantee that the text is at all relevant or correct)". That is the only way you will know whether your tool can let you succeed at the task in hand.
Because yes, it can probably write the program "strawberry".count("r"), but with many other tasks, your suggested solution wouldn't work either. Whenever it has to write a program that's a bit more complicated, it fails and badly. For example, I asked it to write a Python script that would count characters from a larger set. It gave me a valid Python program that counts characters. From another set. I asked it to count the number of characters that were in the lowercase alphabet and a small set of punctuation. It gave me a program that counted letters in the lowercase or uppercase alphabet and no punctuation at all. When I pointed this out, it gave me a new program which had the following criteria:
1. The punctuation I asked it to count were mostly not counted.
2. It told me I needed to import the re (regular expression) library, but never used it.
3. It would now count some characters in Unicode code ranges that didn't include the characters I asked for but did include several others.
4. It stopped counting any letters.
5. The only character from the example string that was counted anymore was ".".
But sure, that's a trivial strategy that works flawlessly.