Re: Neither unquantifiable nor undefineable
"Also, your posting (Possibly unintentionally, feel free to clarify) seems to frame a binary choice of opinion on if we CAN achieve AI, especially forms of general AI."
I'm not quite sure how you read that into my comment.
In fact I probably agree with you. My feelings are that we will, incrementally, approach levels of machine "intelligence" (in various directions) which would be generally recognised as such, but not necessarily resemble human intelligence that closely.
A more nuanced view, is to ask what we are actually trying to build AI for. What's our agenda? If it's human simulacra, then I think that's way, way off anything even nearly achievable (in my lifetime and probably beyond). Or is it engineering - building machine intelligences to address (human) problems?
"I also don't buy some of your assertions on definitions."
I think I did say that not everyone (including my scientific peers) is likely to agree with me on this one :-)
"They are pretty essential in most cases, ..."
But are they? My example of the 19th century development of our understanding of electromagnetism was intended to confront that assertion. Bear in mind that electromagnetism (or in fact electricity and magnetism, before their linkage was recognised) may well, to the early 19th-century mind, have appeared every bit as mysterious and mystical as sentience or consciousness appear to us today. But the resolution was not achieved by pinning down definitions. Likewise Darwin and Wallace's elucidation of what "life" actually is/means was not based on definitions (no élan vital required!).
" ...but there is a real pitfall in trying to pin them before you really understand the idea or system you are defining."
Very much the case. As an illustrative example, the 19th-century idea of the "ether" arose, one might say, out of a desire to define "what space is". It turned out to be wrong - and, tellingly, Einstein again did not in fact arrive at relativity theory by pondering what space(time) "is". He did not try to define it. Rather, he successfully discovered through reason and thought (as well as physical) experiment how it behaves, the role it plays in the physical world.
Definitions, to my mind, are -- on the basis of historical observation --- certainly essential in mathematics, but not so much (or even a hindrance) in science.