Re: Being able to make the Distinction
Here's my personal theory, arrived at after taking a "Mind as Motion" aka "neuroscience 101" module as part of a Cybernetics MSc, and er, some interesting mushrooms:
Our neurons, as I learned on my course, are kept forever on the very-edge of firing. This is done both by self-regulation of electrolytes within the neurons, and also the neurons continually optimising their connections to other neurons i.e. synapses, so that they don't fire too much or too little. If a neuron fires continuously, it runs out of energy, but it adapts itself to need a higher threshold before firing to avoid doing that in future, or else it would die. If all your neurons fired at once, you would have a seizure, and perhaps die. When they do fire, it is because that incredibly finely-balanced chemical mix inside the neuron, influenced by all of its synapses, has tipped over a threshold that causes a positive-feedback cascade to occur, turning it momentarily into an electrochemical battery and passing that pulse along to connected neurons, which in turn only fire if they get exactly enough input from other neurons to tip them over their self-determined thresholds.
They are extremely complex systems, so much so that our biggest supercomputers couldn't fully simulate a single neuron at molecular scale in real-time. So my professor claimed in 2013, at least.
Now, because these neurons are so finely tuned by self-regulation to be just on the very border of firing or not-firing, and because they receive multiple inputs from different synapses that each add an analogue signal (i.e. the concentration of various electrolytes inside the neuron) - the difference between firing or not-firing could be down to the presence of a single molecule - and the difference between the production or non-production of that single molecule at a synapse could be down to an unknowable "Quantum" property e.g. the spin of an electron.
Thus, in trying to stay exactly on the infinitesimal border between firing or not-firing, the biological neuron could be optimising itself to amplify quantum properties. And complex behaviour of a unique system that is governed by unknowable quantum properties is something that could perhaps be called a soul.
LLMs on the other hand, are identical buckets of bits running on deterministic silicon. Every instance of an LLM is the same pile of statistics, i.e. the same pattern of connections and weights, with randomness added only as noise in one or more of its layers. Even if that randomness could be quantum randomness, it cannot adapt its own weights on the fly, and both its weights and state are known. If you could show me an LLM where each INSTANCE had its OWN weights, that were all quantum-unknowable and self-adapting on-the-fly, then I might be able to call it something like alive.