Unverifiability: Welcome to the Real World, where things are more complex than they seem.
That's why we have statistics.
A book from 1997:
...it's actually a book about statistics and running experiments on your new AI-gorithm to see whether it actually is as good as you think it is.
(Also look out for the Black Swan, i.e. the falling-down-the-stairs-HAL-9000-level breakdown that you didn't even know was coming)