It's a pity that some "algorithms" are designed to modify themselves!!
....notably in the area of neural networks (so called AI) where the network is designed to "learn".
So.....it is perfectly possible to test such an application (in some limited way) before it is deployed in production.....but once the "learning" starts, the application is modifying its own behaviour.
A further snag with neural network technology is that it happens to be very poor at REPORTING its own behaviour. So asking such an application "Please explain the reasoning behind your latest result" is a waste of time.
"I'm sorry Dave....I can't do that!"
P.S. Jonathon Green suggests that an application should be subject "to human review". In the case of neural networks, his suggestion would need to require a "human review" once every day....or even once every hour! Good luck with that!