AI revisited
Along similar lines re computers "learning" the wrong thing. I once heard a story about war game software related to convoys and navy escorts*. One of the important things about a convoy is that it travels at the speed of the slowest ship in the convoy. The navy escort of course has weapons and can shoot and sink ships - to guard against attacks on the convoy by the enemy.
In one simulation of an enemy attack, the navy ships started shooting at the slowest members of their own convoy, causing them to sink, and thus speeding up the whole convoy.
Not quite the real world example one wants.
*Might have been US software.
Biting the hand that feeds IT
