What was good about Robby in forbidden planet
Was that in the event of a contradiction in programming the default action was to
1. Stop work completely
2. Start damaging itself so as to make conflicting orders financially expensive for the operator
3. Allow for contradiction state to be cleared but not remove the conflict
This way the operator is the one who is ultimately responsible, if action 1 results in damage then operator did it. Action 2 is the punishment for bad orders and 3 gives the operator a chance to avoid bad outcomes but over use results in same punishment.
Thus, in this model, the vendor's responsiblity is limited to defining the contradictions so any resultant damage is the operator's fault i.e. "we tried to stop you doing it but you insisted upon that action"
I am not saying it is possible to define all the contrdictions but so long as they are included as discovered then vendor has made all reasonable efforts and it is how human law works
Biting the hand that feeds IT
