Re: Abuse, plain and simple.
"my contract has always included language preventing me from working, directly or indirectly, for that customer for a couple years or so"
It sounds like you have a badly written an IR35 caught contract, at least in UK terms.
The contract should be between ClientCo or AgencyCo and YourCo not you. As someone said in a previous comment, start YourCo2 which never had such a contract.