You're either an employee. Or you're not. If you are caught in the middle it means you're working *as* an employee, for a sole employer, for an extended length of time, and get all the benefits of that, while also trying to not pay the tax that you would have to if you were an employee. And you do this by pretending that you're contracting round when actually you're working for one place, at one time, for - say - one year, and just don't want to be an "employee".
IR35 should have caught up with you years ago, for sure you've had plenty of warning that it was happening.
Work as a contractor, without employee benefits or restrictions, or become an employee and pay your taxes.
If you're working "as a contractor" and this hurts you - get employed by the client, or raise your prices.