Base it on time in contract
IR35 has been a mess since it was introduced. It is so subjective.
Why not have simple rules that say "if you have worked for a single client for more than x months then you pay full tax" and an extra rule to stop leaving for a month and rejoining. As far as the value of x, then 24 seems fair.
It is wrong that there are contractors who have out lasted all of the permanent staff. I worked with one chap who had been a contractor for 20 years at a Govt dept and previous to that was employed for 10 years. So hardly taking any risks.