So would any agency actually want you outside of IR35 since it would adversely affect their own profits
The problem is that it's not the agency who decides. All the big agencies had turned around and said they would work hard to determine the true status of their contractors fairly, then the govt decided that it would be the public sector client who would decide. The agency has no say, it's only role would be to collect the taxes.