Re: :(
As a shareholder, technically you're not obligated to loan your shares to a short seller. If you're so inclined, you can request certificates of your shares of a company. That'll prevent your broker from being able loan your shares. It'll also completely slow down the whole process of selling your shares if you need to do so urgently, so it comes with pitfalls.
And, as you say, if the short seller does shit on the listed company and expose a truly hollowed out shell of an entity for the world+dog to see, every shareholder who held those shares, including yourself, failed to take the required due diligence before investing and/or failed to continue in said due diligence in that company and that's going to be a reminder of at least three core principals of investing.
1) Never invest what you can't risk losing completely.
2) Understand what you're investing in, continuously.
3) Diversify, diversify, diversify
Let's say the obverse happens, taken to an extreme - not only did the short seller write a fraudulent report, but also violated SEC regs and/or other laws. The company responds with proof against all the questions that short seller wrote and steamroll them with your truthful responses and counterpoints and the short seller report writer is investigated for SEC violations and/or criminal conduct and is exposed for what they are.
The stock may take a little ding while the dust settles out quickly, but if you're in it for the long haul and you have a strong conviction in the traded entity, then you've got nothing to worry about.
At the end of the day, the market's forces correct themselves justly. It may hurt if you're on the wrong side of the ledger and weren't prepared for it.