Re: self-signed CA
As it's not one vulnerability, the answers to your questions can vary.
"What can actually happen to me or my system and how?"
The standard risk is that a certificate that your system trusts can be used to impersonate something else, either breaking encryption on a connection of yours or diverting that traffic. This requires that the attacker has malware on your system or controls a link in your network. Malware is the more likely option here, though it must be noted that local malware could do nasty things to ongoing connections anyway. Such certificates could also theoretically be used to make your system trust a binary that isn't signed with a key your OS provider normally would trust, although the paper is talking mostly about network certificates here so that's probably not the risk they have in mind.
"How can I protect against such 'bad' events?"
You could analyze a lot of your typical traffic and see if anything unexpected happens, but that will likely take a long time. Since the primary risk is malware, scan your system frequently to eliminate it should it become known.
"How can I check my systems to see if they are vulnerable and/or compromised already?"
Most things that have root certificates stored somewhere will allow you to view and edit the list. You could go into your network configuration (search for the instructions for your OS) and start auditing your certificates. The problem here is that, when you find something in there that doesn't appear on a list of trusted certs, you may not know where that came from. You could look for some of the red flags discussed in the article such as long cert lifetimes or you could delete the cert (keeping a backup copy) and see if things break. This takes a lot of time and effort and may not be needed in many cases.