"The only potential problem is that this may affect performance. ..."
Tell me about it.
When Debian released fixes for SPECTRE and MELTDOWN the performance of the E3815 processors in my customers' Intel NUCs fell off a cliff.
Not just a few percent, not even a few tens of percent, but between two and three orders of magnitude!
The users would click an icon and then go have coffee to wait for the response.
To make the devices usable again I had to compile custom kernels for them.