Re: The security folks will say...
I agree, although it depends on what sort of shell is running. Is it a standard bash shell or is it running some custom application instead, which doesn't allow any access to the underlying OS, let alone uploading code?
If you are sshing into a full bash shell, the supplier will need to provide a patch. If you are sshing into a menu driven configuration program with no ability to upload code, then you probably don't need to patch. To exploit the latter you would first need a zero day buffer overflow of some sort to gain any access to the underlying OS, in which case, Meltdown/Spectre is the least of your problems.
As you say, if there is a patch that affects performance, but you can guarantee that no external code is run on the device and you only log on once a year, you can decide for yourself, whether the risk of not patching is worth it.