It used to be on ROM
Back in the old days, "firmware" really was "firm" because it was blown onto a ROM and could not be changed. Unless you physically read the ROM, you would never see those bytes. From the end user's point of view it might as well have been wires. After all, even CPUs had microcode, and nobody got upset about not being able to change the microcode; it was just part of the hardware.
The Free Software Foundation says it starts caring at the point where the bytes can be changed post manufacture, i.e. the "firmware" can be updated. Obviously a boundary has to be set somewhere, but it's hard to understand why this exact boundary: what if the updates are not compulsory and you could carry on using the first version of the firmware (or the version that was current at the time you bought the device) if you want? Would that not be equivalent to having brought it in a ROM as part of the device? Should they perhaps refine the boundary to say they start caring at the point where the updates become essential, for example because they are security fixes, instead of just saying they care about bytes that could in theory be updated but don't have to be?
Biting the hand that feeds IT
