Your forgetting something
Current thinking in cosmology is that for the cosmic background radiation to be as uniform as it is the early univese had to have expanded faster than light propogates. Its called "Inflation".
Einsteins equations allow for the fabric of space to expand faster than light, just not for objects to move faster than light relative to their local frame of reference. And Einstein's numbers also allow for an object to travel faster than light IF it isn't moving relative to its space-time location, that is instead of moving the ship you move the bubble of space-time AROUND the ship by expanding space on one side and collapsing it on the other. Look up " Alcubierre drive" for the details.
So, FTL drives are not possible using current tech, but in a thousand years, who knows.
Also, your comment makes the classic assumption that Einstein is 100% correct, which he isn't, since his theory's can't account for gravity at the quantum scale, or things like singularities without some fancy number dancing.
Your right in the following, IF interstellar travel is impractical then the 'Fermi Paradox' ceases to be a paradox. After that you start making assumptions that are just as suspicious as those you criticize.