It would cause riots
Many people seem to be getting bent out of shape by the thought of having a few people coming here from Syria. Imagine the fuss if anyone from another star system were to visit. There would no doubt be a referendum and we'd end up voting to leave the Galaxy.