It all depends....
My straw poll of 4 female colleagues suggests that women couldn't care less whether a man has a beard and that it doesn't signify anything with regard to sexism. However, there was one important caveat: if they had felt inclined to get up close and personal with aforesaid man, 3 out of the 4 would care: there's a thing called 'beard burn'.