"I think it's fine to say cocktail casual"
So... specifying that women are to wear a short dress and heels IS fine, as long as you don't state it clearly and distinctly but instead use some euphemistic formulation that people not in the know might misunderstand and misinterpret?
Make your f**kin mind up love.