It seems like you're taking from the phrase "male privilege" an accusation that this is somehow men's fault. I think the other commenter said it perfectly well, just because a privilege is afforded to a group doesn't mean that group is responsible for that privilege. It's still a privilege that they have.

In your example, wearing dresses might well be considered "female privilege", yes. I doubt you'd hear this talked about because it's quite minor, but you're not wrong. Another example which illustrates it well is "pretty privilege" - we all know that life is easier for attractive people. That doesn't mean they are oppressing average or unattractive people personally. They just have advantages. Not their fault, still a privilege.