I think it may be related to the lore
Humans in this setting are said to be born of the Erdtree itself (at least for as long as Marika has been a god) and the lore implies that sex isn't even something any of the people in the Lands Between do for fun anymore
So within this setting, gender itself could be seen as a vestigial tail of sorts that cannot be fully shed by humanity because evolution has been halted which is just part of the larger themes of stagnation and imperial decline
Or I could be high and maybe it has nothing to do with the lore