It’s still seen as weird most of the time.

But to the extend that there is a difference, it’s due to the general power imbalance between genders in our still pretty patriarchalic societies. And gender norms and gender stereotypes, people have a set idea of what men and women are like and “old woman takes advantage of young man” doesn’t really fit into that; consider how hard it is for men who have been raped to get justice, especially if the offender is a woman.