I partly disagree with your reasoning but I agree 100% with your conclusion…

I think that statistically heterosexual women have some significantly different healthcare needs than lesbian women and gay men and straight men also have some statistical differences, but as a healthcare professional you have no right whatsoever to refuse to treat based on those differences.

(I wouldn’t count referral to a specialist as a refusal to treat.)