To be completely honest, I have trouble believing it would be an issue in competitive sports at all. I'd be frankly shocked if there was a single professional athlete not juicing on whatever undetectable performance enhancers are available on the market, legal or not.
Imo, it should just come down to capability. If a trans woman has an unfair advantage, then she gets prohibited. If she doesn't, she doesn't. I don't mean to infer that this is simple--the procedure for determining this needs to be left to each sport, but I have difficulty imagining trans athletes actually being that far ahead. Particularly if they've actually had gender reassignment surgery. Put them through stress tests relevant to the activity, and plot them next to the bell curve of other players in the field. If they're too far ahead, don't let them compete.
I honestly don't think it's an issue. I can't help but feel that a big reason it's controversial is because people suspect that someone's trying to "infiltrate" the sport, which has been a big part of anti-trans rhetoric for both sides (male/female) since the 50s. It being the area of queer rights where the least progress has made, not at all helped by the near-complete abandonment of trans peoples by the rest of the queer community, I really wouldn't be shocked if this was still a continuing fear.
Or maybe I'm just putting my own perceptions onto the present situation. I'm not going to pretend like that couldn't be the case.
Does this apply to everyone?
As long as their motives don't interfere with the notion of safe space, then yes. If someone, male or female, is going in to make an exhibit out of the men or women in there, they've violated that trust. Using the bathroom associated with the gender you identify with, assuming you have a gender identity, is built into that notion. One restroom to act as a safe space for men, and one for women (who admittedly need it more, due to having less protection and power within our society).
Deliberately entering the room that doesn't correlate to your identity is damaging. To yourself, if you are being barred from the room you wish to use, or fear using it. Or to those who do identify with the correlated gender, for infringing upon their safe space.
The problem as I see it is that we too often correlate gender segregation with sexual segregation. And we shouldn't, because it's been a long time since that made sense with the public perception of society. As a gay man, I'm going to use the men's room. Because I identify as male. Lesbians use women's rooms, because they identify as women. We trust that these groups aren't going to attempt to rape the straight men/women in the rest room (though alma might disagree).
I'm also not going to pretend that it's always clear what the motives are, particularly for trans peoples who have not begin transitioning, or who do not desire to. They may or may not care about trying to pass for their gender to others. If they start peeping, they're no different than any other deviant.
But it's not like they're going to be acting like drag queens. A trans woman in a women's restroom isn't going to be paying any more or less attention to the other women in there than anyone else. The other women are likely paying more to her. But forcing her to use the men's room is surely the worse option.