Most people don't want to share restroom facilities with people of the opposite sex. Arbitrary or not, that right outweighs the right of the small number of people who do want to do so, regardless of their stated reason for wanting to. Thus, that's our social rule. You can always work to change the minds of the people, but the wrong way to do it is to pass a law forcing them to comply with something that makes them uncomfortable. Doing that is the opposite of the principles of a free society.
I'll keep that in mind when SCOTUS goes 7-2 in favor of codifying the right of **** folks to marry in June. I'm lazy, but there are about 72,000 examples of how the US system doesn't and wasn't intended to function the way you describe. Were you thinking, perhaps, of Galt Gulch?