From my personal experience, I can say with certainty that it absolutely fades with time. Over almost two decades of wearing skirts openly in public I have only had one or perhaps two negative circumstances, and those inevitably involved drugs or alcohol.
Getting dressed in the morning , I just don't sweat the matter; what I wear is what I wear. If somebody takes umbrage with that, it's their problem unless they make it mine, and in the latter circumstance I can usually best their argument.
There is vast wisdom in that statement, and I concur with it. Most of the battle is in our own minds, but when we can cogently and articulately defend our choice of attire in the face of criticism it does confer extra confidence.It's when you do have those little scares, but they don't result in a problem, that the fear starts to fade.
 When I bother getting dressed whilst in solitary-confinement that is. Half the time I'm in my night-clothes.