Actually we can't avoid being mislabelled for what we wear. We can just try reducing how many people mislabel us by adorning masculine traits. The rest, happens in people's minds.
Does it matter? Depends on the situation. In LGBT hostile regions, wrong label may lead to reputation/physical risks, but this situation may help finding new friends (solidarity with a perceived common point). In a market, that same wrong label may result to different behaviour from the vendor (being helpful or "forgetting" about a good deal you could benefit from). When you go to a rendez-vous where you need to do a good impression to get something (loan, good car, oral exam, dating, job appointment, etc.) this may do a lot of difference. At an event, the speaker looking for someone to invite with them is more likely to choose the skirted man (same happens for the drunk man looking for someone to speak with in the metro). Etc. Note that I don't mean it's always in the bad direction: sometimes going against the general opinion will grant you things you wouldn't have had otherwise: my first job appointment, I made a lot of well known mistakes, and this accumulation - that I was not even aware about, I realized only later - played in my favor to get the job, they made me seen as "different" from interviewers, eventually in a sense of "interresting".
So for the "does it matter?" question, the answer depends on what can change due to this label, and whether you would welcome these changes or not.
As for struggling with it, this is indeed the main reason why I wear skirts so rarely: entering a long-term relationship with a woman is much more important to me than wearing a skirt (I can't imagine the first to never happen, whereas I would have no problem to accept the second to never happen anymore). The second reason, is quite a consequence of the first, I don't want to put my brother's reputation at risk as we're living 200m from each other. The attention that is drawn on me by wearing a skirt is not an issue at all, I like it a little bit in fact.
Regarding difference between US and UE, I can only tell for the place I live in (which is not speaking about differences, in fact). There are streets where I have no problem being skirted, but a lot of places I wouldn't go skirted out of certain hours and even certain places I wouldn't go skirted at all (mainly in and around big cities). These places are very insecure if you walk on them alone, even as a sportive young man. I will not expand more on this, but there are "groups" of people in these places, I'm not "one of them", which is sufficient for them to threaten me and the law will protect them if harm happens. Eventually in drab/boring I can pass without being noticed but this is not even guaranteed, thus doing so while wearing a skirt is inconscience or having friends there. So I wouldn't say it's better in the UE than in the US, I believe they're quite "comparable" (quotes intended) on that matter. Eventually as a colleague told me once:
There is the principle of shit's conservation: everywhere you have it in equal quantity, but it takes different forms.
, content ideas
, and 
.
