In my Torah studies I've learnt that not only should you do the right thing, but you should also be seen to be doing the right thing. The example that I was given was that an observant Jew shouldn't enter a non-kosher restaurant to use the facilities, lest someone think that he/she might be eating there.
At first glance this seems absurd. Why should anyone be so suspicious? What is it their business anyway? Should we be more concerned with the "appearance" of doing right (or wrong) or the actual practice??
Certainly you are correct, we should be more concerned about what we do than how we look. But this does not mean that we can completely ignore the way things appear to others.
We cannot be invisible. Nobody lives in a vacuum, unless you are a vacuum cleaner bag. Our actions impact others whether we like it or not. Every individual contributes to the social fabric. And so we are not only responsible for our actions, but also for the impression they make, because we are responsible for the morality of others, not just our own. Any behaviour that may counteract the furtherance of goodness is a moral problem.
It isn't about my reputation as much as about my influence. When I do something that looks wrong, even if I have a perfectly good explanation as to my innocence, the damage is done.
If I enter a non-kosher restaurant to use the facilities, while I have not broken any law of keeping kosher, I have crossed the divide between kosher and not kosher, and invite others to do the same. If I take shelter from torrential rain under the awning of a house of ill-repute, I give credibility to that place that it does not deserve.
But there's a deeper reason not to do something that just looks wrong, even if it isn't wrong, and even if no one is looking. Not only can such activity affect others, it can affect us too.
Stage actors know that when you play a character, you can sometimes become it. The self we project to others can sometimes be absorbed in our own identity. And so by looking like you are doing something wrong, you may come to actually do it. By feeling comfortable in a place that you don't really belong, you may end up thinking you do belong there. You can't remain immune from your surroundings.
This law teaches some powerful lessons. You affect your surroundings and your surroundings affect you. We build a community together, and so we are all responsible for it. Your morality is my business.