The last sentence seems to suggest that isolating yourself is psychologically unhealthy. For some people it actually improves psychological health, don't you think?

Certain people will always be mistreated by others because of how they look. If an individual is in that "higher risk of mistreatment group" (e.g., fat or ugly), of course being around others will make him/her anxious. Avoiding others, therefore, would serve to bolster self-esteem as there will be no one there to constantly reinforce negative self-assessments. Put simply: Loners get less negative feedback because they get less feedback. When you're not around others you don't have to worry about your looks, since you like yourself just fine--it's the other people that are the potential problem.