I guess I don't think about or deal with such things. When you get to a certain age (I think), hopefully you realize what a waste of time it is to worry about how others 'see' you. I look at it like this. My life's too short to waste it on people who are unsupportive or negative. I dropped a friend last year for this very reason. I felt badly about it, but realized at the same time, what a negative influence she'd been on my life for the ten years I'd known her. And it was time to stop swimming in that pit of negativity with her.
I think what becomes important later on is what kind of people do YOU want to spend your time with. Most everyone in my life is supportive of my choices (dietary and otherwise), and I am of theirs. I just don't want folks in my life who want me to feel 'badly' about myself.
Hope this makes sense.