It's because vegetarians frighten people.
If intelligent people make the conscious decision to abjure from meat, then there has to be a good reason. People like to live life and eat meat without thinking about it, but when you appear, they are forced to think about it. And as far as I'm concerned, as soon as they get to the conclusion that their steak used to be a living creature capable of feeling pain or emanating emotion, they will not be able to live so ignorant. And ignorance is bliss, after all.
It's also why they like to confront vegetarians. On numerous occasions, I have been approached by a meat eater who tried a quick compromise, like "Well, I guess everyone should live as they want... can't really do much by avoiding meat, anyway." And then they try to walk away so that they can rationalize their argument as an unchallengable statement, a blank check to continue eating meat, and keep their conscience clean. You must never let them walk away until they see that there is absolutely nothing they can say against you.
So yeah. Vegetarians are actually a big thorn in people's conscience. It's like a very fat person lying on the couch, eating doughnuts. Then a tall, athletic individual walks by on their way to the swimming pool. The athletic person would remind the fat person that they are living a different, better life, and the fat person would have to think twice before stuffing the next doughnut in their face.