oddizido - do you find that the majority of people simply don't care that you're vegan and about veganism in general, or do they just have no reason to comment on it to you, which gives the impression that they don't care? What I mean is that if they assume that everyone around them, you included, eats meat, then they have no reason to bring up the topic of not eating it. What I've found is that if I just mention that I'm vegan, people immediately show interest by asking me many questions (sometimes as veiled criticism) and start trying to justify eating meat, even if I didn't try to debate them. So, it seems to me that most people I've met do care, at least momentarily when they first learn that I'm vegan.
Like you, I also don't tell people unless it directly affects the situation. I wouldn't mention it out of the blue to start a debate.
I can understand why some vegans prefer to stay silent about it, if they don't want to be put through interrogations and debates when the situation isn't even about that. For example, I mentioned that I'm vegan when an old friend and a new friend and I ate at an art gallery, and again when coworkers and I got lunch, and other places where my purpose for being there has nothing to do with veganism, yet it was relevant to questions about what to eat. The way that the new friend and relatively new coworkers responded led me to feel like I have to always be prepared to defend or support veganism, in case the topic comes up. I've learned that being in a minority often means that you have to have well-thought-out reasons to explain it, because it's like you're in an underdog position, whereas the majority of people usually perceive themselves as not needing any well-thought-out reasons, because they're in the majority.
I had no idea that many people would react negatively toward it when I simply mentioned, "I'm vegan." I didn't try to get into debates; I just mentioned it when food was offered to me. I felt that a general statement would help other people know what to offer and what not to offer me, instead of me just saying "no" to so many particular items without explanation. When I said that once at work, a new coworker immediately replied, "Do you want to know how you know when someone is vegan?" I asked how, and she said, "They tell you. They always tell you." She then proceeded to tell me about her sister who's one of those "crazy vegans" involved in PETA. I said that I wasn't involved in activism (and wasn't at the time), hoping that she would get that I wasn't going to try to convert her. I said that I'm vegan for individualistic reasons and that I don't think everyone should be one. I later questioned that...I do think the world would be a better place if every human were vegan or at least mostly vegan. But my purpose was to set her mind at ease, in case she was worried that I would personally try to convert her. I was really taken aback by the response and felt like it would be little different if I were some other minority and she responded, "Do you want to know how you know when someone is gay/feminist/atheist? They tell you. They always tell you."
I feel like by not staying silent (even if it's just briefly stating that we're vegan, when it's relevant to a meal), as long as we're peaceful even when people are saying unfair criticisms, we give people a counterexample to preconceived notions they had about us.