There are different levels of cruelty. Here's my thought process on the subject:
1. The obvious: Meat. It is a carcass of an animal that has obviously been killed.
2. Non-meat animal products: Dairy, eggs and honey. The extraction of these items could be cruelty free, but most often aren't.
3. Animal by-products: Gelatin, casein, glycerin, shellac, etc. This is the "gross" category and most people (even non-vegetarians) don't like the sound of them.
4. Trace ingredients: This would include bone char and other small portions of cross contamination (cooking veg*n food on a grill that hasn't been thoroughly cleaned and food processed in a factory that also uses milk, eggs or fish). If you want to avoid this category, you should never eat at a regular restaurant or eat mainstream packaged food ever again. You can only eat at 100% pure vegan restaurants and 100% vegan certified foods. This will limit you severely. And this is the level that is being debated in this thread.
5. Ingredients that cause indirect harm: Palm oil (vegan in itself, but considered bad for displacing wildlife) and production of vegetables (you know millions of insects and other small wildlife are getting crushed by farm equipment by planting and harvesting). There is also the human factor of ethical working conditions. Humans are animals, too. So, if workers are mistreated or regularly injured by bad practices, you may not want to be buying those products.
I draw my line at #3. No ingredients from #1 to #3, for sure. But I still try my best to avoid #4 & #5. That's about the best way I can present my feelings on this subject.