It's probably something a lot of people never consider before veganism comes up. I think watching nature shows early in life and having a scientist for a mother are the only reasons I know what yeast is, as it's really not relevant in day to day life what category it falls into. It would hardly surprise me if most people believed fungi were a kind of plant, and had never even considered that some things we eat belong to neither the animal or plant kingdoms.
It's also probably a product of the term "animals" and "animal products" being used in short descriptions of veganism, which could leave people thinking vegans value those things for the fairly arbitrary reason that they fall into a certain category.
Jesus saved me, but the file was corrupted.