i've been doing some research on this as well, and i have found some compelling research that suggests that US (and i suppose western in general) consumption of soy is excessive. The amount that i read was that in asia, the average person consumes three grams of soy a day, where as a western 'RDA' is somewhere around 22 grams a day to be considered "heart healthy" among other things. So, the consumption is excessive in relationship to what asians commonly consume.
similarly, i haven't read much about it affecting size growth (as in, how tall, how much they'll weigh, etc), so much as affecting the sexual growth and development of boy children particularly. Some of the research that i've read discusses these ideas--which of course may be correlary rather than causary--but i would still take this information seriously.
Finally, when you're dealing with breast milk, it's a raw form, with far less estrogen per volume, or other hormones, than with a pasturized cow's (or goat's) milk. When you're talking about raw milk, in moderate to low consumption, the numbers or amounts of hormones per volume greatly decrease (as do antibodies etc) because you're actually looking at a broader nutrient spectrum particularly in regards to fat and protien. So, raw milk may be a better choice than soy milk. breast milk, of course being best for babies--and some ween earlier than others. Some as young as 12 or 14 months, others as old as 4 or 5 years old.
westonaprice.org may be a place to check out. they're not a veggie organization, but they have some interesting ideas and research that i've enjoyed reading.
also, ryan and i have decided to go off soy, taking our consumption down to 3 grams a day or less. I'm doing well on less. for the last three weeks, i've only had about 12 grams total. it's been an interesting journey.