So, if it is true, what is so wrong with being a bit stunted. What in heck does that mean anyway? He makes it sound like you will be itty bitty. I am female, thus have plenty of estrogyn, and am 6 feet tall. I know many women who are naturally large like that. It is genetics, not whether I ate cabbage or soy beans.
Also, pardon my saying, but shame on him for trying to scare you out of one of the healthiest life-styles around. As a doctor, he should know the dangers of eating too much meat. Such statements seem to come from someone who is not exactly curious, and unable to step outside "conventional" knowledge.
Doctors are not magical. In many cases, they push drugs for drug companies because they get a nice kickback. This also goes for philosophies about what you should eat.