if soy generally made peoples boobs bigger, someone smart would have cashed in on it by now, and you'd see tv commercials for soy-supplements everywhere, and loads of men with bras on in chinatown. it hasn't made mine any bigger.
if you're concerned about your hormone levels, a trip to the doctor and a blood test could allay or confirm your worries. even if your hormones are out of whack, i don't know if it'd be easy to narrow it down to soy.
have you considered other factors- like the pill (if you're female) or medication or otherwise induced gynecomastia (if you're male), for example? are you drinking bucketloads of soya milk? i guess you could measure yourself then cut it out for a few months and see if there is any change. almond milk is nice. ricemilk is nice. i don't like hemp milk. i think a little of most things in moderation is probably ok.