The jury's still out on this one, but I've not noticed any difference in my cycles with eating soy vs. not eating soy.
But understand, I eat soy in "whole" form like tofu, miso, soya beans and edamame.
One thing I do avoid is an ingredient
called Soy Protein Isolate (not to be confused with soy itself). It's in a lot of packaged veg*n foods and to me, it's right up there with hydrogenated oils. It's unknown if SPI is bad for your health, but since it's a processed ingredient, I prefer not to take chances (but that's just me).
To help quell your other fears: men will not grow breasts with soy (neither will females - sorry), soy will not increase the risk of breast cancer, men will not have lowered spem count/females will not become infertile when eating soy and IME, it will not throw off cycyles.
Stress, not eating properly, or too much weight loss (or weight loss that's too rapid) will throw off a cycle long before a block of tofu will.
Each person is different and some will have sensitivites that others will not have so if you're noticing that you're sensitive, the best course is to avoid soy completely. If you're not sensitive, though, I would say enjoy soy in it's "whole" form and don't worry about the scare-mongers.