It's really hard to say if one should eat soy or not because it's effects are dependent on so many factors. It is really a personal decision but I would say the only people who should stay away from it completely are those with a strong history of breast cancer. My master's research was on genistein, one of the estrogenic compounds. My research did show that it had a very slight negative effect on the reproductive system in the male mouse (yes it was animal research but this was 10 years ago before I became veg). But again that was an isolated compound that was being used at a very high concentration.
If you are worried about it, I would stick to the fermented products which do appear to be safer - miso, tempeh and soy sauce for example. Tofu is not fermented though. It is processed, coagulated soymilk (sounds lovely, I know). But again, the effects of tofu might be very positive for an individual. If you are eating tofu you should choose only the organic varieties as soy can be genetically modified and full of pesticides. Highly processed soy products like tvp, soy protein isolate are unhealthy for many reasons.