I'm sorry, but I'm gonna have to say no you're not right to not have spayed/neutered them. Had you just spayed the first cat this problem wouldn't've occured in the first place. Does this make you a bad person? No. But you need to be more responsible with your animals. Try to work out a deal with a vet or the ASPCA to get all the female cats spayed. Its more expensive than the males, but it only takes one male to impregnant however many females you have. So if you can't do both, do all the girls. Sometimes shelters will sell you spay/neuter certificates for less than the cost of the surgery. Any subsequent litters of kittens should either be adopted out through you as kittens, or brought to the shelter as kittens so people who can afford to have them vaccinated and spayed will do so.
You really ought to have the cats tested for Feline Leukemia and Aids and then vaccinated for it if the test comes up negative. It would be VERY harmful to the cat community in your area to have any FeLV/FIV cats around infecting other cats. Rabies vaccine is also very important. I don't know where you live to know what the laws are but look into it. In NY if you've been feeding an animal for 2+ wks, its your animal, and you are legally and financially responsible for it and its actions. Also, if you have not vaccinated your pet against rabies in NY, you can be fined up to $10,000.