I've fostered dozens and dozens of kittens... I worked at an animal shelter, and it was part of the job. I always took the ones that were likely to die, because it's so normal for a foster-mom to get attached to the little lovelies, and be so heartbroken if one doesn't make it that she'll never want to do it again...(and usually screams this whilst throwing a stapler at my head).
If this is something you're going to do regularly, you can look at it like this: The sooner your current group goes back, the sooner you can take the next. After a few litters, you may find you've managed to save a whole extra litter just my letting go.
One foster-mom turned the day into a celebration...she'd bring cookies for adopters, treats and toys for all the other shelter cats, and turn it into a festive occasion. She, and some other adopters, would stay the whole day to tell people funny stories about the kitties, and about each one of thier personalities. <-- Very valuable in making sure each kitten goes to the RIGHT home.
I saw foster-moms get so attached, that they wouldn't bring them back until they were 10 wks or older, in which case it'll be harder to find homes. You wouldn't be doing them any favors...believe me. They would have to spend MORE time at the shelter...on into their "teenage" time, increasing thier risk of catching a disease from the shelter, and making the new home experience even more stressful.
You're doing a beautiful thing!!!