Eggs were actually the first foods I gave up, even before meat, and I think they are probably more immoral than some meats.
However I wonder if vegans might be better off conceding the arguments, or side stepping them, when this kind of backyard chicken argument comes up. It is such a minor issue compared to the intense cruelty on modern day egg farms, that a better answer might be: "not sure, but what's more important is that you don't buy any eggs at the shop, since that is really cruel" and push the argument back in that direction. Most people asking that question (assume they're not trolling you) are just testing out how extreme vegans are. They are not really going to raise backyard chickens.
I realise this is not a very vegan argument, but as a matter of reducing animal cruelty, making your arguments more acceptable and palatable to the general population, and not seeming extreme, it may make some sense. Once the person enquiring has seen that vegans are not a bunch of extremists, they can become vegan, and probably will never raise backyard chickens anyway.
I cannot accept that it is fundamentally immoral to eat an egg in every situation. The real issue is that in practice it will be immoral in at least 99.9% of cases and I think that's what the argument should be. Let's say tomorrow I find a chicken plant nearby is closing down and all the chickens are being killed. I go there and ask to rescue one. They agree. I take her home. I keep her for a while in an environment, my garden, where she has no predators, and no fences, and could in theory wander off into the nearby forest and become a wild chicken, but she chooses to stay in my garden where I give her food and treat her as a friend. One day, she lays an egg and dies later that day in an accident (ie not for health reasons so I know the egg is fine). I bury the chicken respectfully. I know for certain that the egg is unfertilised because she has had no contact with other chickens for a year. In this situation, it can't be immoral to eat the egg, can it?
I wouldn't eat the egg in that situation, but I wouldn't say it would be wrong to do so.