I am the same way!!! I had a friend whose cat was frail, grumpy, and his hair was falling out. He had been sick for a long time, and once he got better, the hair and weight took a long time to come back. Noone in their house would cuddle or pet him, because of his bald patches and frailty.
Whenever I went over I would spend the whole time cuddling and loving him, and trying to convince my friend that there was nothing "gross" about petting him. Just because her cat doesn't look "cute" doesn't mean she shouldn't cuddle it, love it, and spend time with it.
I totally have a soft spot for him, he's cute to me, even with bald spots!! haha.