I think there is a difference between enjoying someone else's cat and occasionally feeding it, and wanting to 'steal' someone's cat away from them. When I first moved to my current home, there was a gorgeous/huge/black cat that came to my back door every morning/night. I fed it. I don't know if it belonged to anyone...and it certainly wasn't friendly. I could only get within a few feet from it. If I tried to put my hand out to it, he would immediately scratch me...so I just kept feeding him whenever he came to my door. I probably would have taken him in if he had let me. Certainly no one every came to me and said, 'that is my cat'.
Your circumstance is a little different. I think it is rude/awful, to say the least, if these people are actually trying to steer your cat away from you. It's your cat, whether or not anyone agrees/disagrees with the fact that it is an indoor/outdoor. I have four cats, two go outside and two stay in. It's their choice. I always initially try to keep pet cats in, but end up letting them dictate what their preference is. The two that stay indoors are perfectly happy doing so...they know they have the option of going out if they want to...they choose not to. And I also have a cat the 'visits' the neighbors when they are in their yard....he is a very friendly/lovable soul. They do not, however, let him in their home.
Anyway, I digress....(I had caffeinated coffee this morning...I'm a wee bit hyper) I think it's awful that your neighbors are doing this, especially since you have already spoken to them about it.
If, however, there is some reason for them to believe the cat is not being taken care of, then they should approach you with what they think the problem is. And that's all I have to say on the subject...for now...