your diet would be veiwed as restrictive by lots of people, but whether or not its too restrictive for you, and whether or not it has a positive or negative effect on your physical and mental health, is another thing.
i think your therapist perhaps has concerns about the restictions your diet places on you, the control that you have chosen to maintain over your food, and the concern and perhaps anxiety that you might have around eating the 'wrong' foods, and has maybe put it all together in her head as meeting some of the potential criteria for an eating disorder?
whether she's right or wrong, its her job to think like this- i'm on a self imposed restrictive diet, (i have food intollerances and sensitivities) though mine is slightly different to yours (vegan, no wheat, yeast, gluten, sugar, very small amounts of fruit, no alcohol, no mushrooms, no moulds or fermented foods- like beer, miso, vinegar, msg, soysauce, etc) and i've had similar concerns raised to me/about my diet by people who don't understand the reasons behind it, who are trained to look for problems, and by lots of people who haven't seen how ill i get if i don't eat like this (if they see me when i've 'slipped' and eaten something i shouldn't, then they get it, lol).
to many people in the mainstream, its unfathomable to live/eat like this, and to be honest, if i knew someone who ate like me, and who didn't get obviously ill very quickly if she didn't maintain this diet, and especially if she was underweight (i'm so not, lol) or slightly inclined to other mental health issues ( OCD's, depression, excessive anxiety, control, guilt, fear of underacheiving, etc) the first thought in my head would be 'eating disorder', and in many cases it would be justifiable.
i think it depends on a lot of things. how strict your criteria is for something being 'unhealthy' and how rigorously you maintain your regime, for a start. i could pick holes in almost any food item, and label it as unhealthy or not 'right' nutrition wise, if i wanted to, to further an eating disorder. if i was going through this myself, a few points i might consider might be:
do i have a history of obsessive or compulsive behaviour?
what do i do when i think i might have eaten something i perceive to be unhealthy? (fret, panic, punish myself, puke, feel guilty?)
do i make my diet work for me to a positive end, or am i a more of a slave to it? (can i enjoy my life and day to day stuff without carrying a list of unsuitable foods around with me, assessing my nutrient intake repeatedly, assessing the nutritional value of everything i ate so far that day, and struggling to work out what completes my food intake needs for the day, etc)
how do i manage/deal with/approach cravings for unhealthy food?
(if i crave cookies, do i tell myself its a bad thing to do, or feel guilty about it, or aspire to do/be/think better, or do i take the challenge of finding a way to make (or buy) some cookies that aren't full of crap, and then enjoy them guilt and worry free?)
ETA: i think that if you feel that your diet is causing you worry or stress, (you just said above that you're petrified of getting fat and damaging your body- sounds like worry and stress to me!) its time to talk to somebody about dealing with these problems- before they (and perhaps your diet restrictions) get out of hand.
ps: you can be relatively healthy and a little overweight. also, worry isn't always rational or in perspective: while its great to use common sense food wise, how would you feel if you had the perfect healthy diet, and then you damaged your body by breaking your neck tripping over a bag of organic lentils in the health food store? sometimes you need to find a balance of whats 'ok' and whats 'good enough', and be able to maintain your happiness as well!