People dissing what you love just because you love it are bores but that's just my opinion.
Collecting and things like Pokemon and Yu-Gi-Oh appeal to children's innate sense of delight in collecting and "completeness". Many adults never outgrow the comforts of collecting but switch to more "adult" things like coins or stamps or expensive antiques. It amounts to the same thing and it stimulates and soothes the same part of the brain. If your dad collects ANYTHING then feel free to make fun of him.
Before Pokemon came along people, esp. children collected butterflies, stones, four-leafed clovers. You name it. It's a very human trait and unless it's taking up all your time or money or you start collecting cats or your feces in little jars I would not let other people's hang-ups get to you. Tell him he's making you neurotic and that he will be responsible for your therapy if he keeps harrassing you. Or tell him that in deference to his opinion that your tastes are too juvenile you switching to collecting Playboy. That'll learn him.
But that's just MY opinion.
I helped my stepson collect Pokemon for years. He's now 12 and still digs them. He's sharing his collection with his half-sister, my daughter, and is teaching her a dumbed down version of the card game. He's funded his snowboarding obsession by selling off some of his nicer cards.