I think you aren't intolerant. I am, and I can tell about two minutes after I have something with dairy (at least uncooked) involved...the inside of my nose swells and I can't breathe as well. Not to mention stomach problems.
However, occasionally I have had products that contain casseine (though I stear clear of whey), and don't notice a difference.
Not everyone is sensitive, but lost of folks are.