Going by what I've read and listened to I'd say that you could do with upping your HDL a little bit and lowering your LDL a bit or just upping your HDL. The doctor was probably a little unsure as to what to think as he/she probably doesn't encounter those kind of numbers with their "normal" patients.
I have encountered it being said that the total cholesterol number isn't really that important, it was believed to be, however the evidence hasn't really backed that up, it's the ratio's that now seem to be important i.e. anything better than 1 to 4, you have 1 to 2 so he/she is probably concerned only that your HDL numbers are just so low.
It's something you will have to look into personally and decide for yourself but while the keeping the total figure down is regularly banded about as the way to go there was a french study where the cholesterol level remained the same but the ratio's changed from 1 to 10 to 1 to 4 and there was a big reduction in cardio vascular incidents/death within the study group.
By your own admission you could do with being more active, here's a few tips I took from WEBMD with regards to raising HDL.
5 Ways to Raise Your HDL Cholesterol
Some diet and lifestyle changes help boost HDL cholesterol levels:
Get active. Physical activity can boost your HDL level. Get at least 30 minutes a day of moderate activity, most days of the week.
Lose extra weight. If you're overweight, losing extra pounds can help raise your HDL levels, as well as cut your LDL ("bad") cholesterol levels.
Choose better fats. The healthier choices are monounsaturated and polyunsaturated fats. You’ll find these in plants, nuts, and fish like salmon or tuna. And, like everything you eat, keep your portion sizes small. Fats pack a lot of calories in small amounts.
Alcohol in moderation. Drinking moderate amounts of alcohol is linked to higher HDL levels. If you don't drink now, check with your doctor before you start, since alcohol has some risks not related to cholesterol.
Stop smoking. Kicking the cigarette habit can raise your HDL level.
WebMD Medical Reference