BMI of 20 is slim but still healthy.
I wouldn't worry about your weight if I were you, as long as you eat well and are healthy then you'll just look healthy. I have a friend who's skinny but it's her natural weight, in fact she eats more than an average person would yet she looks skinny. On somebody else who'd dieted to that weight they might look underweight, but you can see by looking at her that she's natural smaller framed.
BMI isn't all that accurate and should only be used as a rough guide. Things like bone mass, body fat percentage and frame size aren't taken into account.
As long as you eat a healthy amount, I wouldn't worry about people saying you're too skinny.