Unless you start with sandpaper that's a lower grit than about 220, it's hard to do serious damage by sanding as long as you pay attention.