I used to get that a lot when I was using a dremel for restoration work. I never did find a good way to even it out.
If it's shaving good, I would just leave well enough alone.