I found the same thing when I first started shaving with a DE. The skin on my neck was pretty well wrecked from the Fusion. Trying to get it smooth was a guarantee for discomfort (or worse). Months later I've finally got to the point where all the bumps/ingrown hairs/etc. have healed up and I can get pretty close.
Congrats on the improvements.
Ant