I'm new at this and probably am doing this wrong, but so far its working for me. The belt sander is the way to go!
I basically have the scales in final shape. Then I cut the spacer out and glue brass to each side. The whole spacer is oversized.

I then glue the spacer to one side of the scales. I drill through the pin hole through the spacer into a piece of scrap wood. Then after I pin the other scale on I GENTLY sand the spacer down to size.

On the 5 or 6 sets I've done I haven't ruined any scales yet. Your fear and paranoia about ruining all that work are good "guardians" to keep you from going to far and keeping a light touch on the sander.

If you want the spacer to fit perfectly you have to sand just a hair off the scales. that way everything looks like one piece.

Get a belt sander and a scroll saw. Life will be much easier and you'll enjoy the process a lot more.