Finishing the wedge/spacer end

I have a problem finishing the assembled spacer/wedge end of the scales using CA.

Shaping and finishing the scales and wedge separately is relatively easy. However no matter how much time I spend, when assembled after finishing, the junction between the wedge and scales is not smooth. If I apply CA after assemblage in order to smooth the end, the scales and wedge become glued together. I don't want this.

Any suggestions on solving this are welcome.

What I've done is to use a flat glass plate with wet and dry sandpaper to flatten out the wedge and make sure it fits flat against the scales.
It is a bit of a long process but worth it to get the best fit IMHO