If you want to get flush the wedge and the scales , you will probably have to assemble CA the wedge to the scales, then CA all. After that redrill the pin hole.
Get ready for a lot of hand sanding...