If you're not going to use the scales anymore could you drill holes from the other side and push the inlay out?

I imagine that as well as being a press fit, the inlays will also be glued in...