A Pin Drill to straighten the hole out
Pin and the washers should cover your mistake, the wedge end is simply not as critical as the pivot end, you can also glue the wedge to the "Off" side to make SURE it can't shift...
In the future drill the holes with the Scales done, and stuck together with either Rubber Cement or Double side tape,,