Exactly, the drill bit you need will be very thin so prone to 'drift'. I have damaged a nice pair of scales that way.

A non-bevelled wire cutter works better for me.