A diamond burr from Harbor Freight, ($8 for the set) and a pin vice will taper the hole to size nicely. Fifth from the left hand side works well for me.

Beveling the pin hole peens and fills...