I have an idea that may help if you don't have a drill press and are worried about the bit dancing around. DISCLAIMER: I most certainly have not tried this, it's just an idea.

Use a small drill...