Lawson,

Instead of using a drill bit, switch over to a fine, cone-shaped grinding point.

Make sure that whatever is doing the grinding (Dremel, small drill press, etc.) cannot wander.

Also...