Along the same lines a sturdier set up might be to drill and tap a 1/2" block of steel or brass and use a cup faced set screw.

A lot of knifemakers also use a cup faced punch in a drill press...