These guys seem to think the oil in witch hazel is fine:

Homemade After Shaves : How To Make Homemade After Shave Lotion & Balm