Whenever I am grinding or polishing, I use bare hands.
I don't linger on one spot, and if it is too hot to comfortably hold, I dunk it in water or let it air-cool.