I had trouble here for a long time.

I keep it light and use short light buffing strokes.

Experiment a bit to find best directions: e.g. From nose to ear vs ear to nose, Going south-east/west...