For myself its mostly about the way the hair grows. Mine on the upper lip grows more out to the sides than it does straight down so if I was to make a pass straight down I would be cutting across the grain. For my first pass I tend to shave slightly across my lip then come back with a straight down pass. This might be something to think about, pay attention to the way the hair grows.