Are you switching hands when you switch sides of your face? If so it might just take more training for your hands to do their jobs equaly well on opposite sides. If not I'd check and make sure you really are keeping the same stroke direction when you move over to the opposite side of the face, The feeling you describe could be going froma comfortable with the grain stroke to a harsh across the grain stroke.