+1 I do the exact same thing. From my chin down to my Adam's apple my beard grows from left to right. Have not found a way yet to get it with the straight. I use a DE razor and can get it acceptable. I don't like going ATG with either the straight or DE. Too much irritation no matter what. Usually just a N to S pass and a little XTG.
On the Adam's apple and either side of it just a S to N pass just as stated above. If I try to do a N to S ATG pass I get irritation and ingrown hairs no matter how good the razor edge, face prep, or light touch.