One of the big advantages of a straight razor is you have full control over the aggressiveness of the blade. So when going against the grain you can lay the blade flat against your skin and it won't irritate the skin at all. I have the same problem as you on my neck, and after I got some experience I started shaving it against the grain on the first (and only) pass in that area. I wouldn't necessarily recommend this technique for a beginner, but it's worth considering after you get a bit of experience controlling the blade and keeping the blade sharp, because although it sounds rough it helps reduce irritation because (a) it's only 1 pass against your skin (b) the blade is flat against your skin so no scraping (c) your blade is sharp enough to simply snip off the whiskers on contact without being shoved against the skin.