Sounds like the razor. This is a characteristic of multi-blade cartridge razors. If it were the soap you'd have the problem on your entire face even if you didn't go against the grain.