There is one other possibility that you might explore and that is cleaning and sterilizing your brush and razor. It's rare but it is possible there might be something on either or both thats contributing to the irritation.

At the risk of being labeled an alarmist (which I'm not) or starting a panic (which I normally only do when my nieces boyfriends see me honing my straights ) here is a link you might want to check out.

Disinfecting a brush - Badger & Blade