I agree with Stuart as well.
The other thing is not that you are using soap, but maybe too much if you wash with a harsher soap, then add a soap for the shave it could have drying qualities.
There...