Yep, I would guess that it's the soap, drying out as the shave progresses.