FUD, maybe you're just more careful on the 2nd+ passes, and thus, it takes longer.
btw, I currently make the lather more watery for each subsequent pass. On the last pass it is almost like soapy...