Soap residue can clog your pores and cause acne, and cause skin dryness. It's best to rinse the lather off completely.