Maybe your not totally washing it off after shaving? Any soap should rinse right off unless you're making it too dry and it's adhering to your face.