It's possible that one's own nose becomes de-sensitized to the fragrances that one wears so it seems that the soap smells don't linger. I suspect they remain longer than we realize and can be detected by those near and dears. When washing my face for the morning's shave, I can often get hints of yesterday's aftershave from areas where I've missed with a later-in-the day wash and rinse.