I give mine a good rinse and don't worry about it,
I use whatever soap fits my fancy for the day and yes I have a few that leave a light linger of scent on the brush.
By the time I have soaked and...