Are you sure it's water and not oil or glycerin coming out of the soap?

If water came out, it would mean the soap doesn't hold water, and then the water would evaporate. If it's not water coming out, it won't evaporate.

If you're worried about microbial growth, add a little lavender, rosemary or tea tree oil to the soap. These are antimicrobial, but will alter the scent.