I soak the whole thing. I'd be more worried about the bristles standing on their end during the soak.
My thinking is this; it's just a shaving brush. As far as getting the handle wet, how on earth...