I used to not rinse my brush after shaving, but after a while the hairs started falling out. Now granted it was just a cheap vdh brush and it was probably not completely due to leaving soap in it but it only takes a few extra seconds to rinse it, so I rinse now after every shave. I give them all a good wash maybe once or twice a year.
-dan-