Just paranoia, may be. but there's enough articles out there, ie.:
How to Sterilize a Shaving Brush
By Constance Barker, eHow Contributor
Shaving brushes offer the age-old technique of applying shaving cream lather to the face with a more beneficial experience than using your fingers. The shaving brush not only exfoliates before the razor meets the skin, it also lifts whiskers for a closer shave. Shaving brushes require sterilizing to remove harmful bacteria. Clean and disinfect a shaving brush regularly for a germ-free shaving routine.
That I don't think it's a bad idea.