Theres no need to blow dry your brush and it probably wont do it any harm unless you really cane it with the dryer.

I just tend to flick mine dry and then foof it with a towel and hang it up. Job...