Are you supposed to rinse-clean the brush every time after the shave?

On my brush, there's usually good left-over shaving cream. And as expensive as my shaving cream is, I'd like to conserve as...