I would be MUCH more worried about a brush than a razor personally. The very act of shaving uses soap all over your face which is a disinfectant. You are also stropping the edge at least 30+ times on linen and leather pre-shave and I give it at least 10 more laps post shave as well then let it air dry over night or at least 8 hours.

Dry polished steel is a very sterile environment unlike rusty pitted steel.

There is is nothing wrong with using caution, but unless you are a barber shop using the same razor on numerous people I just don't see the danger.