Could be a couple things on the razor burn. Many sponges come with built in detergents, particularly to prevent bacteria build up. All of them (unless they're naturals) have some sort of chemical make up, and that could be irritating your face.

Only you can tell if you're technique is to blame.

Personally, I rinse the razor with short spurts of water, or I wipe it with my finger. The water doesn't hurt the edge, and the finger I'm careful enough with that I know it doesn't hurt the edge of the blade either. It is possible, however, that your wiping technique on the sponge is degrading your shaving edge and therefore resulting in razor burn.

Only you can really say.

Peace,
Jim