My initial though would be that you don't get the brush loaded enough.

I leave about a teaspoon of warm water on top of the soap while in the shower to soften it, makes is much easier to load,...