I build lather on face too and have had this happen to me. I think the reason was not loading the brush enough, and also not working the lather as long as necessary.

Obviously, combinations of...