Hi Ben,
Agreed...you do remove the oil with polar surfactants like soaps...after all, it is exactly due to their polar nature that soaps can effectively remove oil (hydrophobic part of soap binds to the oil, dissolving it and the hydrophilic head of the soap binds to water allowing it to be washed away). I still would like to minimize the use of baby oil on face.
Personally, if I end up using soap to wash the oil off after my shave, I end up washing off most of my face's essential oils, leaving my face dry...My face, in particular doesn't like "hard" soap after a shave. Also, I have noticed that one to two uses of baby oil shave and brush, coats your brush with oil. I had to resort to washing the brush a few times with soap (mild detergent) and hot water to remove the "gunk". I stopped doing this.
I prefer Glycerin...it moisturizes and is water soluble (no soaps required to remove it from face and keeps your brush problem free).
At the end of the day, whatever one does is a matter of personal preference, I think regardless of the science behind it .
Thanks,
-Robert