If you load the brush with shave soap first you should be fine. Soap
including shave soap interacts with oil so the oil will wash off. You can always
add one final re-soap and re-lather step to the brush clean rinse step.
The puck of soap will be damp and soft so swirl up a bit of
extra soap. Work the brush and fresh soap on the palm of
your hand for a bit then rinse clean and shake the water out.
So unless you are dipping the brush in oil and letting it
soak in oil you are OK. Some knot glue could dislike
an oil soak. A painter would not give this question a
second thought other than the cleaning step at the
end of the day.